I was tempted to do something similar when I got my Yellow because the low availability of the CM4s back then, but it seemed that, at least with older Radxas there were problems with kernels or compatibility. It might work with the newer one if you say it's mainline kernel but haven't found anything in the forum in quick search...

I am keen to try. But I'm even more keen for someone else to tell me they have tried it and the result of said test because I am not keen on my home automation system being down for a longer period of time...