What I don't understand is why it is so much effort to use linux on a phone. Surely these 8 core ARM monsters these days should be more than enough to handle a full kernel. Hopefully it's not a driver issue where manufacturers only contribute the necessary drivers to the android kernel, not the linux one.

Android is running on Linux. It's not the kernel that's the problem, it's the application layer.

Battery/sleeping is the main challenge I believe, not processing power. Linux laptops still struggle a lot with sleep. And Windows laptop too btw.