I worked with ARM boards, I know a bit about it. Booting into Linux is never hard, it's all about using uboot, sometimes with tiny patches on top. I think it's actually even easier with android phones, as you don't have access to the low level bootloader, you just use fastboot stuff.

Having basic framebuffer in BIOS/UEFI is neat for toy OSes, but not very relevant for something practical. You gotta need proper driver for GPU. And if you're just starting, UART console is actually more preferable way to interact with board, IMO.