How is the bootloader/peripheral compatibility on the non-SBC ARM systems these days? Can you plug in a boot disk on different machine and expect it to just work? My main problem with ARM is that many manufacturers act as if they're special little snowflakes and deserve to have their custom patched kernel/bootloader/whatever.

This is the goal of the Arm SystemReady compliance label. The selection is still pretty limited and what's out there is generally buggy, but there's a few boards out there you can buy like the Orion O6 [0]. If you just want a stable system with predictable performance, you're probably better off with a more traditional system though.

[0] https://radxa.com/products/orion/o6/

Afaik a lot of bootloaders are proprietary/wonky, a lot of SOCs run custom bootloaders.

However if you do manage to boot things up, hardware with open-source drivers should just work, for example Jeff Geerling has couple of videos on youtube about running his RPi with external AMD graphics cards connected via PCIe, and it works.