I'm guessing that's with whatever stock software/firmware stack Raspberry PI themselves ship, surely can be made a lot faster by not having to probe various hardware ports you know aren't used, for example.

I didn't have it complete in time, but you can use pi-gen-micro from Raspberry Pi (it's a newer project) to build a very lightweight boot image now, on the order of a yocto/buildroot image, but with Pi OS as the base.

Just pruning a bunch of systemd units will get you to half that time.

Yes, but you could do that with most devices running linux.