ARM boards need a custom kernel and bootloader. These aren't things managed by the distro. The userspace IS managed by the distro and IS standard. Using yocto to manage userspace maybe made sense 15 years ago, but it has long since become far more trouble than its worth. Debian supports most architectures out of the box, has good cross-building infrastructure, many thousands of ready-to-use packages, and is non-weird.

Debian handles userspace fine but there is a lot of image customization needed a lot of the time. I have experience with developing images for a beaglebone black, where I forked the bespoke image builder (to create user accounts, mount points, set up udev rules, device tree overlays, ssh keys, etc) and that was a pain to maintain. I suppose I could have create debian packages for some of those things, but I would still have needed to set up repositories. Now I'm using an SoM with Yocto and while the learning curve is quite a bit steeper, it is much easier to customize for our experiment's needs. Also, AI assistants are really good at spitting out yocto recipes (that usually don't work immediately, but gets you 90% of the way there).