Why? I'm running an Orange Pi 5+ with a fully generic aarch64 image of Home Assistant OS and it works great. Is there some particular feature that doesn't work on mainline?

for server use you can live with generic images. When you want stuff like HDMI audio out and all, generic images usually won't do.