> 500MB is just not enough RAM for a good HA experience

Strong disagree: my experience has been great, my HA has been running on a Zero 2W for more than 2 years! I have several HACS plugins enabled - just not any if the video or AI stuff. The same Pi concurrently runs PiHole. For a while, it also acted as a git mirror via iSCSI-backed Gitea, but I had to migrate Gitea off of it since it was memory-hungry. You can do a lot with 500MB in headless mode.