If you want to run EspHome inside HA, and you recompile the devices (every release of EH), you want a decent processor/disk. The ESP stuff is a surprisingly heavy compile for a puny microcontroller.

A recent RPi is sufficient to handle a few plants - though, yes, recompilation will take time. The ESP is a beautiful piece of software, hence I highly recommend it. My native language has an expression that describes this situation perfectly: the appetite grows with eating. Next thing you know you have 2k or more entities, and your HA even handles some video feeds.

The important thing is that it's pretty much always easy to make an upgrade thanks to the good design of the backup system. Don't forget to set up backups in either case, it's a sin to not use such a complete system :)