Looks like in Chrome on mobile you can set it per site, but you need to dig into accessibility for that. I don't see a way to do that in FF, so you would have to set the default which would effect all sites.

Firefox remembers your zoom settings on each site:

https://support.mozilla.org/en-US/kb/font-size-and-zoom-incr...

So no, you don’t have to set the default.