Whenever I switch android phones, I need to setup each app from scratch, unless is a google app.

What do you mean "set up"? Like when I get a new phone, my previous apps just reinstall themselves. I might have to log back in, but that only takes a sec with a password manager.

Does it not work like that for you?

I've only gone from Nexus to Pixel to Pixel though (all Google phones) so maybe that's why?