Is it that hard? Every time I moved to a new phone, whatsapp's backups are in my google drive and restored without any problem whatsoever

It’s not hard, but if you do one step out of order, your backup becomes unusable and all your history goes bye-bye.

If you don't back up to google drive, the process is much much more hairy. The transfer looks smooth but I have seen it fail in multiple instances.

These days I learn not to get attached to my message history