SMB on macOS is and always has, and probably will always be utter shit.

Mount something over NFS< and you'll be relieved about how snappy things remain. Snappy relatively of course.

Yes, there's some bug in the backupd that panic.. no matter smb/nfs

I tried moving to NFS, but the level of complexity of NFS auth is just comical. I gave up after trying to set up a Kerberos server on the Synology that I was trying to access. It's too much.

Using unauthenticated NFS, even on a local network, is too dodgy imo.