Any sufficiently-advanced automated rsync would have a filter for caches.
Except only ssh is filtered. Just commenting on what I see, not what should be
For sure, you need to exclude whatever "dotfiles" you don't want copied (or explicitly copy the ones you want), particularly caches and other giant hidden things.