My biggest criticism with SPM is that there doesn't seem to be a way to use it with git worktree since the package cache is centralized as opposed to being at the directory/project/workspace level.

Can you elaborate on the use case? I've started putting my local package dependencies into Vendor/ submodules in the repo I'm using them. I can't use worktrees with this setup?