Try checking what M-x magit-toggle-verbose-refresh has to say, you may be able to identify what's slow and optimize around it

I would guess the roundtrip of intercepting every file stat call in the repo, it's very lovely. But will check.