How is it crazy? It's perhaps not granular (the repository is the boundary, and that's that), but you can definitely restrict who can pull or push as easy as you can make rules for SSH.

Plenty of not-very-granular "enterprise" systems out there, it's not exactly unique to not always have full ACLs on the smallest of objects.