> If I clone a repo, I want a pretty limited history for that repo when I clone. If I start to go back in time, spin up a worker to go fetch that stuff from the VCS when I need it.

You want blobless clones:

    git clone --filter=blob:none <url>