How many other people have hit the pain of injecting ssh keys for gitlab into a docker container building your code in CI?

Compiling code shouldn't hit network by default.

why you need ssh key for CI? The CI_JOB_TOKEN or scoped PAT also works via http to pull/push the code.