That's why I build project specific images in CI to be used in CI. Running apt-get every single time takes too damn long.
Alternatively you can use a local cache like AptCacherNg