gmm1990 5 days ago

Why not run some open source ci locally or the google equivalent ec2, if you’re already going to the trouble of this much customization with running GitHub ci?

▲

a_t48 5 days ago | parent [-]

It was half a day of work to make a drop in action.yml that does this. Saved a bunch of money (both in bandwidth and builder minutes), well worth the investment. It really wasn’t a lot of customization.

All our builds are on GHA definitions, there’s no way it’s worth it to swap us over to another build system, administer it, etc. Our team is small (two at the time, but hopefully doubling soon!), and there’s barely a dozen people in the whole engineering org. The next hit list item is to move from GH hosted builders to GCE workers to get a warmer docker cache (a bunch of our build time is spent pulling images that haven’t changed) - it will also save a chunk of change (GCE workers are 4x cheaper per minute and the caching will make for faster builds), but the opportunity cost for me tackling that is quite high.

▲

gmm1990 4 days ago | parent | next [-]

Ah interesting, I was just curious. I’ve wasted some time setting up ci runners stuff on bare metal servers just because I’ve heard runners from gitlab/github can be expensive

▲

fmbb 5 days ago | parent | prev [-]

Doesn’t the official docker build push action support caching with the GitHub Actions cache?

▲

a_t48 4 days ago | parent [-]

Yes but one image push for us is >10GB, due to ML dependencies. And even if it is intelligent and supports per layer caching, you can’t share between release branches - https://github.com/docker/build-push-action/issues/862.

And even if that did work, I’ve found it much more reliable to use the actual docker BuildX disk state than to try and get caching for complex multi stage builds working reliably. I have a case right now where there’s no combination of —cache-to/from flags that will give me a 100% cached rebuild starting from a fresh builder, using only remote cache. I should probably report it to the Docker team, but I don’t have a minimal repro right now and there’s a 10% chance it’s actually my fault.

	▲	kylegalbraith 4 days ago \| parent [-]
		You should try this with Depot [0]. I’m a founder of it and this is definitely one of the use cases we built it for. We persist your layer cache to a real NVMe device and reattach that automatically across builds. No more needing to save your layer cache via networks or resource constrained GitHub Actions cache. [0] https://depot.dev