I really like the SourceHut CI, because:

1. When the build fails, you can SSH into the machine and debug it from there.

2. You can super easily edit & run the manifest without having to push to a branch at all. That makes it super easy to even try a minimum reproducible example on the remote machine.

Other than that, self-hosting (with Github or preferrably Forgejo) makes it easy to debug on the machine, but then you have to self-host.

▲

Nextgrid 6 hours ago | parent [-]

Self-hosted runners with Github is a whole world of pain because it literally just runs commands on the host and does not handle provisioning/cleanup, meaning you need to make sure your `docker run` commands don't leave any leftover state that can mess up future/concurrent builds. It doesn't even handle concurrency by itself, so you have to run multiple instances of the runner.

▲

watermelon0 2 hours ago | parent | next [-]

Github supports ephemeral runners which are limited to a single job.

You can use `workflow_job` webhook to be notified of a new job, after that you need to call `generate-jitconfig` API to get a just-in-time configuration token, and then you can start a Github runner in ephemeral mode with the JIT token.

This allows you to orchestrate Docker containers, KVM instances, etc., which are used for a single time, and then destroyed.

There are some open source projects, such as using ephemeral Kubernetes pods with the ephemeral runners.

▲

palata 6 hours ago | parent | prev [-]

That was included in my "but then you have to self-host" :-).

As I said, I really like the SourceHut CI.

▲

jcgl 2 hours ago | parent [-]

Do you have any experience self-hosting SourceHut? I’d really like to do so, but I get weak knees every time I look at the docs for it.

	▲	palata 2 hours ago \| parent [-]
		I don't, but I would be curious. But I'm happy to contribute (money) to SourceHut, they're doing a good job.