Yeah, that's a good rule. Except, do you want to build Debug or Release? Or maybe RelWithDebugInfo? And do you want that with sanitizers maybe? And what the sanitizers' options should be? Do you want to compile your tests too, if you want to run them later on a different machine? And what about that dependency that takes two hours to compile, maybe you just want to reuse the previous compilation of it? And if so, where to take that from? Etc. etc.

Before long, you need another script that will output the train of options to your `build.sh`.

(If Fortune 500 companies can do a one-line build with zero parameters, I suspect I'd be very bored there.)

You still inevitably need a bunch of CI platform-specific bullshit for determining "is this a pull request? which branch am I running on?", etc. Depending on what you're trying to do and what tools you're working with, you may need such logic both in an accursed YAML DSL and in your build script.

And if you want your CI jobs to do things like report cute little statuses, integrate with your source forge's static analysis results viewer, or block PRs, you have to integrate with the forge at a deeper level.

There aren't good tools today for translating between the environment variables or other things that various CI platforms expose, managing secrets (if you use CI to deploy things) that are exposed in platform-specific ways, etc.

If all you're doing with CI is spitting out some binaries, sure, I guess. But if you actually ask developers what they want out of CI, it's typically more than that.

The other rule is that script should run as a user. Solely on that working directory.

There are too many scripts like that which start, ask for sudo and then it's off to implementing someones "great idea" about your systems network interfaces.

You’re not wrong but your suggestion also throws away a lot of major benefits of CI. I agree jobs should be one liners but we still need more than one…

The single job pipeline doesn’t tell you what failed. It doesn’t parallelize unit and integration test suites while dealing with the combinatorial matrix of build type, target device, etc.

At some point, a few CI runners become more powerful than a developer’s workstation. Parallelization can really matter for reducing CI times.

I’d argue the root of the problem is that we are stuck on using “make” and scripts for local build automation.

We need something descriptive enough to describe a meaningful CI pipeline but also allow local execution.

Sure, one can develop a bespoke solution, but reinventing the wheel each time gets tiring and eventually becomes a sizable time sink.

In principle, we should be able to execute pieces of .gitlab-ci.yml locally, but even that becomes non trivial with all the nonstandard YAML behaviors done in gitlab, not to mention the varied executor types.

Instead we have a CI workflow and a local workflow and hope the two are manually kept in sync.

In some sense, the current CI-only automation tools shouldn’t even need to exist (gitlab, Jenkins, etc) — why didn’t we just use a cron job running “build.sh” ?

I argue these tools should mainly only have to focus on the “reporting/artifacts” with the pipeline execution parts handled elsewhere (or also locally for a developer).

Shame on you GitLab!

How do you get caching of build steps with this approach? Or do you just not?

Sounds like the Lotus philosophy, "simplify and add lightness".

[dead]

Agreed with this sentiment, but with one minor modification: use a Makefile instead. Recipes are still chunks of shell, and they don’t need to produce or consume any files if you want to keep it all task-based. You get tab-completion, parallelism, a DAG, and the ability to start anywhere on the task graph that you want.

It’s possible to do all of this with a pure shell script, but then you’re probably reimplementing some or all of the list above.

There are various things that can be a reasonable candidate for the "top level" build entrypoint, including Nix, bazel, docker bake, and probably more I'm not thinking of. They all have an entrypoint that doesn't have a ton of flags or nonsense, and operate in a pretty self contained environment that they set up and manage themselves.

Overall I'm not a fan of wrapping things; if there are flags or options on the top-level build tool, I'd rather my devs explore those and get used to what they are and can do, rather than being reliant on a project-specific script or make target to just magically do the thing.

Anyway, other than calling the build tool, CI config can have other steps in it, but it should be mostly consumed with CI-specific add-ons, like auth (OIDC handshake), capturing logs, uploading artifacts, sending a slack notification, whatever it is.

I tried to drive this approach at a previous job but nobody else on the team cared so I ended up always having to mirror all the latest build changes into my bash script.

The reason it didn't catch on? Everyone else was running local builds in a proprietary IDE, so to them the local build was never the same anyway.

I always use, no matter what I am using underneath, a bootstrap script, a configure script and a build step.

That keeps the cli interface easy, expectable and guessable.

This sounds a lot like "use Nix".

I haven't used Datomic, but you're right that the part that requires over there is "single consistent source of truth." There's only ever a single node that is sequencing all writes. Perhaps as a result of this, it provides strong [verified ACID guarantees][1].

What I got from Hickey's talk is that he wanted to design a system that resisted the urge to encode everything in a stored procedure and run it on the database server.

[1]: https://jepsen.io/analyses/datomic-pro-1.0.7075

Alternative take: CI is the successful monetization of Make-as-a-Service.

No, you keep your build system declarative, but you support a clean plugin API that permits injection into the build lifecycle and allow configuring/invoking the plugin with your DSL.

How does that script handle pushing to ghcr, or pulling an artifact from a previous stage for testing?

In my experience these are the bits that fail all the time, and are the most important parts of CI once you go beyond it taking 20/30 seconds to build.

A clean build in an ephemeral VM of my project would take about 6 hours on a 16 core machine with 64GB RAM.

You must be very lucky to be in a position where you know what needs to be done before the run begins. Not everyone is in that position.

At my place, we have ~400 wall hours of testing, and my run begins by figuring out what tests should be running and what can be skipped. This depends on many factors, and the calculation of the plan already involves talking to many external systems. Once we have figured out a plan for the tests, we can understand the plan for the build. Only then we can build, and test afterwards. I haven't been able to express all of that in "a bit of yaml" so far.

Are you not worried about parallelisation in your case? Or have you solved that in another way (one big beefy build machine maybe?)