Wanna DRY out your github actions yaml?

Generate it from Dhall, or cue, or python, or some real language that supports actual abstractions.

If your problem is you want to DRY out yaml, and you use more yaml features to do it, you now have more problems, not fewer.

▲

phanimahesh 8 hours ago | parent | next [-]

This. So true. Yaml has always been an overly complicated format, with weird quirks ( like norway becoming false in a list of country codes ).

I find it an absolute shame that languages like Dhall did not become more popular earlier. Now everything in devops is yaml, and I think many developers pick yaml configs not out of good reasons but defaulting to its ubiquity as sufficient.

▲

true_blue 8 hours ago | parent | next [-]

>norway

yaml 1.2 was released in 2009, and it fixed this problem. this is an implementation issue.

https://yaml.org/spec/1.2.2/#12-yaml-history

▲

TheDong 7 hours ago | parent | next [-]

If your data format is so complicated that all commonly used implementations are not compliant with your spec, maybe it's a problem with the data-format.

Every single implementation people actually use seems to be a messy mix of yaml 1.1 and 1.2....

Maybe if the yaml project wants to consider this fixed, they should have written some correct reference parsers themselves for any languages in need, and encouraged their use.

	▲	tkfu 7 hours ago \| parent [-]
		I noted this in reply to the comment above, but: the YAML 1.2 spec doesn't actually mandate that parsers use the Core Schema. They left it as a recommendation. So I don't consider it to be "fixed" at all.

▲

tkfu 7 hours ago | parent | prev [-]

I would not say it "fixed" the problem. It removed the _recommendation_ for parser implementations to use the regex `y|Y|yes|Yes|YES|n|N|no|No|NO|true|True|TRUE|false|False|FALSE|on|On|ON|off|Off|OFF` for parsing scalars as bools, it changed the _canonical_ presentation of bools from `y|n` to `true|false`, and it introduced the "schema" concept. It also introduced and recommended the use of the Core Schema, which uses `true|True|TRUE|false|False|FALSE` as the regex for inferring bools from scalars. But unsurprisingly, since using this schema is only a recommendation and not a requirement, many implementations elected to keep their backwards-compatible implementations that use the original regex.

So the Norway problem persists.

	▲	true_blue 5 hours ago \| parent [-]
		the recommendation was what caused the norway problem. it now strongly recommends not to do this, and it says that a Yaml parser should use the core schema unless instructed otherwise. going against the recommendation while saying that you're yaml 1.2 compliant feels like an issue that should be raised with the parser to me. I've never run into this issue in practice though. is there a parser that says that it's Yaml 1.2 compliant that uses that regex? I don't know of one.

▲

theamk 7 hours ago | parent | prev [-]

Why introduce all-new langugage, like Dhall, just for configuration? This seems like a total waste of time. And you still need to use "real" language (or bash) to write glue to connect to github configuration.

The config generators are very simple, and should to be written in whatever language your developers already know - which likely means Python or Javascript or Go.

▲

nirvdrum 8 hours ago | parent | prev | next [-]

Having used CI systems and application frameworks that support YAML anchors for configuration, adding in a programming language would be a massive amount of complexity for very little gain. We're not talking about dozens of locations with hundreds of lines of shared code.

Asking the team to add a new build dependency, learn a new language, and add a new build step would create considerably more problems, not fewer. Used sparingly and as needed, YAML anchors are quite easy to read. A good editor will even allow you to jump to the source definition just as it would any other variable.

Being self-contained without any additional dependencies is a huge advantage, particularly for open source projects, IMHO. I'd wager very few people are going to learn Dhall in order to fix an issue with an open source project's CI.

▲

baq 8 hours ago | parent | next [-]

Your team doesn't know YAML, it knows github actions. There's zero transferable knowledge when switching from github actions to kubernetes deployments, as there is precisely the same zero correlation between kubernetes and ansible configs. 'It's all YAML' is a lie and I'm continuously surprised so many people are falling for it for so long. YAML is the code-as-data, but the interpreter determines what it all means.

▲

nirvdrum 7 hours ago | parent [-]

Oh, for goodness' sake. We know YAML syntax and that's the only part that's relevant here. Pointing out that different software uses different keys for their configuration or even takes different actions for keys that happen to share the same name isn't particularly insightful. We haven't been bamboozled.

▲

baq 7 hours ago | parent [-]

I don’t understand even more now. If you freely admit they’re different languages, the only reason to keep using the stupid deficient syntax is momentum, and while it isn’t a bad reason, it is costing you and everyone else in the long run.

▲

nirvdrum 7 hours ago | parent [-]

Huh? I'm using YAML because that's the language used to configure GitHub Actions. You may not like YAML, and that's fine. But if we collectively had to learn the unique way each project generates their GitHub Actions config, that would be a massive waste of time.

YAML isn't that hard. Most GitHub Actions configs I see are well under 500 lines; they're not crumbling under the weight of complexity.

	▲	baq 6 hours ago \| parent [-]
		Assembly isn't hard either and yet almost nobody is writing it anymore, for a reason, just as nobody (to an epsilon) is writing jvm opcodes directly. Somehow the industry decided assembly is actually fine when doing workflows. I'm saying GHA should use a proper programming language instead of assembly.

▲

WorldMaker 7 hours ago | parent | prev | next [-]

Use the language you are already working in? Most languages have good YAML serialization and I think in most languages a function call taking a couple parameters that vary to produce slightly different but related objects is going to be as readable or more readable than YAML anchors.

▲

nirvdrum 7 hours ago | parent [-]

That would be better, but it's an option I already have available to me and it's just not attractive. AFAIK, GitHub Actions requires the config files to be committed. So, now I need to guard against someone making local modifications to a generated file. It's doable of course, but by the time I've set all this up, it would have been much easier for everyone to copy and paste the six lines of code in the three places they're needed. YAML anchors solve that problem without really creating any new ones.

If generating your GitHub Actions config from a programming language works for you, fantastic. I'm just happy we now have another (IMHO, attractive) option.

▲

WorldMaker 5 hours ago | parent [-]

Most of the debate here is that a lot of us don't find YAML anchors attractive. It can be one of the papercuts of using YAML.

I mostly agree with the article that with GitHub Actions specifically, I try to refactor things to the top-level "workflow" level first, and then yeah resort to copy and paste in most other cases.

I'm a little less adamant that GitHub should remove anchor support again than the original poster, but I do sympathize greatly, having had to debug some CircleCI YAML and Helm charts making heavy use of YAML anchors. CircleCI's YAML is so bad I have explored options to build it with a build process. Yeah, it does create new problems and none of those explorations got far enough to really improve the process, but one of the pushes to explore them was certainly that YAML anchors are a mess to debug, especially when you've got some other tool concatenating YAML files together and can result in anchor conflicts (and also other parts of the same YAML that depend on a particular form of how anchor conflicts overwrite each other, oof). I don't see GitHub Actions necessarily getting that bad just by enabling anchors, but I have seen enough of where anchors become a crutch and a problem.

	▲	nirvdrum 5 hours ago \| parent [-]
		That's fair. And I'm not arguing that YAML anchors can never be a problem. I am saying that layering in a whole custom build system to handle a 250 line ci.yml file is not the trade-off I'd make. What I'd hazard to say most teams do in that situation is duplicate config, which is not without its own problems. I think YAML anchors is a fine solution for these cases and don't think they'll lead to total chaos. Alas, not all config options can be hoisted to a higher level and I'm trusting a team has explored that option when it's available. If you're dealing with 10s of files that are 1000s of lines long, then YAML anchors may very well not be the ideal option. Having the choice lets each team find what works best for them.

▲

theamk 7 hours ago | parent | prev [-]

> not talking about dozens of locations with hundreds of lines of shared code.

:) :) :)

.github/workflows in my current project: 33 files, 3913 lines total, 1588 lines unique.

(and this was _after_ we moved all we can into custom actions and sub-workflows)

	▲	nirvdrum 5 hours ago \| parent [-]
		Ouch. That sounds terrible with or without YAML anchors. GitHub Actions has overall been a great addition, allowing projects to integrate CI directly into their PR process. But, I never understood why it didn't have simpler paths for the very common use cases of CI and CD. Virtually any other dedicated CI product is considerably easier to bootstrap.

▲

usrme 8 hours ago | parent | prev | next [-]

Seconded. I've had huge success with generating workflows with CUE. Would definitely recommend it to anyone struggling with YAML.

▲

anttiharju 7 hours ago | parent | prev | next [-]

Or just use composite actions, it's not 2020 anymore.

Templating GitHub Actions is very powerful (I've worked with such a setup) but it has its own headaches and if you don't _need_ custom tooling better to not have it.

I can wish for improvements on the native setup without reaching out for the sledgehammer.

	▲	anttiharju 7 hours ago \| parent [-]
		I think most of the pain with GitHub Actions goes away if you use actionlint, action-validator, prettier/yamlfmt in a single ci job to validate your setup. Can even add them as git hooks that automatically stage changes and give quick feedback when iterating.

▲

esafak 7 hours ago | parent | prev | next [-]

If we're going there let's just stay there and not translate it back to YAML, which is absolutely inappropriate for specifying CI pipelines.

▲

Spivak 8 hours ago | parent | prev [-]

Just introduce a templating language and the CI to generate the yaml as part of your pipeline as way to make it simpler?

Above a certain level of complexity, sure. But having nothing in between is an annoying state of affairs. I use anchors in Gitlab pipelines and I hardly curse their names.