It reads a lot like the Crowdstrike SNAFU. Machine-generated configuration file b0rks-up the software that consumes it.

The "...was then propagated to all the machines that make up our network..." followed by "....caused the software to fail." screams for a phased rollout / rollback methodology. I get that "...it’s critical that it is rolled out frequently and rapidly as bad actors change their tactics quickly" but today's outage highlights that rapid deployment isn't all upside.

The remediation section doesn't give me any sense that phased deployment, acceptance testing, and rapid rollback are part of the planned remediation strategy.

▲

tptacek 3 hours ago | parent | next [-]

I don't think this system is best thought of as "deployment" in the sense of CI/CD; it's a control channel for a distributed bot detection system that (apparently) happens to be actuated by published config files (it has a consul-template vibe to it, though I don't know if that's what it is).

▲

EvanAnderson 3 hours ago | parent | next [-]

That's why I likened it Crowdstrike. It's a signature database that blew up the consumer of said database. (You probably caught my post mid-edit, too. You may be replying to the snarky paragraph I felt better of and removed.)

Edit: Similar to Crowdstrike, the bot detector should have fallen-back to its last-known-good signature database after panicking, instead of just continuing to panic.

▲

eastdakota 3 hours ago | parent | prev [-]

That’s correct.

▲

tptacek 3 hours ago | parent [-]

Is it actually consul-template? (I have post-consul-template stress disorder).

▲

threatofrain 29 minutes ago | parent | next [-]

I'd love to hear any commentary on Consul if anyone else has it.

	▲	tptacek 21 minutes ago \| parent [-]
		I think Consul is great, for what it's worth; we were just abusing it. https://fly.io/blog/a-foolish-consistency/ https://fly.io/blog/corrosion/

▲

mh- 25 minutes ago | parent | prev [-]

Did you know: PCTSD affects more than 2 in 5 engineers.

▲

navigate8310 3 hours ago | parent | prev | next [-]

I'm amazed that they are not using any simulator of some sort and pushing changes directly to production.

▲

Aeolun 3 hours ago | parent | prev [-]

I’m fairly certain it will be after they read this thread. It doesn’t feel like they don’t want, or are incapable of improving?