IMO it was a worthwhile experiment that probably went too far before being rejected.

Having the maintenance/ops requirements directly feed back into the design and engineering of the product is genuinely powerful. Especially for certain kinds of products, if executed with discipline you can get really good engineering reaction time with no degradation of quality, and often with much cheaper infra costs.

The problem is what you said though. People with both skillsets are rare, and combining them is legitimately a third skill on its own. And even "both" skillsets is a simplification once you start accounting for data, security, compliance, large scaling DB work etc.

It was mostly a dead end for these reasons but it was worth exploring for its potential. And some genuinely good engineering techniques went mainstream out of it, not a total waste.

> Ops is a specialized skillset completely orthog[o]nal to the business of building applications.

Exactly this was the idea: persons in a DevOps role have skills in both areas, i.e. are very wide skill set, so that one can assign them to very diverse tasks for more flexibility. Additionally, these persons can implement synergies to make things more efficient in areas that intersect both development and operation.

it only really worked as part of a team.

Where it worked well is the "devops" or sre/sysadmin would sit in the same team and take away the rough edges of the infra.

conversly they would also keep an eye on the web people and see what they are doing and why, learning the tooling they use and try and rationalise it so they don't install 50 messaging queues.

yeah, I am a software developer, not a system administrator.

In the early days of devops it was primarily focused on supporting developer productivity (CI/CD) but quickly morphed into sysadmin once cloud computing took off.

At one point I was very hopeful that the progression from Infrastructure as a service to Platform as a service was going to simplify things. Instead we seem to have ended up with mind boggling complex services.

Imo a dev with no ops exp is like a chef who never tastes their own meals.

I disagree: ops is related to dev, because good dev work makes ops easy. But they are different: being good at ops requires a mindset of curiosity, risk management, and being willing to chase things to to last details.

The rigid process around agile is absolutely mind-blowing to me, and it seems like it's everywhere in corporate environments, but my current client dials it up to 11. Maddening.

It's not that crashes don't happen, it's just that they don't "randomly" happen. The error codes from bad hardware are predictable (against kernel bugs... well I use an "allnoconfig" kernel, which disables like 95% of the kernel). Bad hardware, bad network, bad RAM, all of these are simulateable in QEMU or Linux (tc netem / blkdebug).

If I count my test suite for the ISO, I have ~2500 unit tests (placement groups tests, Raft behaviour tests, disk perf tests, SigV4 crypto tests, DHCP tests, kernel rollback tests), ~110 fault-injection tests (latency, network, RAM), ~60 block-store failure tests (e.g. network interrupt while two disks try to sync), "chaos tests" (generating random event scenarios for an hour), bit flip tests, "fsync lies to me" tests, power cut tests, round-trip tests, "power cut while in the middle of sync between datacenters", SMART degradation tests, load tests, memory pressure tests, file descriptor resource starvation, NTP drift tests, metadata DB loss tests, DNS failover, ....

Then for security everything is turned off, kernel modules are off, eBPF is off, io_uring is off (lots of CVEs), user namespaces are off, most network and file drivers are off, /bin/sh is off, S3 endpoint is minimal, etc.

Most rules from aerospace are said to be "written with blood", some crash caused this rule to be legalized. But with ~40k lines you can go through each line and say "what if this fails". With a kubernetes cluster it's more or less based on hope and happy-path testing.

I also scraped other competitors for user complaints, esp. about data loss / sync engine bugs and in total I now have about 4000 tests for 40k lines of backend code and another ~1000 tests for the 15k lines for the sync engine code (about 1 test for every 15 lines).

I'm still not entirely confident (still need lots of beta testing), but a chance of data loss of .01% would be pretty high. It really depends on the amount of testing whether you need an on-call person at all and most orgs are more engineered towards "just hire someone to be on-call" rather than doing proper testing.

I know Google SREs who do feature development on the infrastructure support they support alongside their devs.

Why is your response so pithy? Google invented the term SRE, and I think how they use SRE is pretty relevant to the discussion.

It turns out it's easier to hire people who specialize in scalability, reliability, debugging of distributed complex systems, cost ops, efficiency gains, etc. than it is to instantly ensure every single engineer that works there is an expert in all those things, without anybody on staff who is an expert in them. Much like database skills, IT security skills, etc.

It's still something everyone can theoretically learn, of course, and someone who works in that role is most effective by instilling those disciplines in others as they go, and making everyone successful.

He's right you know.

A cultural issue that many organizations face. Same reason most large companies have an infosec department that runs separately from IT.

ZIRP goes back to 2010 actually, when interest rates dropped so much in an attempt to stop the great recession.

We should do it again but the right way this time.

Give literal zero interest loans from a state bank based on social value and economic priorities, just like they do in China. And set up an extreme progressive tax (90% top bracket) on returns that are paid out to owners to make sure everything gets invested back into production.

More investment is the driver of economic growth. All the bad parts of ZIRP came from letting companies use the money for unproductive activity. If it's used efficiently the downward pressure from higher product supply will be stronger than inflation from more money in the system. And you get more demand in the labor market leading to higher wages.

It wasn't COVID that kicked off ZIRP, it was the global financial crisis in 2008.

That’s what companies hire for. I’ve learned those skills and no one ever really wants them.

If you build tools with that context, they’ll have to replace you with someone that has that context. There’s few of them, and they’re expensive.

It’s easier to hire folks that should be interns or juniors that know that one tool, and then replace the whole department every decade.

It’s why everything is YAML now, despite it being horribly verbose and clunky. Can’t have our people needing to know how load balancing works, just add another 95 lines of YAML boilerplate so they don’t need to learn 5 lines of nginx config.

That's reinforced by how hiring is done though. I rarely if ever run into companies that hire for fundamentals. Only the Google tier of companies do that.

You have no idea what you're talking about if that's what you think SREs are.