I study and write quite a bit of tech history. IMHO from what I've learned over the last few years of this hobby, the primary issue is quite simple. While hardware folks study and learn from the successes and failures of past hardware, software folks do not. People do not regularly pull apart old systems for learning. Typically, software folks build new and every generation of software developers must relearn the same problems.

I work at $FANG, every one of our org's big projects go off the rails at the end of the project and there's always a mad rush at the end to push developers to solve all the failures of project management in their off hours before the arbitrary deadline arrives.

After every single project, the org comes together to do a retrospective and ask "What can devs do differently next time to keep this from happening again". People leading the project take no action items, management doesn't hold themselves accountable at all, nor product for late changing requirements. And so, the cycle repeats next time.

I led and effort one time, after a big bug made it to production after one of those crunches that painted the picture of the root cause being a huge complicated project being handed off to offshore junior devs with no supervision, and then the junior devs managing it being completely switched twice in the 8 month project with no handover, nor introspection by leadership. My manager's manager killed the document and wouldn't allow publication until I removed any action items that would constrain management.

And thus, the cycle continues to repeat, balanced on the backs of developers.

I've also considered a side-effect of this. Each generation of software engineers learns to operate on top of the stack of tech that came before them. This becomes their new operating floor. The generations before, when faced with a problem, would have generally achieved a solution "lower" down in the stack (or at their present baseline). But the generations today and in the future will seek to solve the problems they face on top of that base floor because they simply don't understand it.

This leads to higher and higher towers of abstraction that eat up resources while providing little more functionality than if it was solved lower down. This has been further enabled by a long history of rapidly increasing compute capability and vastly increasing memory and storage sizes. Because they are only interacting with these older parts of their systems at the interface level they often don't know that problems were solved years prior, or are capable of being solved efficiently.

I'm starting to see ideas that will probably form into entire pieces of software "written" on top of AI models as the new floor. Where the model basically handles all of the mainline computation, control flow, and business logic. What would have required a dozen Mhz and 4MB of RAM to run now requires TFlops and Gigabytes -- and being built from a fresh start again will fail to learn from any of the lessons learned when it was done 30 years ago and 30 layers down.

> While hardware folks study and learn from the successes and failures of past hardware, software folks do not

I've been managing, designing, building and implementing ERP type software for a long time and in my opinion the issue is typically not the software or tools.

The primary issue I see is lack of qualified people managing large/complex projects because it's a rare skill. To be successful requires lots of experience and the right personality (i.e. low ego, not a person that just enjoys being in charge but rather a problem solver that is constantly seeking a better understanding).

People without the proper experience won't see the landscape in front of them. They will see a nice little walking trail over some hilly terrain that extends for about a few miles.

In reality, it's more like the Fellowship of the Rings trying to make it to Mt Doom, but that realization happens slowly.

I think part of it is that reading code isn't a skill that most people are taught.

When I was in grad school ages ago, my advisor told me to spend a week reading the source code of the system we were working with (TinyOS), and come back to him when I thought I understood enough to make changes and improvements. I also had a copy of the Linux Core Kernel with Commentary that I perused from time to time.

Being able to dive into an unknown codebase and make sense of where the pieces are put together is a very useful skill that too many people just don't have.

I have a theory that the churn in technology is by design. If a new paradigm, new language, new framework comes out every so many years, it allows the tech sector to always want to hire new graduates for lower salaries. It gives a thin veneer of we want to always hire the person who has X when really they just do not want to hire someone with 10 years of experience in tech but who may not have picked up X yet.

I do not think it is the only reason. The world is complex, but I do think it factors into why software is not treated like other engineering fields.

I think this is too simple. First of all, hardware people have high incentive to fully replace components and systems for many reasons. Replacement is also the only way they can fix major design mistakes. Software people constantly do fix bugs and design mistakes. There is certainly no strong culture to document or dig up former mistakes made, but it's not like they don't learn from mistakes, it's just a constant process. In contrast to hardware, there is usually no point in time to retrospect. The incentives to rejuvenate systems are low and if considered often seem expensive. Software engineers self motivation is often ill-minded, new devs feeling uncomfortable with the existing system and calling for something "modern". But if the time comes to replace the "legacy" systems, then you are right, no one looks back at the former mistakes and the devs that know them, are probably long gone. The question is whether we should ever replace an software system or focus more on gradual and active modernization. But the latter can be very hard, in hardware everything is defined, most of the time backed by standards, in software we usually don't have that, so complex interconnected systems rarely have sane upgrade paths.

I would boil this down to something else, but possibly related: project requirements are hard. That's it.

> While hardware folks study and learn from the successes and failures of past hardware, software folks do not. People do not regularly pull apart old systems for learning.

For most IT projects, software folks generally can NOT "pull apart" old systems, even if they wanted to.

> Typically, software folks build new and every generation of software developers must relearn the same problems.

Project management has gotten way better today than it was 20 years, so there is definitely some learnings that have been passed on.

In my experience, a lot of the time the people who COULD be solving these issues are people who used to code or never have. The actual engineers who might do something like this aren't given authority or scope and you have MBAs or scrum masters in the way of actually solving problems.

"While hardware folks study and learn from the successes and failures of past hardware, software folks do not." Couldn't be further from the truth. Software folks are obsessed with copying what has been shown to work to the point that any advance quickly becomes a cargo cult (see microservices for example).

Once you've worked in both hardware and software engineering you quickly realize that they only superficially similar. Software is fundamentally philosophy, not physics.

Hardware is constrained by real world limitations. Software isn't except in the most extreme cases. Result is that there is not a 'right' way to do any one thing that everyone can converge on. The first airplane wing looks a whole lot like a wing made today, not because the people that designed it are "real engineers" or any such BS, but because that's what nature allows you to do.

IME, "Why systems fail" almost always boils down to a principal-agent problem. This is another way of expressing the Mungerism "show me the incentive, I'll show you the outcome".

Systems that "work" tend to have some way of correcting for or mitigating the principal agent problem by aligning incentives.

I'd also point out that hardware is a much older discipline, in terms of how long it's been operating at scale. It's had more time to formalize and crystallize. Intel is 56 years old. Google is 27.

Most of the time, there's no need to study anything. Any experienced software engineer can tell you about a project they worked on with no real requirements, management constantly changing their mind, etc.

How do you study software history? Most of the lessons seem forever locked away behind corporate walls - any honest assessments made public will either end careers or start lawsuits

This is an interesting distinction, but it ignores the reasons software engineers do that.

First, hardware engineers are dealing with the same laws of physics every time. Materials have known properties etc.

Software: there are few laws of physics (mostly performance and asymptotic complexity). Most software isnt anywhere near those boundaries so you get to pretend they dont exist. If you get to invent your own physics each time, yeah the process is going to look very different.

Some consequences of NOT learning from prior successes and failures: (a) no more training for the next generation of developers/engineers (b) fighting for the best developers, and this manifests in leetcode grinding (c) decrease in cooperation among team mates, etc.

This is one part of the issue. The other major piece of this that I've seen over more than two decades in industry is that most large projects are started by and run by (but not necessarily the same person) non-technical people who are exercising political power, rather than by technical people who can achieve the desired outcomes. When you put the nexus of power into the hands of non-technical people in a technical endeavor you end up with outcomes that don't match expectations. Larger scale projects deeply suffering from "not knowing what we don't know" at the top.

I think there is a ton more nuance, but can still be explained by a simple observation, which TFA hints at: "It's the economics, stupid."

Engineering is the intersection of applied sciences, economics and business. The economics aspect is almost never recognized and explains many things. Projects of other disciplines have significantly higher costs and risks, which is why they require a lot more rigor. Taking hardware as example, one bad design decision can sink the entire company.

On the other hand, software has economics that span a much more diverse range than any other field. Consider:

- The capital costs are extremely low.

- Development can be extremely fast at the task level.

- Software, once produced, can be scaled almost limitlessly for very cheap almost instantly.

- The technology moves extremely fast. Most other engineering disciplines have not fundamentally changed in decades.

- The technology is infinitely flexible. Software for one thing can very easily be extended for an adjacent business need.

- The risks are often very low, but can be very high at the upper end. The rigor applied scales accordingly. Your LoB CRUD app going down might bother a handful of people, so who cares about tests? But your flight control software better be (and is) tested to hell and back.

- Projects vary drastically in stacks, scopes and risk profiles, but the talent pool is more or less common. This makes engineering culture absolutely critical because hiring is such a crapshoot.

- Extreme flexibility also masks the fact that complexity compounds very quickly. Abstractions enable elegant higher-level designs, but they mask internal details that almost always leak and introduce minor issues that cause compounding complexity.

- The business rules that software automates are extremely messy to begin with (80K payroll rules!) However, the combination of a) flexibility, b) speed, and c) scalability engender a false sense of confidence. Often no attempt is made at all to simplify business requirements, which is probably where the biggest wins hide. This is also what enables requirements to shift all the time, a prime cause for failures.

Worse, technical and business complexity can compound. E.g. its very easy to think "80K payroll rules linearly means O(80K) software modules" and not "wait, maybe those 80K payroll rules interact with each other, so it's probably a super-linear growth in complexity." Your architecture is then oriented towards the simplistic view, and needs hacks when business reality inevitably hits, which then start compounding complexity in the codebase.

And of course, if that's a contract up for bidding, your bid is going to be unsustainably low, which will be further depressed by the competitive bidding process.

If the true costs of a project -- which include human costs to the end users -- are not correctly evaluated, the design and rigor applied will be correspondingly out of whack.

As such I think most failures, in addition to regular old human issues like corruption, can be attributed to an insufficient appreciation of the economics involved, driven primarily by overindexing on the powers of software without an appreciation of the pitfalls.

As someone who's learning programming right now, do you have any suggestions on how one would go about finding and studying these successes and failures?

There are rational explanations for this. When software fails catastrophically, people almost never die (considering how much software crashes every day). When hardware fails catastrophically, people tend to die, or lose a lot of money.

There's also the complexity gap. I don't think giving someone access to the Internet Explorer codebase is necessarily going to help them build a better browser. With millions of moving parts it's impossible to tell what is essential, superfluous, high quality, low quality. Fully understanding that prior art would be a years long endeavor, with many insights no doubt, but dubious.

Agree 100%.

I know a lot of people on here will disagree with me saying this but this is exactly how you get an ecosystem like javascript being as fragmented, insecure, and "trend prone" as the old school Wordpress days. It's the same problems over and over and every new "generation" of programmers has to relearn the lessons of old.

I think this is a downstream of effect of there being no real regulation or professional designations in software which leads to every company and team being wildly different leading to no standards leaving no time for anything but crunching since there are no barriers restricting your time, so nobody spends time doing much besides shipping constantly.

Software just feels so much more ephemeral than hardware. I haven't yet met a single 'old software enthusiast' in my life, yet there are so many enthusiasts for older hardware.

I was so annoyed when I found out the OTP library and realized we’ve been reinventing things for 20+ years

Yes, and it's because there aren't very many textbook ways to do software engineering, because it's evolving too fast to reach consensus.

I’ve read one tech history book and I really enjoyed it. any you recommend?

Indeed.

That's why we see every now and then "new" programming paradigms which were once obsolete.

... are you saying that hardware projects fail less than software ones? just building a bridge is something that fails on a regular occurence all over the world. Every chip comes with list of erratas longer than my arm.

Software folks treat their output as if it's their baby or their art projects.

Hardware folks just follow best practices and physics.

They're different problem spaces though, and having done both I think HW is much simpler and easier to get right. SW is often similar if you're working on a driver or some low-level piece of code. I tried to stay in systems software throughout my career for this reason. I like doing things 'right' and don't have much need to prove to anyone how clever I am.

I've met many SW folks who insist on thinking of themselves as rock stars. I don't think I've ever met a HW engineer with that attitude.