The lesson from “big software projects are still failing” isn’t that we need better methodologies, better project management, or stricter controls. The lesson is "don't do big software projects".

Software is not the same as building in the physical world where we get economies of scale.

Building 1,000 bridges will make the cost of the next incremental bridge cheaper due to a zillion factors, even if Bridge #1 is built from sticks (we'll learn standards, stable, fundamental engineering principles, predicable failure modes, etc.) we'll eventually reach a stable, repeatable, scalable approach to build bridges. They will very rarely (in modernity) catastrophically fail (yes, Tacoma Narrows happened but in properly functioning societies it's rare.)

Nobody will say "I want to build a bridge upside-down, out of paper clips and can withstand a 747 driving over it". Because that's physically impossible. But nothing's impossible in software.

Software isn't scalable in this way. It's not scalable because it doesn't have hard constraints (like the laws of physics) - so anything goes and can be in scope; and since writing and integrating large amounts of code is a communication exercise, suffers from diseconomies of scale.

Customers want the software to do exactly what they want and - within reason - no laws of physics are violated if you move a button or implement some business process.

Because everyone wants to keep working the way they want to work, no software project (even if it sounds the same) is the same. Your company's bespoke accounting software will be different than mine, even if we are direct competitors in the same market. Our business processes are different, org structures are different, sales processes are different, etc.. So they all build different accounting software, even if the fundamentals (GaaP, double-entry bookkeeping, etc.) are shared.

It's also the same reason why enterprise software sucks - do you think that a startup building expense management starts off being a giant mess of garbage? No! IT starts off simple and clean and beautiful because their initial customer base (startups) are beggars and cannot be choosers, so they adapt their process to the tool. But then larger companies come along with dissimilar requirements and, Expense Management SaaS Co. wins that deal by changing the product to work with whatever oddball requirements they have, and so on, until the product essentially is a bunch of config options and workflows that you have to build yourself.

(Interestingly, I think these products become asymptotically stuck - any feature you add or remove will make some of your customers happy and some of your customers mad, so the product can never get "better" globally).

We can have all the retrospectives and learnings we want but the goal - "Build big software" - is intractable, and as long as we keep trying to do that, we will inevitably fail. This is not a systems problem that we can fix.

The lesson is: "never build big software".

(Small software is stuff like Bezos' two pizza team w/APIs etc. - many small things make a big thing)

I agree with you on "don't do big software project" Specially do not fast scale them out to hundreds of people. You have to scale them more organically ensuring that every person added is a net gain. They think that adding more people will reduce the time.

I am surprised on the lack of creativity when doing these projects. Why don't they start 5 small projects building the same thing and let them work for a year. At the end of the year you cancel one of the projects, increasing the funding in the other four. You can do that every year based on the results. It may look like a waste but it will significantly increase your chances of succeeding.

>Building 1,000 bridges will make the cost of the next incremental bridge cheaper due to a zillion factors, even if Bridge #1 is built from sticks (we'll learn standards, stable, fundamental engineering principles, predicable failure modes, etc.) we'll eventually reach a stable, repeatable, scalable approach to build bridges. They will very rarely (in modernity) catastrophically fail (yes, Tacoma Narrows happened but in properly functioning societies it's rare.)

Build 1000 JSON parsers and tell me if the next one isn't cheaper to develop with "(we'll learn standards, stable, fundamental engineering principles, predicable failure modes, etc.)"

>Software isn't scalable in this way. It's not scalable because it doesn't have hard constraints (like the laws of physics)

Uh, maybe fewer but none is way to far. Get 2 billion integer operations per second out of a 286, the 500 mile email, big data storage, etc. Physical limits are everywhere.

>It's also the same reason why enterprise software sucks.

The reason enterprise software sucks is because the lack of introspection and learning from the garbage that went before.

You have to be able to turn away unsuitable customers.