>> We did have one nasty bug that showed up in late October (or was it early November?)

Having worked in Ecommerce & payment processing, where this weekend is treated like the Superbowl, birth of your first child and wedding day all rolled into one, a nasty POS bug at this time of year would be incredibly stressful!

You could have, in principle, implemented the new system to be able to run in "dummy mode" alongside the existing system at regular stores, so that you see that it produces the 'same' results in terms of what the existing system is able to provide.

Which is to say, there is more than one approach to gradual deployment.

> They are failing because of political scheming and bunch of people wanting to have a finger in the pie - "trillions spent" - I guess no one would mind earning couple millions.

Not (necessarily) wrong, but if you start small, Important People may not want to bother with something that is Unimportant and may leave things alone so something useful and working can get going. If you starting with an Important project then Important People will start circling it right away.

Political corruption is like environmental radiation: a viable fix is never 'just get rid of political corruption'*. It's an environmental constant that needs to be handled by an effective approach.

That said, parent's size- and scope-iterative approach also helps with corruption, because corruption metastasizes in the time between {specification} and {deliverable}.

Shrink that, by tying incremental payments to working systems at smaller scales, and you shrink the blast radius for failure.

That said, there are myriad other problems the approach creates (encouraging architectures that won't scale to the final system, promoting duct taped features on top of an existing system, vendor-to-vendor transitions if the system builder changes, etc).

But on the whole, the pros outweigh the cons... for projects controlled by a political process (either public or private).

That's why military procurement has essentially landed on spiral development (i.e. iterative demonstrated risk burn-down) as a meta-framework.

* Limit political corruption, to the extent possible in a cost efficient manner, sure

> There is no solution because these projects are not failing because of technical reasons.

There is no technical solution. There are systems and governance solutions, if the will is there to analyze and implement them.

> Gradual growth inevitably results in loads of technical debt.

Why is this stated as though it's some de facto software law? The argument is not whether it's possible to waterfall a massive software system. It clearly is possible, but the failure ratios have historically been sufficiently uncomfortable to give rise to entirely different (and evidently more successful) project development philosophies, especially when promoters were more sensitive to the massive sums involved (which in my opinion also helps explains why so many wasteful government examples). The lean startup did not appear in a vacuum. Do things that don't scale did not become a motto in these parts without reason. In case some are still confused about the historical purpose of these benign sounding advices, no, they weren't originally addressed at entrepreneurs aiming to run "lifestyle" businesses.

Software is a component of a product, if not the product itself. Treating software like a product, besides being the underlying truth, also means it makes sense to manage it like one.

Technical debt isn’t usually the problem people think it is. When it does become a problem, it’s best to think of it in product-like terms. Does it make the product less useful for its intended purpose? Does it make maintenance or repair inconvenient or costly? Or does it make it more difficult or even impossible to add competitive features or improvements? Taking a product evaluation approach to the question can help you figure out what the right response is. Sometimes it’s no response at all.

Designing or intending a system to be used at massive scale is not the same as building and deploying it so that it only initially runs at that massive scale.

That's just a recipe for disaster, "We don't even know if we can handle 100 users, let's now force 1 million people to use the system simultaneously." Even WhatsApp couldn't handle hundreds of millions of users on the day it was first released, nor did it attempt to. You build out slowly and make sure things work, at least if you're competent and sane.

> Gradual growth inevitably results in loads of technical debt that is not paid off as Product adds more feature requests to deliver larger and larger sales contracts.

This isn't technical debt, necessarily. Technical debt is a specific thing. You probably mean "an underlying design that doesn't perfectly map to what ended up being the requirements". But then the world moves on (what if a regulation is added that ruins your perfect structure anyway?) and you can't just wish for perfect requirements. Or not in software that interacts directly with the real world, anyway.

Yes, it can be very difficult to add “scale” after the fact, once you already have a lot of data persisted in a certain way.

You have to design for scale AND deploy gradually

we get paid to add to it, we don’t get paid to take away

There's nothing wrong with technical debt per se. As with all debt, the problem is incurring it without a plan or means to pay it off. Debt based financing is the engine of modern capitalism.

Gradual growth to large scale implies an ongoing refactoring cost--that's the price of paying off the technical debt that got you started and built initial success in small scale rollouts. As long as you keep "servicing" your debt (which can include throwing away an earlier chunk and building a more scalable replacement with the lessons learned), you're doing fine.

The magic words here to management/product owners is "we built it that way the first time because it got us running quickly and taught us what we need to know to build the scalable version. If we'd tried to go for the scalable version first, we wouldn't have known foo, bar and baz, and we'd have failed and wouldn't have learned anything."

Gradual growth =/= many tacked on features. Many tacked on features =/= technical debt. Technical debt =/= "everybody is afraid and frozen." Those are merely often correlated, but not required.

Whatsapp is a terrible example because it's barely a product; Whatsapp is mostly a free offering of goodwill riding on the back of actual products like Facebook Ads. A great example would be a product like Salesforce, SAP, or Microsoft Dynamics. Those products are forced to grow and change and adapt and scale, to massive numbers doing tons of work, all while being actual products and being software systems. I think such products act as stark rebukes of what you've described.

[dead]

This works for Clojure, git and even Linux. It seems there's a human who understands the entire system and decides what's allowed to be added to it. But these things are meant to be used by technical people.

The non-technical people I know might want to use Linux but stay on Windows or choose Mac OS because it's more straightforward. I use Windows+WSL at work even though I would like to use a native Linux distribution.

I know someone who created a MUD game (text online game) and said to him I wanted to make one with a browser client. He said something we could translate as "Good, you can have all the newbies." Not only was he right that a MUD should be played with a MUD client like tintin++, but making a good browser client is harder than it seems and that's time not spent making content for the game or improving the engine.

My point is that he was un uncomprimising person who refused adding layers to a project because they would come at a cost which isn't only time or dollars but also things like motivation and focus.

The uk government development service as you call it is not a service. It’s more of a declaration of process that is adhered to across diverse departments and organisations that make up the government. It’s usually small teams that are responsible for exploring what a service is or needs and then implementing it. They are able to deliver decent services because they start small, design and user test iteratively and only when there is a really good understanding of what’s being delivered do they scale out. The technology is the easy bit.

UK GDS is great, but the point there is that they're a crack team of project managers.

People complain about junior developers who pass a hiring screen and then can't write a single line of code. The equivalent exists for both project management and management in general, except it's much harder to spot in advance. Plus there's simply a lot of bad doctrine and "vibes management" going on.

("Vibes management": you give a prompt to your employees vaguely describing a desired outcome and then keep trying to correct it into what you actually wanted)

> and those systems go out to tens of millions from day 1

I like GDS (I even interviewed with them once and saw their dev process etc) but this isn't a great example. Technically GDS services have millions of users across decades, but people e.g. aren't constantly applying for new passports every day.

A much better example I think is Facebook's rollout of Messenger, which scaled to billions of actual users on day 1 with no issues. They did it by shipping the code early in the Facebook app, and getting it to send test messages to other apps until the infra held, and then they released Messenger after that. Great test strategy.

GDS's budget is about £90 million a year or something. There are many contracts that are still spent on digital, for example PA consulting for £60 million (over a few years) which do a lot of the gov.uk home-office stuff, and their fresh grads they hire cost more to the government than GDS's most senior staff...

SWIFT? Hold my beer. SWIFT did not launch anything substantial since its startup days in early 70-ies.

Moreover, their core tech did not evolve that far from that era, and the 70-ies tech bros are still there through their progeniture.

Here's an anecdote: The first messaging system built by SWIFT was text-based, somewhat similar to ASN.1.

The next one used XML, as it was the fad of the day. Unfortunately, neither SWIFT nor the banks could handle 2-3 orders of magnitude increase in payload size in their ancient systems. Yes, as engineers, you would think compressing XML would solve the problem and you would by right. Moreover, XML Infoset already existed, and it defined compression as a function of the XML Schema, so it was somewhat more deterministic even though not more efficient than LZMA.

But the suits decided differently. At one of the SIBOS conferences they abbreviate XML tags, and did it literally on paper and without thinking about back-and-forth translation, dupes, etc.

And this is how we landed with ISO20022 abberviations that we all know and love: Ccy for Currency, Pmt for Payment, Dt for Date, etc.

I don't think this applies in any way to companies contracted to build a massive system for a government with a clear need. Linus is talking about growing a greenfield open-source project, which may or may not ever be used by anyone.

In contrast, if your purpose is "we need to manage our country's accounting without pen and paper", that's a clear need for a massive system. Starting work on this by designing a system that can solve accounting for a small firm is not the right way to go. Instead, you have to design with the end-goal in mind, since that's what you were paid for. But, you don't launch your system to the entire country at once: you first use this system designed for a country in a small shop, to make sure it actually handles the small scale well, before gradually rolling out to more and more people.

No Linus Torvalds would stand against people in projects from article, he would slam the door and quit.

Those projects that author pointed out are basically political horror stories. I can imagine how dozens of people wanted to have a cut on money in those projects or wanted to push things because “they are important people”.

There is nothing you can do technically to save such projects and it is NOT an IT failure.

Works with implementations and not APIs though.

A bad API can constrain your implementation and often can't be changed once it's in use by loads of users. APIs should be right from day one if possible.

This is a really dense paragraph of lifetime-accumulated wisdom in that single quote.

I think you'll find that is exactly what people do. However, payroll solutions are highly customized for every individual company and even business unit. You don't buy a payroll software in a box, deploy it, and now you have payroll. Instead, you pay a payroll software company, they come in and get information about your payroll systems, and then they roll out their software on some of your systems and work with you to make sure their customizations worked etc. There's rarely any truly "off-the-shelf" software in B2B transactions, especially the type of end-user solutions that also interact with legal systems.

Also, governments are typically at least an order of magnitude larger than the largest companies operating in their countries, in terms of employees. So sure, the government of Liechtenstein has fewer employees than Google overall, but the US government certainly does not, and even Liechtenstein probably has way more government employees than Google employees in their country.

$100M maybe. But pretty much all tech needs an initial investment before you can start making profit. It takes a lot of development before you can get a product that anyone would want to pay for.

> We can build huge things like airliners, massive bridges and buildings without starting small.

We did start small with all of those things. We developed rigorous disciplines around engineering, architecture, material sciences. And people died along the way in the thousands[0][1]

People are still dying from those failures; The Boeing 737 MAX 9 crash was only two years ago.

> Incremental makes less sense to me when you want to go to mars.

This is yet another reason why a manned Mars mission will be exceedingly dangerous NOT a strike against incremental development and deployment.

[0] https://en.wikipedia.org/wiki/List_of_building_and_structure...

[1] https://en.wikipedia.org/wiki/List_of_accidents_and_incident...

All of the things you mentioned are designed and tested incrementally. Furthermore software has been used on Mars missions in the past, and that software was also developed incrementally. It’s proposed as the best way because it’s a way that has a track record

Counterpoint: the idea that your project will be the one to scale up to the millions of users/requests/etc is hubris. Odds are, your project won't scale past a scale of 10,000 to 100,000. Designing every project to scale to the millions from the beginning often leads to overengineering, adding needless complexity when a simpler solution would have worked better.

Naturally, that advice doesn't hold if you know ahead of time that the project is going to be deployed at massive scale. In which case, go ahead and implement your database replication, load balancing, and failover from the start. But if you're designing an app for internal use at your company of 500, well, feel free to just use SQLite as your database. You won't ever run into the problems of scale in this app, and single-file databases have unique advantages when your scale is small.

Basically: know when huge scale is likely, and when it's immensely UNlikely. Design accordingly.

You can by making a bigger and bigger rocket though.

Knowing that the rules for your first small deployment are not the same as the rules for everywhere, is valuable for designing well. Trying to implement all of those sets of rules in your initial deployment, is not a good idea. There is a general principle that you shouldn't code the abstraction until you've coded for the concrete example 2 or 3 times, because otherwise you won't make the right abstraction. Looking ahead is not the same as starting with the whole enchilada for your initial deployment.

But what hasn’t even been tried at a small scale definitely won’t work at a huge scale.

Which is absolutely true, and a reason to try at medium scale second. But what doesn't work at small scale, almost certainly won't work at huge scale.