> you're trying to twist reality to fit a premise that is just impossible to make true: that estimates of how long it takes to build software are reliable.

It's not binary, it's a continuum.

With experience, it's possible to identify whether the new project or set of tasks is very similar to work done previously (possibly many times) or if it has substantial new territory with many unknowns.

The more similarity to past work, the higher the chance that reasonably accurate estimates can be created. More tasks in new territory increases unknowns and decreases estimate accuracy. Some people work in areas where new projects frequently are similar to previous projects, some people work in areas where that is not the case. I've worked in both.

Paying close attention to the patterns over the years and decades helps to improve the mapping of situation to estimate.

Software estimates for projects that don't involve significant technical risk can be made reliable, with sufficient discipline. Not all teams have that level of discipline but I've seen existence proofs of it working well and consistently.

If you can't make firm delivery commitments to customers then they'll find someone who can. Losing customers, or not signing them in the first place, is the most harmful thing to everyone in the organization. Some engineers are oddly reluctant to accept that reality.

There is an enterprise methodology that increases precision of project estimation.

1. Guess the order of magnitude of the task (hours vs days/months/years)

2. Add known planning overhead that is almost order of magnitude more.

Example: if we guess that task will take 30min, but actually it took 60min - that’s 100% error (30min error/30min estimate).

But if the methodology is used correctly, and we spend 2h in a planning meeting, same estimate and same actual completion time results in only 20% error, because we increased known and reliable part of the estimate (30min error / 2h30min estimate)

There’s no binary switch between estimable and not. Depends a lot on industry and novelty of work. Then estimates will be given in ranges and padded as needed by previous work. This gets a project into regularity.

> I used to work in the semiconductor industry writing internal tools for the company. Hardware very rarely missed a deadline and software was run the same way.

Former Test Engineer here. It was always fun when everyone else’s deadline slipped but ours stayed the same. Had to still ship on the same date even if I didn’t have silicon until much later than originally planned.

What was the thing you were estimating? R&D?

I think you were estimating time to build things that were out of R&D and you had specifications that were actual specifications you were building up to.

In SaaS my experience is: someone makes up an idea not having any clue how existing software is working or is laid out, has no specifications beside vague not organized bunch of sentences. Software development team basically starts R&D to find out specifications and what is possible - but is expected to deliver final product.

This aligns with my experience in the semi industry. SWEs tend to see trimming scope as moving the goalpost and do not consider as an option. Providing advance notice is mostly about client management, and clients are often surprisingly receptive to partial solutions.

Yes. I took over the project management of a job where the previous project manager had spent a year planning it out, but development had not yet started. The client was furious, understandably.

I abandoned the plans from the previous PM and discussed the job with the developer who ballpark estimated that the work would take 2 months. After a quick analysis I adjusted this to 14 weeks.

But the account manager thought this sounded too long and insisted that we plug everything in to a Gantt chart, define the shit out of everything, map the dependencies, etc, which showed that the development would only take 6 weeks.

The project ended up taking 14 weeks.

In another life, I would do things like measure the cost in developer time of bugs making it into developer repos vs. the cost in time of running tests in CI to catch such bugs, so evidence based decision making. It was mostly ignored, and at first I was surprised. A multi million dollar organization of people making negative EV plays, which I chalked up to the political pressures being more important than the wastage. More on that later.

As far as estimates go, I've also struggled with the industries cult(ural) rituals. I tried to put forward a Gaussian based approach that took into account not only the estimate of time, but the expected uncertainty, which is still probably off the mark, but at least attempts to measure some of the variance. But again, the politics and the rigidity of the clergy that has built around software development blocked it.

On the bright side, all this has helped me in my own development and when I think about software development and estimating projects. I know that outcomes become more chaotic as the number of pieces and steps compound in a project (i.e. the projects normal curve widens). You may not even get the project at all as defined at the outset, so my normals approach is still not quite the right tool.

I think this kind of thinking can be helpful when working solo or in a small group who are exposed to market forces. But for solo and small groups, the challenge isn't so much about the estimates, it's about how you're going to fight a battalion of mercenaries hired by big VC money and Big Tech. They can often afford to be inefficient, dump in the market, because their strategy is built around market control. These aren't practices small players can afford, so you need to get creative, and try to avoid these market participant kill boxes. And this is why, coming back to my earlier point, that often times, inefficient practices and politics plays a big role. Their trying to marshal a large number of troops into position and can afford to lose a few battles in order to win the war. The big money plays by a different set of rules, so don't worry if their doing it wrong. Just recognize your in the army soldier!

It's definitely unfair in a sense. But companies that make over-extrapolated roadmap estimates from not enough evidence systematically outcompete those who don't, because their customers greatly prefer companies who give a date and then try their best to hit it over companies who say they don't know when the product will be ready for X and you'll just have to wait and see.

You estimate your best and then during the project the people who keep changing the spec every two weeks ask why the deadline is slipping.

> This is why COMPANIES NEED ACCURATE ESTIMATES. They are not optional to running a company.

Sure, but even accurate estimates are only accurate as long as the assumptions hold.

Market conditions change, emergency requests happen, people leave, vendor promises turn out to be less than accurate.

And most estimates for non-routine work involve some amount of risk (R&D risk, customer risk, etc.).

So pounding the table and insisting on ACCURATE ESTIMATES without a realistic backup plan isn’t good business, it’s just pushing the blame onto the SWE team when (not if) something goes south.

If your business model needs the impossible then it's a bad business model. If your margins are too thin to absorb the schedule uncertainty then don't produce software.

Alternatively treat it like a bet and accept it may not pay off, just like any other business where uncertainty is the norm (movies, books, music).

I would settle for accurate estimates being a requirement if sticking to the estimate and allocations is as well. Every project I've been a part of that has run over on timeline or budget had somebody needling away at resources or scope in some way. If you need accuracy to be viable, then the organization cannot undermine the things that make it possible to stay on track.

Companies need accurate estimates like I need accurate stock market forecasts.

They don't NEED them, but better project estimates can reduce the error bars on other dependent estimates (e.g. estimated sales, estimated ship dates, estimated staffing requirements, etc...), and that might be useful to a business (or not).

I've also seen it argued that real world estimates for things like construction projects are so good because 99% of it is do-overs from similar projects in the past; everyone knows what it takes to pour a column or frame a floor or hang a beam.

Whereas with software most of what was done previously is now an import statement so up to 80-100% of the project is the novel stuff. Skilled leaders/teams know to direct upfront effort toward exploring the least understood parts of the plan to help reduce down-stream risk but to really benefit from that instinct the project plan has to regularly incorporating its findings.

Government contractor's estimation is based on what number is politically acceptable, not how much the project would realistically take. 90% of public projects were overbudget [0].

But you're pretty spot on, as 'professionally acceptable' indeed means politically acceptable most of the time. Being honest and admitting one's limit is often unacceptable.

[0]: https://www.strategy-business.com/article/Why-do-large-proje...

Contractor estimates are just as prone to schedule slippage and cost overruns as anything estimated by software engineers. I doubt anyone's ever argued that giving wrong estimates is hard or impossible. Only that approximately correct ones are, and other industries seem to struggle with that just as much as software. Authors don't finish books by deadlines, so fans are left in the cold. Tunnels take twice as long and cost twice as much. Renovations take a year instead of 3 months and empty your bank account.

Saying "I don't know" is arguably more honest, even if it's not useful for budgets or planning.

Incorrect analogy. Bridge construction is a clearly algorithmic process. All bridges resemble each other, and from an engineering perspective, designing one is not rocket science. Construction itself is a set of well-studied steps that can be easily calculated. If I were to write my operating system 100 times, I could give an estimate accurate to within 10%, but every task I’ve ever done in life is unique, and I have nothing to compare it to except intuitive judgments. Returning to bridges: there is 1% of projects that are unique, and their design can take decades, while construction might not even begin

When the government asks how much project X costs they find ten companies that promise the moon and then deliver a wheel of cheese for five times the estimated cost.

They miss estimates all the time though? It’s an observable fact

There is a bridge in my town that is finally nearing completion, hopefully, this year. It was estimated to be completed 2 years ago.

This changes when it’s a project that has fewer unknowns, where they’ve built the same thing several times before. The same is true in software.

Not a good analogy. Once you build a bridge, it’s done. Software nowadays is never “done”, and requirements constantly change. It’s more akin to building a rope bridge and trying to upgrade it to accommodate cars while it’s in active use.

Ever heard of Big Dig in Boston, for example? Or the Joint Strike Fighter?

Estimations in government contracts are as ridiculous as in software. They just pretend to be able to estimate when things will be done, when, in fact, the contractors are as clueless.

Not being able to say "it is impossible to estimate", does not mean your estimate will be correct. That estimation is usually a lie.

we are the ones qualified to say what needs to be cut to provide reasonable certainty for the deadline. it is not the job of non-technical stakeholders to mitigate risk in technical projects