A lot of ink has been spilled on this topic.

The solution is simple: get better at estimating.

Software engineers act as if they're the only ones in the world who are asked to estimate and then held accountable.

It's a skill issue.

> It's a skill issue.

Maybe. Sort of. Skill hints at the problem, but it's more of an experience issue. More experienced developers can give pretty reliable estimates for work they've already done. The catch is, how often do you ask an engineer to do something they've already done? The beauty of software, is that you don't solve the same problem over and over, because if you did, you'd automate it.

So where does that leave us? Well, developers who've seen a lot of different problems recognize when a new problem they see rhymes with a problem they've solved before. That can allow for a very good estimate. But even then, engineers tend to think in terms of best case, or maybe even typical case. I saw a study on this a few years ago, and it showed how often engineers tended to cluster around the median time for solutions. But, since really long tasks, with big error bars have a tendency to really skew timelines, the solution averages was much farther from these median times.

Believe it or not, lawyers have this problem too. They are taught in law school to come up with an estimate based on expected effort, then they apply a formula similar to: double the estimate, then increase the time units. So a 5 hour task becomes 10days. A 2 week task becomes 4 months. Mind you, this isn't the amount of billable hours they're predicting, it's the amount of calendar time that a given task will take until it's complete. This ends up taking into account a lot of variables that lawyers have no control over. Things like court scheduling, or communication delay. I suspect it also takes into account blind spots we humans have around time estimates in isolation. Like, 1 task, if you can focus on it can be done in so many hours. But if you have to do 5 similarly sized tasks, it takes more than 5 times as long, simply because it's easy to expend resources like brain power, and the estimate ignores the refractory periods necessary once the task is completed. (BTW this was one of the problems with the very first stop watch study in the rail yard, where the rail workers didn't work at their sustainable pace, but worked at their depletion pace).

You're correct to a degree, but IME it's a discipline issue, not a skill issue as such.

What makes software hard to estimate is not so much the work, but how much requirements, priorities and resourcing change on the fly. I've worked in places that did quite strict scrum: once the sprint was planned and agreed to, we pushed back hard on our business area if they wanted to change anything. For practical reasons we didn't outright ban it, but for example, if they wanted something new done "right now" they had to choose something of roughly the same scope to be removed. If they couldn't do that, we'd cancel and re-plan the sprint. They hated having to make that choice, so most things just ended up at the top of the backlog for the next sprint.

The other part was our own management wanting to take people out of the team mid-sprint to work on something else. We never said no, but did say "this means we have to cancel and re-plan the sprint (and tell our business area)".

Basically, we made the people making the changes take responsibility for the impact on deadlines. Our estimates in that scenario were quite good (within the bounds of the sprint, at least).

Of course, the longer the estimation window the less well this works. The only way to estimate accurately is to constrain scope and resources, which is not actually what management/business want.

It's literally impossible to know what you don't know. Sometimes you discover things in the process of your work that couldn't have reasonably been accounted for on first estimate, and it is not good practice to always give an estimate that assumes a 5x difficulty multiplier for some unforeseen reason.

Yes, if this is an "every time experience" then there could be a skill issue or possibly some other undiagnosed or unrecognized systemic factor

If you're hiring architects and engineers to design and build your home, you might already have a pretty good idea of the home you want. The people you've hired provide estimates on cost and timing based on solidly known quantities. They've put in a basement like that before. They've worked with your weird materials. Their vendors report on material status daily.

Software development is not surrounded by this sort of infrastructure or codification. My discovery process establishes these lines of communication, and I have no idea when I'll uncover a black box or when one will be imposed upon me.

No, it is a funding issue. Nobody wants to pay software engineers for actually doing prestudies to learn enough to do a proper estimate. They want a good estimate but do not want to pay for it.

Nonsense. Code is copy-pasteable; other things are not.

One can give very accurate estimates of how long it takes to build a brick wall because building brick walls takes time and labor. You can make highly accurate estimates of how long it takes based on how long it has taken to do the exact same task in the past.

But suppose I laid one brick and then could copy-paste it, then copy-paste that into four bricks, then eight, until I have a wall. Then I can copy-paste the entire wall. Once I have a building I can copy-paste the entire building in five seconds.

The ability to copy and paste an entire building is very valuable but how long does it take to create that copyable template? No one knows because it has never been done before.