I think our processes are terrible as an industry. I have brought this up many times but we don't understand what actually works when something goes right and what failed when something went wrong. Adding to this is that engineers love tools and process so they tend to credit tools and process with success because we like the machine. Giving it credit where credit wasn't due leads to slowly growing more elaborate process and tools over time. This love of tools and process is a fundamental flaw in our culture and it is a big part of why big teams fail and small ones can get things done.

There are two fundamental truths to software, or any real organizational level problem. First, you don't know what the solution is until you have actually built it and are using it and second designing and building something is a non-polynomial growth problem.

The first part of the problem we sort of get, sometimes. The solution is iteration for the same reason it has always been. Assess, step, assess, step isn't just a good way to train a NN, it is also a great way to do pretty much anything where you don't know the optimal solution. Take the gradient of the situation and then take a right sized step in the right direction. Think you can have a perfect design before you start coding? You are basically saying you can take one big step from the start to the end. Either you have a small problem to solve or you are deluding yourself. Successful software is iterative. It always was and always will be. If your retrospective says things like 'if we had just done X from the start' be very careful because you are falling into the hindsight trap. You really couldn't have known X was the right thing. There is a reason you didn't see X. Just accept the iterative nature and own it. Try for appropriate step sizes, do good regular assessments, keep the iterations tight and you will probably be ok.

The second problem, NP growth, is where things really fall off the rails though. People get iterative, they see it work, even if they don't understand what they are really doing, but NP complexity growth is a real killer. The problem is that it actually IS true that if you took more time and put all the pieces together and solved it all as one problem you technically could eventually find the better solution. But more than likely the heat death of the universe will catch you before you do. Oh, yeah, and the total information storage needed to document the combinations tried will likely kill you too. There is only one good solution to NP growth, accept a local minimum and divide and conquer.

NP complexity growth is the foundational problem that needs to be attacked and the why things work or don't. Even more than iterative in many cases. As a problem grows its complexity, the possible number of solutions to check, grows in an NP way. The only solution is to drop the number of options to consider. You have to divide the problem and admit a local optimum is the best practical solution. People -sort of- get this by pretending to break the problem up and give it to different people or teams but then totally blow it. Jira is an example of totally blowing it. So you broke the problem down and you broke the teams into smaller pieces to address those sub problems but then you threw it all in one place again in Jira and you had all the teams in the same standup. You can't do that. That is the point of divide and conquer. You do that and you get lost because the problem just got too big again when you put all the pieces together. Also, communication scales up with people, even without problem size changing. Create too big of a team and the communication eats all the available work. Divide and conquer -requires- not communicating, or at least being exceptionally careful about how you communicate between problems.

The processes and tools we have created and love to use so much are the heart of why things don't work and we need to start admitting that. They give us a false sense that we can make a team bigger or take a bigger problem on. That is a mistake.

If you have done a good job of dividing a problem up, and correctly sized teams, then you have created problems that are clear enough not to need status boards and the like. Sure, go ahead and use them if your small team likes that. Be my guest, but you probably shouldn't. If a team is iterating on their problem and the problem is appropriately scoped then the team knows the state of their entire piece so well that the status boards slow them down. Why put in a jira ticket when you can just deal with it? Why break your internal team communications like that? Team management and project management become easy with small teams since your options are limited and the problem is small so it is all obvious. If you are saying to yourself 'well how will we know the whole thing is on track' well if you divided correctly then every level has a human sized understanding to deal with and is keeping track of their piece. That includes the team that owns teams. They should have designed the teams working for them, and the problems those teams are dealing with, in such a way that the working memory state is enough. They also designed the communication to that team in a way that they stay informed -without- joining that team and in doing so joining all teams. In other words they don't micro manage because that breaks divide and conquer. If any level is lost then the problem may not have been broken down well or has changed. A good iterative team catches this and raises the flag quickly so the divide can happen again if needed. The team leading the team has the job of monitoring to help figure this out, but monitoring in very limited ways so that they don't end up micro managing and collapsing the divisions.

A good military know this and a bad one has forgotten it. In WWII we had task forces for everything. They could stand up a TF, get it training so that it was a coherent entity, execute the mission needed and tear it apart. We were amazing at it. When WWII ended we did big things because we carried our understanding of the operational level of war, how to break apart problems and teams, into industry. We went to the moon. Now however we have standing task forces in the US military that are essentially the leftovers from WWII. We crate new task forces, badly, that are really just the existing ones renamed which means they have their old job and new job and nothing has really been broken out and isolated correctly. We suck at war and a big reason for this is that we have forgotten the operational level of war lessons from WWII.

This is a long rant to get to this final point. The author doesn't get the real reason why '20%' does the work. It is because we hire and create massive teams that can't get anything done because their communication has scaled to 1000% of their capacity. So, naturally, a small core team forms that can effectively communicate and get a job done, by ignoring he other 80%. It isn't the other 80%'s fault, it is the organizations fault for not breaking things up and creating small teams where the size of the problem is understandable and actionable and, most importantly, not re-merging the problem and the teams with stupid things like Jira boards.

The real solution is the same set of solutions that work time and time again. Create small teams. Give them clear problems to solve and the right tools and authority to solve them. Put bounds on what they should be doing so they, and you, don't get distracted. Understand that a problem is an evolving iterative thing and lean into that. If 80% of your workforce isn't doing things then your organization is broken. Start figuring out how to fix it. Collaboration isn't bullshit. It is fundamental. We just need to actually, intentionally, design that collaboration based on the actual things that shape it. NP growth and iterative understanding.

This.

What everybody keeps forgetting over and over again is that software is super complicated even if it can be changed from a keyboard without the use of physical morphing tools.

People who do not themselves generate software are in the position of telling the people who generate software how to do it and what the constraints should be on the outcomes.

Accept that it is complicated and that you cannot know in advance when it will be done unless it is a super simple request.

It is indeed more like oil field exploration than it is like sweeping the floor.

You cannot really know where the solution to a complicated problem lies in advance and therefore you cannot predict how long it will take you to find it.

People on the finance side just need to face the fact that there is risk that cannot be eliminated in advance or even quantified particularly accurately.

If your investors cannot stomach this, they probably need to invest in something other than software development.

Good luck with finding that in 2026.

[dead]