I worked for a company that.. Used msql sever a lot and we would run into a heisenbug every few months that would crash our self hosted msql server cluster or it would become unresponsive. I'm not a database person so I'm probably butchering the description here. From our POV progress would stop and require manual intervention (on call). Back and forth went on with MS and our DBAs for YEARS pouring over logs or whatever they do.. Honestly never thought it would be fixed. Then one time it happened and we caught all the data going into the commit and realized it would 100% reproduce the crash. Only if we restored the database to a specific state and with this specific commit it would crash MS SQL Server. NDAs were signed and I took machete to our code base to create a minimal repro binary that could deserialize our data store and commit / crash MS SQL sever. Made a nice powershell script to wrap it and repro the issue fast and guess what? Within a month they fixed it. Was never clear on what exactly the problem was on their end.. I got buffer overflow vibes, but that's a guess.

As someone who works with hardware, hard to repo bugs can take months to track down. Your code, the compiler, or the hardware itself (which is often a complex ball of IP from dozens of manufacturers held together with a NoC) could all be a problem. The extra fun bugs are when a bug is due to problems in two or three of them combining together in the perfect storm to make a mega bug that is impossible to reproduce in isolation.

Sometimes, a "bug" can be caused by nasty architecture with intertwined hacks. Particularly on games, where you can easily have event A that triggers B unless C is in X state...

What I want to say is that I've seen what happens in a team with a history of quick fixes and inadequate architecture design to support the complex features. In that case, a proper bugfix could create significant rework and QA.

I think in general, bugs go unfixed in two scenarios:

1. The cause isn't immediately obvious. In this case, finding the problem is usually 90% of the work. Here it can't be known how long finding the problem is beforehand, though I don't think bailing because it's taking too long is a good idea. If anything, it's those really deep rabbit holes the real gremlins can hide.

2. The cause is immediately obvious, but is an architecture mistake, the fix is a shit-ton of work, breaks workflows, requires involving stakeholders, etc. Even in this case it can be hard to say how long it will take, especially if other people are involved and have to sign off on decisions.

I suppose it can also happen in low-trust sweatshops where developers held on such a tight leash they aren't able to fix trivial bugs they find without first going through a bunch of jira rigmarole, which is sort of low key the vibe I got from the post.

At Amazon we had a bug that was the result of a compiler bug and the behaviour of intel cores being mis-documented. It was intermittent and related to one core occasionally being allowed to access stale data in the cache. We debugged it with a logic analyzer, the commented nginx source and a copy of the C++ 11 spec.

It took longer than 2 days to fix.

I had a job that required estimation on bug tickets. It's honestly amazing how they didn't realize that I'd take my actual estimate, then multiply it by 4, then use the extra time to work on my other bug tickets that the 4x multiplier wasn't good enough for.

It's like remodeling. The drywall comes down. Do you just put up a new sheet or do you need to reframe one wall of the house?

Sometimes you find the cause of the bug in 5 minutes because its precisely where you thought it was, sometimes its not there and you end up writing some extra logging to hopefully expose its cause in production after the next release because you can't reproduce as its transient. I don't know how to predict how long a bug will take to reproduce and track down and only once its understood do we know how long it takes to fix.

>I can’t imagine spending more than a day on one.

You mean starting after it has been properly tracked down? It can often take a whole lot of time to go from "this behavior is incorrect sometimes" to "and here's what need to change".

Yep. Also, sometimes you figure out a bug and in the process you find a whole bunch of new ones that the first bug just never let surface.

> unless fixing a bug requires a significant refactor/rewrite, I can’t imagine spending more than a day on one

Race conditions in 3rd party services during / affected by very long builds and with poor metrics and almost no documentation. They only show up sometimes, and you have to wait for it to reoccur. Add to this a domain you’re not familiar with, and your ability to debug needs to be established first.

Stack two or three of these on top of each other and you have days of figuring out what’s going on, mostly waiting for builds, speculating how to improve debug output.

After resolving, don’t write any integration tests that might catch regressions, because you already spent enough time fixing it, and this needs to get replaced soon anyway (timeline: unknown).

> That said, unless fixing a bug requires a significant refactor/rewrite, I can’t imagine spending more than a day on one.

The longer I work as a software engineer, the rarer it is that I get to work with bugs that take only a day to fix.

I find most bugs take less time to fix than it takes time to verify and reproduce.

Its odd at first, but springs from economic principles, mainly sunk cost fallacy.

If you invest 2 days of work and did not find the root cause of a bug, then you have the human desire to keep investing more work, because you already invested so much work. At that point however its best to re-evaluate and do something different instead, because it might have a bigger impact.

Likelihood that after 2 days of not finding the problem, you wont find it after another 2 days is higher than starting over with another bug that on average you find the problem earlier.

How is this for a rule of thumb: the time it takes to fix a bug is directly related to the age of the software.

In my experience, the vast majority of bugs are quick fixes that are easy to isolate or potentially even have a stack trace associated with them.

There will always be those “only happens on the 3rd Tuesday every 6 months” issues that are more complicated but…if you can get all the small stuff out of the way it’s much easier to dedicate some time to the more complicated ones.

Maximizing the value of time is the real key to focusing on quicker fixes. If nobody can make a case why one is more important than other, then the best use of your time is the fastest fix.

> Is odd. It’s virtually impossible for me to estimate how long it will take to fix a bug, until the job is done.

Learning how to better estimate how long tasks take is one of my biggest goals. And one I've yet to even figure out how to master

I do agree that you should be able to fix most bugs in 2 days or less. If you have many bugs taking longer to fix, it may be an indication that you may have systemic issues. (e.g design, architectural, tooling, environment access, test infrastructure, etc)

I think the worst case I encountered was something like two years from first customer report to even fully confirming the bug, followed by about a month of increasingly detailed investigations, a robot, and an osciliscope.

The initial description? "Touchscreen sometimes misses button presses".

> It’s virtually impossible for me to estimate how long it will take to fix a bug, until the job is done.

In my experience there are two types of low-priority bugs (high-priority bugs just have to be fixed immediately no matter how easy or hard they are).

1. The kind where I facepalm and go “yup, I know exactly what that is”, though sometimes it’s too low of a priority to do it right now, and it ends up sitting on the backlog forever. This is the kind of bug the author wants to sweep for, they can often be wiped out in big batches by temporarily making bug-hunting the priority every once in a while.

2. The kind where I go “Hmm, that’s weird, that really shouldn’t happen.” These can be easy and turn into a facepalm after an hour of searching, or they can turn out to be brain-broiling heisenbugs that eat up tons of time, and it’s difficult to figure out which. If you wipe out a ton of category 1 bugs then trying to sift through this category for easy wins can be a good use of time.

And yeah, sometimes a category 1 bug turns out to be category 2, but that’s pretty unusual. This is definitely an area where the perfect is the enemy of the good, and I find this mental model to be pretty good.

Yeah, "no bug should take over 2 days" tells me you've never had a race condition in your codebase.

> It’s virtually impossible for me to estimate how long it will take to fix a bug, until the job is done.

This is explained later in the post. The 2 day hard limit is applied not to the estimate but rather to the actual work: "If something is ballooning, cut your losses. File a proper bug, move it to the backlog, pick something else."

I believe the idea is to pick small items that you'd likely be able to solve quickly. You don't know for sure but you can usually take a good guess at which tasks are quick.

I've seen people spending 4 months on a hard to replicate segfault.

Not sure why would you ever need to refactor for fixing a bug?

you cannot know. that’s why the post elaborates saying (paraphrasing) “if you realize it’s taking longer, cut your losses and move on to something else”

> That said, unless fixing a bug requires a significant refactor/rewrite, I can’t imagine spending more than a day on one.

oh sweet sweet summer child...

Bugs taking less than 2 days are great to have as a target but will not be something that can be guaranteed.

> It’s virtually impossible for me to estimate how long it will take to fix a bug, until the job is done.

Now I find that odd.

Reminds me of ids policy of "As soon as you see a bug, you fix it"

"...if you don't fix your bugs your new code will be built on buggy code and ensure an unstable foundation and if you check in buggy code someone else is going to be writing code based on your bad code and well you know you can imagine how wasteful that's going to be"

16:22 of "The Early Days of id Software: Programming Principles" by John Romero (Strange Loop 2022) https://www.youtube.com/watch?v=IzqdZAYcwfY&t=982s

From the post:

> That’s not to say we don’t fix important bugs during regular work; we absolutely do. But fixits recognize that there should be a place for handling the “this is slightly annoying but never quite urgent enough” class of problems.

So in their case, fixit week is mostly about smaller bugs, quality of life improvements and developer experience.

Depending on the size of the team/org/company, working on anything other than the next feature is a hard sell to PM/PO/PgM/management.

Bugs have priorities associated with them, too. It's reasonable for a new feature to be more important than fixing a lower priority bug. For example, if reading the second "page" of results for an API isn't working correctly; but nobody is actually using that functionality; then it might not be that important to fix it.

Where have you worked where this was practiced if you don’t mind sharing?

I’ve seen very close to bug free backends (more early on in development). But every frontend code base ever just always seems to have a long list of low impact bugs. Weird devices, a11y things, unanticipated screen widths, weird iOS safari quirks and so on.

Also I feel like if this was official policy, many managers would then just start classifying whatever they wanted done as a bug (and the line can be somewhat blurry anyway). So curious if that was an issue that needed dealing with.

I'd love to see an actual bug-free codebase. People who state the codebase in bug-free probably just lack awareness. Even stating we 'have only x bugs' is likely not true.

Any modern system with a sizeable userbase has thousands of bugs. Not all bugs are severe, some might be inconveniences at best affecting only a small % of customers. You have to usually balance feature work and bug fixes and leadership almost always favours new features if the bugs aren't critical to address.

Many of the bugs have very low severity or appear to small minority of users under very specific conditions. Fixing these first might be quite bad use of your capacities. Like misaligned UI elements, etc. Critical bugs should be done immediately of course as a hotfix.

In the places that I worked, features came before all else, and bugs weren't fixed unless customers complain

Thing is, if you follow a process like scrum, your product owner will set priorities; if there's a bug that isn't critical, it may go down the list of priorities compared to other issues.

And there's other bugs that don't really have any measurable impact, or only affect a small percentage of people, etc.

This is the 'Zero Defects'[1] mode of development. A Microsoft department adopted it in 1989 after their product quality dropped. (Balmer is cc'd on the memo.)

1. https://sriramk.com/memos/zerodef.pdf

In your experience, is there a lot of contention over whether a given issue counts as a bug fix or a feature/improvement? In the article, some of the examples were saving people a few clicks in a frequent process, or updating documentation. Naively, I expect that in an environment where bug fixes get infinite priority, those wouldn't count as bugs, so they would potentially stick around forever too.

Would love to work on a project with this as a rule but I am working on a project that was build before me with 1.2 million lines of code, 15 years old, really old frameworks; I don't think we could add features if we did this.

I wrote up my thoughts on this into a longer post: https://killthehippo.com/posts/fix-bugs-or-add-new-features

Yes, it's a leadership failure and probably time to go, it only gets worse. In my experience. It's a vicious cycle where, as velocity slows, inexperienced leadership gets more and more panicked and starts frantically rearranging projects, features and people in desperate attempts to fix the problem, obviously exacerbating the communication breakdown and gridlock further. It also builds resentment and can turn pretty toxic as everyone starts just looking out for themselves.

Symptoms of a dysfunctional company where communication has broken down, everyone with any authority is running around EXACTLY like a headless chicken, waving around frantically (giving orders). Margins are probably thin as a razor, or non-existent. They will micromanage your work time to death. You will be treated as a commodity factory machine and if you start using your brain to solve actual problems, you will be chastised. Deadlines everywhere keep everyone's brain shut off and in panic mode. No time to properly engineer anything. Nobody has the time to check anyone else's work, causing "trust" that isn't even blind, just foolish. You as the software guy end up debugging and fixing EVERYONE's mistakes. When the bug is in hardware/electronics, everyone knows who's actually to blame, but everyone still expects YOU to fix it, and they're immensely disappointed when you can't save the day.

These places cannot and will not change. If you can, find employment elsewhere.

This is not uncommon but I've mostly managed to avoid it, because it's a management failure. There is a delicate process of "managing the customer" so that they get a result they will eventually be satisfied with, rather than just saying yes to whatever the last phone call was.

Yes, usually not worth it to spend too much time on proper engineering if the company is still trying to find a product-market fit and you will be working on something else or deleting the code in a few months.

> did anybody have had this kind of experience?

Yes, the issue is not you, it's a toxic workplace. Leave as soon as you can.

> But I am saddened by the wider context, where a dev posts 'we fix bugs occasionally' and everyone is thrilled, because the idea of ensuring software continues to work well over time is now as alien to software dev as the idea of fair dealing is to used car salesmen

This is not the vibe I got from the post at all. I am sure they fix plenty of bugs throughout the rest of the year, but this will be balanced with other work on new features and the like and is going to be guided by wider businesses priorities. It seems the point in the exercise is focusing solely on bugs to the exclusion of everything else, and a lot of latitude to just pick whatever has been annoying you personally.

That is why I stand on the side of better law for company responsibilities.

We as industry have taught people that broken products is acceptable.

In any other industry, unless people are from the start getting something they know is broken or low quality, flea market, 1 euro shop, or similar, they will return the product, ask for the money back, sue the company whatever.

A company creating the conditions that allow for high quality engineering has always been the exception, not the norm

Very interesting post, thank you!

I'd also be curious to know the following: how many new errors or regressions were caused by the bug fixes?

The unkind world we live in would see this role being abused quickly and a person not lasting long in this role. For one, in the wrong team, it might lead to devs just doing 80% of the work and leaving the rest to the janitor. And the janitor might get fed up with having to fix the buggy code of their colleagues.

I wonder if the janitor role could be rotated weekly or so? Then everyone could also reap the benefits of this role too, I can imagine this being a good thing for anyone in terms of motivation. Fixing stuff triggers a different positive response than building stuff

If the bug affects 1 customer and the feature affects the rest, is the old feature complete?

It's not binary.

Because the goal of most businesses is not to create complete features. There's only actions in response to the repeated question of "which next action do we think will lead us to the most money"?

Bugs can get introduced for other reasons besides “feature not completed”.

until we develop a way for MBA's with spreadsheets to quantify profit/loss w.r.t. bugs, it will never be valued.

As I pointed out in the "criticisms" section, I don't see fixit weeks as a replacement for good technical hygiene.

To be clear, engineers have a lot of autonomy in my team to do what they want. People can and do fix things as they come up and are encouraged to refactor and pay down technical debt as part of their day to day work.

It's more that even with this autonomy fixits bugs are underappreciated by everyone, even engineers. Having a week where we can address the balance does wonders.

When the world learned the wrong reason for why the Mercator Projection was adopted.

Because marginal work is only marginally rewarded. Spending one week and coming back to whoever with a nice piece of paper saying we fixed 60 bugs will earn a lot more rope from non-technical folk than fixing 3 bugs per week - the latter just looks like cleaning up your incompetence.

I would be weirdly happy to have a role whose entire job was literally just deleting code. It is extremely satisfying.

> this is not care when the priority system is so broken that it requires a full suspension

I'm not sure I understand this line. The whole point of the fixit is to address the bugs which are considered "low priority" because they only appear in a edge case or are not quite 100% perfectly polished but still matter over the long tail of people using the product.

Or do you propose that every issue like this needs to be fixed before doing anything else?

To be fair, the blog post does not explicitly say anywhere that the week was for bug fixes only.

Shape Up was my first thought too. I just left a team where I introduced cycles of six weeks of feature development and two weeks of bug fixing, tech debt, and anything else the developers decided to tackle.

It depends on the stage and size of your team and company of course, but for us the result was more predictable delivery and happier, more-engaged developers.

For anyone curious to learn more: https://basecamp.com/shapeup/2.2-chapter-08#cool-down

I've never had a good experience with individual metrics leaderboards. On one team we had a JIRA story point tracker shown on a tv by a clueless exec. Devs did everything they could to game the system and tasks that required uncertainty (hard tasks) went undone. I believe it contributed to the cog culture that caused an exodus of developers.

However, I love the idea of an occasional team based leaderboard for an event. I've held bug and security hackathons with teams of 3-5 and have had no problem with them.

> Generally, if they were worth fixing - they would have been fixed.

???

Basically any major software product accumulates a few issues over time. There's always a "we can fix that later" mindset and it all piles up. MacOS and Windows are both buggy messes. I think I speak for the vast majority of people when I say that I'd prefer they have a fix-it year and just get rid of all the issues instead of trying to rush new features out the door.

Maybe rushing out features is good for more money now, but someday there'll be a straw that breaks the camel's back and they'll need to devote a lot of time to fix things or their products will be so bad that people will move to other options.

A greedy algorithm (in the academic sense, although I suppose also in the colloquial sense) isn't the optimal solution to every problem. Sometimes doing the next most valuable thing at a given step can still lead you down a path where you're stuck at a local optimum, and the only way to get somewhere better is to do something that might not be the most valuable thing measured at the current moment only; fixing bugs is the exact type of thing that sometimes has a low initial return but can pay dividends down the line.

ROI is in reduced backlog, reduced duplicate reports and most importantly mitigation of risk of phase transition between good enough and crap. This transition is not linear, it’s a step function when the amount of individually small and mildly annoying at worst issues is big enough to make the experience of using the whole product frustrating. I’m sure you can think of very popular examples of such software.

They said they only pick bugs that take 2 days to fix.

Places where you can move fast and actually do things are actually far better places to work for. I mean the ones were you can show up, do 5 hours of really good work, and then slack off/leave a little early.

189 presumably

You criticize the initiative because you judge it doesn't have impact on the product or business. I would challenge the assumption with the claim that a sense of acconplishment, of decision-making and of completion are strong retention and productivity enhancers. Therefore, they're absolutely, albeit indirectly, impacting product and business.

Just because you can't measure the loss of customers who are turned off by your buggy product doesn't mean they don't exist.