Can I do a mea culpa? This is more than 3 decades back. I was a junior programmer (2-3 years in industry) sent to a client site in europe. You can imagine the state of systems those days. I wrote (or rather updated) a fix which would updated the discount and tax rates on orders based on new terms. It would run every day to account for ... whatever. You pick the values from master file, update all the orders and move on.

I wiped out VAT on all orders and for the next month the paper invoices were sent without VAT. So the invoice is $100, VAT is $20, the invoice should be $120, but they were sent as $100.

100s of invoices every day would be my guess.

Nobody noticed.

For a month.

Millions of dollars of revenue and IIRC millions of dollars of VAT.

Until a customer complained to the CEO.

We had a firefight to fix it, not just technical but legal and managerial. We can send a new invoice just for tax. We can redo the invoice. We can send a debit memo. What is the right decision? But what if customers does not pay? What about returns? How will we track returns? Of course we were doing the technical solutions and the client company was front-ending how to handle it business wise.

And the managerial firefight - who did it, what are the safeguards in future? We had a company exec visit the client site to manage the issue.

I was in the hot seat but I was protected by my managers from any fallout. Just do the work. Do not screw up again. (Test every row every column even if you did not change it)

A month later the sales director at the client company got fired.

The grapevine is that this was just the tipping point, but you never know. BTW these were paper invoices printed onsite and mailed out, but I do not know if someone had the job to scrutinize them.

PS: True story, going by old memory, although such legends remain fresh in your mind, forever. Not sure it belongs here, but the mention of firing for a multi-million dollar mistake pulled this into cache memory.

If a single non-malicious code change can break a thing like that with nobody noticing that's a catastrophic failure in testing, QA, and operations (nobody noticed 20% of euros transacted just stopped). It's hard to blame that on IC engineer.