There will have been tests, but there will have been missing end-to-end tests. Test 1 will verify that the new system/product emits billing entries in some expected way ("We did 100 bytes of operations and we see we called the billing system for 100 bytes of stuff, yay, test pass"). Test 2 will be in the billing system ("We provide an incoming bill for SKU#12345 for 100 gigabyte-units and we see it costs $17, yay, test passes"). But they won't test the two things together because it will be harder to do and the teams will have different management chains. Seen it happen several times at several companies. Somebody will have said at some point "we should actually have the tests charge money" and somebody else will have said "well we can't have the tests actually charge money, that's a legal/accounting problem, it might even be a crime" and then nobody would have asked what the next best thing was.

Not even tests but just some basic anomaly detection lol.

Like maybe if the bill amounts increase by like 10M% there should be someone that looks into it

>no tests?

Earlier this week my slopservant implemented several comprehensive changes to a codebase.

It also wrote extensive tests to verify the correctness of the changes.

A few days later I was working on something else and realized, everything had been implemented backwards, in a way that was nonsensical and also completely pointless.

The many tests it had written were just confirming the LLM's idea of correctness, which turned out to be... completely incorrect.

I laughed when I realized, if I had been using Rust, or indeed, formal verification, that wouldn't have helped at all — it would have just written a mathematical proof, proving the correctness of the wrong thing!

Not sure what lesson to take from that (except read the damn diffs, obviously — it was a hobby project okay ;), but it seems like the more reliable this stuff gets, the more we expect it to work properly, the more risky it becomes.

https://en.wikipedia.org/wiki/Normalization_of_deviance

we invoice clients monthly. if we ever sent one a bill 2^30 times too high, I don't think "estimated charges, no action required" would save the relationship :D

This reminds me of a discussion a few months back from a BSD maintainer who had done a lot of volunteer work for AWS over the years. I think he might have even been the person who alerted them to the insecurity of IMDSv1. There was a sense that AWS might have had great talented developers at the time, but they clearly didn't really understand the domain of running a hypervisor service exposed to the public. This feels like a similar situation, where I'm sure they "test" what they know to test, in the way they know, but compared to a bank, well, it wouldn't surprise me the least if no one at AWS ever even thought to ask banks how they handle things like this. Instead of testing that a process works the way you specified it, you need to make sure you even have the right specification in the first place by consulting with prior art and ensuring whatever you translate into software reflects the legal reality of the process you're trying to encode.

There's a similar thing with physical processes. My wife works in geointelligence ground processing and encountered this when her system was expanded to support SAR collections instead of just visual spectrum. Software that passed all of its internal tests was passing nonsense collection parameters because the developers didn't understand the difference between energy collected to a sensor cell reflected from the sun versus energy reflected from your own active scan. The capabilities and limitations aren't the same. You can perform the same processing. The bits on disk won't care. But if the process you encode doesn't accurately correspond to the physics of reality, you're producing nonsense. Memory-safe, syntactically-valid nonsense, but nonsense nonetheless.

This seems to happen quite a bit with software companies.

Who needs tests when you have vibes?

It’s fractions of a penny Peter.

It's just like in Superman III

What could a banana possibly cost Michael, ten dollars?

...in the off chance that one in 100,000 will pay their bill instead of dropping dead from a heart attack (perhaps because there's a heartless OpenClaw running their accounting department), it's all worth it.

Why would you think there are "no tests"?

I'm sorry but anyone that sees a multi million or billion dollar bill on an account that does nowhere near that should not be scared. It's obviously a mistake. Stories like this have happened with banks in my country. Check your account and you have billions in there. Guess what happened to those that withdrew money? The judge told them any reasonable person would know this is a bug. Had to give it back. Same thing here, any reasonable person doesn't get scared.

But someone that susceptible is likely going to have a bad reaction to many possible unexpected things. How would they react to a minor traffic accident? a family member getting hurt? a letter from tax authorities asking questions?

Having that serious an underlying health problem means everyday life represents risk for you. I don't think that means everyone else has to behave differently wrt (in this case wrt to billing mistakes) to keep you healthy.

actually $17B isn't so bad because it's so obviously an error and not something you can fatfinger. if you expected it to be $17 and it was $17,000 you'd be much more worried as that looks like a plausible error on your part

its so far out as to be obviously not real

a smaller error by say, just one or two orders of magnitude are much more believable as a reader

Give that engineer a raise and an official title of "chaos monkey".

On the scale of AWS, you want your production system to be poked at and pushed against. Obviously not something to implement out of the blue in your production if you've never had it. But certainly worth considering.

Even on a smaller scale, it makes a lot of sense to have some deamon randomly bringing nodes down, shut off a database, fill a disk up, etc etc. In production.

Because that brings awareness to all engineers that the stuff that "can happen", will happen. It makes for a culture of defensive, considerate engineering. A culture where someone who finds bugs or holes in production is rewarded, not punished.

A culture where having some automated management for nodes is not a ticket "at the bottom the backlog", but a crucial attribute when picking or engineering the hosting system. That having redundancy in critical services like a database, is not an expense, but actually a cost saver, because chance of critical services failing is 100%, rather then some made up gut-feeling-chance. That the error handling and tests for when a module cannot write a file to disk isn't something that "this senior who left 8 months ago overengineered", but a pattern to be consistently implemented¹. etc.

---

¹ It is one the things I love about rust, that it has built in enforcement for you to deal with all the esoteric errors (e.g. https://doc.rust-lang.org/std/io/enum.ErrorKind.html#variant...) that can occur. Sure, you can write `File::create("foo.txt").unwrap()` but even then, you have made a conscious decision, clearly "documented" that you will crash the program - i.e. postponed the decisions on what (groups of) errors must be handled in what way, to a time you have the information to make that decision. Same design decision with go, where errors are explicitly to be handled by the programmer.

That was just a massive operational failure, not the fault of any single engineer. No change, except hotfixes, should be able to land on prod unless it has at least go through test, staging, and at the scale of Amazon, shadow testing.

Engineers will do what engineers will always want to do, they want to see how things break, and sometimes they manage to fix it.

CSV files are widely known to be used by the most frugal companies so of course it is.

It was the same old kona files for metering until at least a ‘23-24 when I last did metering/pricing work. “Luckily” the size of the EC2 pricing plan was so stupidly large that it was forcing improvements to how those were serialized, and how private pricing plans were created/assigned to different payer ids.

Hey man, vibe coders gonna vine code. No wait. Loop enginering.

We don't need to do stupid things like review or plan or strategize. Put everything into LLM and let the black box take over everything! Yay! LLMs!!!!

Nope, different service. We were introducing new methods using POST and (iirc) some differential pricing based on the actual request params. Which made it a different code path, metering value, price, etc than the existing GET requests.

Unit mistakes happen all the time, which is why you should be using your units library religiously and still being vigilant even then.

Worst case I've found was off by 15 orders of magnitude.

Multiple rockets exploded and space missions failed because of the imperial vs. metric BS, and those mistakes were made by people all with PhDs or equivalences. This is still pretty mild in comparison haha.

Units and datetime will always be the bane of any professions ...

Like F#...? https://learn.microsoft.com/en-us/dotnet/fsharp/language-ref...

Wow:

    In this role you will:
    - Design and build agentic AI systems that analyze, generate, and validate...
    - Build agentic architectures that compose specialized AI agents dynamically...
    - Build AI-driven continuous validation frameworks powered by agentic workflows and large language models that autonomously manage...

> 194,400.00 USD annually

Fuck it, im in.

That job description feels so far beyond parody that I could scarcely believe it until opening the link! What a world.

IRE which stands for Invoice Reliability Engineer.

-$3 trillion! That's the highest earning investment that has ever existed!

3.7 billion. Offered to pay it in monthly installments. Haven’t hears back

Quick do your IPO before the books update

same. over 2t in one day.

The finance fun noone knows about. I knew someone who did this as revenge. Sent some asshole a fake invoice for US$stupidmoney for a couple of months which said asshole ignored, then sent a certified mail 'I forgive asshole US$stupidmoney', and copied the IRS. IRS went to collect on the unpaid taxes, and if you've never been there, the IRS does not play when it comes to that sort of money. I don't know how it ended, but there were multiple lawyers involved, so it was expensive no matter what.

Why? What's the damages? They showed you a wrong number, then later acknowledged it and fixed it. Just because the number was "very big" to you doesn't mean you were actually aggrieved in some way.

…for emotional damage?

Banks and financial institutions are the same. If they haven’t issued you a monthly/quarterly statement yet, they can just apologize and tell you the numbers are wrong please wait for the statement. But it is a major issue if an actual statement has the wrong numbers.

Reminds me of working for a cable company and being told that even if we screwed up and stole from the customer the look back period was only a few months and if we found an error from before that we weren't supposed to correct it.

I've personally noticed and saved multiple $xx,xxx monthly cost billing spikes just by take a daily glance at our cost explorer. I'm in the AWS accounts every day doing investigative work anyway that an extra 30-60 seconds is trivial.

Seeing something "small" like an ECS task that is continuously failing to start properly because of a bug and repeatedly pulls a container image or a lambda function that's taking longer that it reasonably should (takes 5-10 seconds when it's normally a tens or a few hundred milliseconds) can dramatically drive up a bill in short order.

> 2. I hate the idea of wasting money on buying Jeff Bezos a bigger yacht

Then you aren't using AWS. At least half of all the money you give to Amazon is yacht money.

Would be funny if it wasn't so close to true

So uh did you type that out or generate it somewhere?

Number felt high so I wanted to double check and I only get 301500.

I've found personally it's better to use AI to build a deterministic script for calculations like that. (anything that manipulates data should be a script not an AI).

That is literally what Excel is for. Why didn't you use that first of all?

My hunch is the HN formatter swallowed the double asterisk typical of python exponents.

While we're being pedantic, 2^30 is 28 in normal programming languages ;)

But 2^30 is 28.

>> Or just a distracted dev

And a distracted tester? And a distracted pipeline of regression tests?

No, the truth is way worst...

Either way it shows their QA and testing procedures are incompetent. It's just not acceptable for a utility like AWS to move fast and break shit. Should make you question whether it's safe or advisable to use any of their services.

It probably shouldn't be legal for banks, hospitals, governments, or any other critical infrastructure to be hosted on AWS if they do things like this.

Far less scary than a smaller amount

This number is obviously absurd but for other normal amount (say 2 million which can definitely happen with a mistake), can't they claim it in court that will bankrupt you even if your entire worth is just 100K for example?

Ah, there'll be something in the TOS absolving them of responsibility.

Reminds me of the time a Robinhood user killed themselves after Robinhood erroneously showed their account balance at -$730k.

https://www.bbc.com/news/world-us-canada-55990461

I had a similar situation where some dormant account was still charging my credit card.

The account was probably real, made for some purpose 15 years ago. I had ignored the charge for years because it was like $7 a month. Then it went to $300/month, making it worth the time. I could find no invoice email and none of my AWS accounts lined up with the bill.

I tried contacting support, but without the account number involved they had no way to help. I disputed the charge, the bank refunded that month and then went right back to charging the next month(my credit card helpfully accepted the charges despite the dispute).

I had to cancel the credit card entirely to make it stop.

What is this, US health care negotiations?

I think that this might reflect more on Amex to be honest.

Amex realises that threatening would hurt their business trust more than anything. During the great depression, Amex accepted checks from other banks which were falling and paying through their own wallet as a matter of integrity. Amex has always been built around this idea of trust and prestige.

They make most of money from what I have heard on the transaction fees which are more than others (3% compared to 1%). They might get desperate but I am sure that they are one of the last guys who would wanna threaten you if you are paying some large bills for them (as compared to normal credit card companies which might even hire people to extract your loans in some messy situations)

So perhaps be so rich that the credit card company understands it as well and treats ya differently :-D

That's the government's problem

The Federal Reserve? You ressurect Alan Greenspan, and tell him to brrrrr.

Not because of his net worth (he is probably worth 200bn if it came to liquidating) but because he will pull some trick to get the money. Maybe he will chuck SpaceX under debt, draw a dividend then set up a 1000 year payment plan with the bank.

Elon Musk is everyone's problem

Relevant to this, I've recently noticed a trend of mass tort cases being opened up in the past couple years, and they seem to do very well. The way these seem to work is attorneys identify a company who has clearly ripped people off, and what I presume is a repeatable way to guarantee a win (thus translating to a guaranteed settlement offer). Then they advertise for eligible clients, sign those clients individually to contingency agreements, and run the playbook. A couple months ago after signing up for one of these, I received a check for about $350 (after the agreed-upon 40% attorney fee), from Ticketmaster, and I had another one related to AT&T. It took about 10 minutes more effort from me than a typical class-action settlement, because I had to e-sign those representation papers.

So really, there's a third option now, that's much easier than class action, even when class actions don't get certified.

Weirder is what happened a day later. I got an email that said my Chase Amazon Prime credit card was being re-associated with my Amazon.com account.

I never reported this nor took it up with either Amazon or Chase directly. There was a refund of my Whole Foods purchase (they needed to void my purchase and re-ring everything to give me the discounts.. I asked them to refund my purchase and I’d do without my Whole Foods purchase entirely).

Looking back I think at least 3 recent visits were charged to me at full price because of all this. Hard not to think of enshittification and whether Amazon Prime is even worth it, alas.. I live in a fairly rural area at the moment and need delivery.

I can't think of another regulated utility that doesn't provide service to (essentially) all humans directly in their homes.

Everyone knows what water and electricity are, the vast majority couldn't explain what service AWS provides.

Or just own your own hardware. Spend a few bucks at Microcenter, build a machine, and there's simply no mechanism by which they could decide later that you should actually pay 100x more, and then magically suck it out of your bank account.

None of this can happen unless you first cede control.

Most debit/prepaid cards will get rejected. Credit Cards technically have a limit but they really don't. It's an open line to your finances.

This is why traditional bare metal and VPS rental wins for any kind of private or small business use.

If I were to guess this bug is in the "display" part of the system which is probably distinct from the "actually take money from the customer" part of the system. One can imagine they have gates on the "actually take money" part, especially for a large bill like ours which was ~$300b or about 2.5x AWS' 2025 revenue... In one month. Surely if we had actually accumulated that bill they would be the ones with the problems when we can't pay it.

I don’t know how something like this makes it to prod. That’s multiple levels of failure.

I need to drink/eat those too to feel like I'm an AWS ops guy!

We retro-fitted a Terminator T100 model with the brain of the latest LLM models, and then gave'em 2 shotguns...and, you'll never guess what happened next!

Well, actually i guess you can guess what happens next! lol :-D

How about vibe coded health care systems.

Sure! What could possibly go wrong?

No, but they have the internet in the afterlife, apparently.

Airlines are all subject to a lot of the same factors, but there are unequivocally better and worse performers in terms of on-time arrivals, by a lot. Take a look at the Air Travel Consumer report for details.

No, AWS won't go out of business, afterall, people still use IBM mainframes.

There is an enormous amount of money invested in trying to prove the opposite, for this reason not only they will never admit it, they will actively negate AI was used. It happened before:

https://www.theregister.com/software/2026/02/24/aws-would-ra...

Why would they? Changing colors on the status dashboard needs VP approval.

Reminds me of the decimal point scene in Office Space.

https://www.youtube.com/watch?v=3fGHaVn5rGo

But they're going to try anyway.

They need the customers to pay more so they can fix the bugs. It's self-correcting.

"There are no customer actions required at this time."

Of course, you provided heart stress failure tests for free.

Thanks, I panicked logged in and could not find the root cause of the bill.

Thanks AWS, no caffeine needed this morning!

>Estimated bill updates remain paused

Wait what if someones actually getting usage spiked

THIS IS NOT A MUNDANE DETAIL MICHAEL

$1.7-billion isn't a mundane detail Michael!

Some day this industry's gonna end.

I'm sure some businesses are considering moving back to on-prem, but for many, I suspect the cost to find onboard, and pay the SMEs to keep those systems running well enough to not fail due to one reason or another isn't as appetizing to them as the ability to offload that work, along with the legal responsibility.

When something goes wrong, pointing the finger at someone else is far easier for most than pointing it at yourself.

One thing that you need to understand is that the usual business manager absolutely hates depending on technical expertise, and that the modern corporate world is fanatically anti-intellectual.

Vendor lock-in? compliance and security risks? stupid systems that cost the company an arm and a leg? nobody fucking cares.

Now, depending on an 130 IQ Engineer that basically holds the whole enterprise on his head? Anathema!!!!!!! Bus Factor!!!!