Remix.run Logo
yuchen20 a day ago

I got 3 consecutive emails warning that my budget crossed its $18 threshold. Opened it up: cost was 78 million. Thought it was a phishing attempt, logged into my actual account, and... still 78 million. EMOTIONAL DAMAGE.

root-parent a day ago | parent | next [-]

Wanna bet the description of this job post will be updated by the end of the day?

"Software Development Engineer II, AWS Invoicing"

https://www.amazon.jobs/de/jobs/10428480/software-developmen...

"...Our platforms are powered by generative AI, large language models, knowledge graphs, and agentic architectures that dynamically compose specialized agents based on context. We apply these capabilities across three reinforcing areas: intelligent launch readiness — where autonomous AI agents analyze, generate, and validate the information needed to go live in a new market; cloud-native service orchestration — where configuration-driven microservices replace per-launch bespoke engineering with centralized, reusable capabilities so that expanding into a new country becomes a zero-code configuration change rather than a development cycle; and continuous validation..."

ibejoeb a day ago | parent | next [-]

Wow:

    In this role you will:
    - Design and build agentic AI systems that analyze, generate, and validate...
    - Build agentic architectures that compose specialized AI agents dynamically...
    - Build AI-driven continuous validation frameworks powered by agentic workflows and large language models that autonomously manage...

This is invoicing? If ever there was a domain that was purely deterministic, you'd hope it was invoicing.
golly_ned 21 hours ago | parent | next [-]

When I was at AWS, they famously required an extensive "CoE", correction of errors, or post-mortem, in an instance of over-charging a customer $0.26.

The idea is that if we can make small billing mistakes like that, we can make large billing mistakes, and need to invest in the correctness of the systems powering billing.

I have great respect for the engineering culture within AWS during those times. I am glad to have left before seeing it degrade and decline.

root-parent a day ago | parent | prev | next [-]

The irony is, the only purely deterministic thing, will be token consumption...

jdiff a day ago | parent | next [-]

I severely doubt the world ever gets to such a point that the entire world melts into AI hallucination. And token consumption depends on so many other things, it's not all that deterministic either.

serf a day ago | parent [-]

(token usage) is trending towards predictability for a lot of reasons. it's not deterministic but it's getting easier to reason about usage.

LetsGetTechnicl a day ago | parent | prev [-]

How can a random generator be deterministic?

johnbarron a day ago | parent [-]

Truth is, you can never know: https://imgur.com/random-number-generator-bwFWMqQ

rpunkfu 9 hours ago | parent | prev | next [-]

It doesn’t matter which domain it is, AI should be utilized at anomaly / patterns detection and prototyping, with goal of a prototype to become coded as discovered pathway.

I don’t understand why companies still place AI in their system at authority-forming level, if anything it should sit at least behind single guard / validator that is explicitly coded and tested, never in front of presentation layer.

curun1r a day ago | parent | prev | next [-]

I’m not so sure about that. I can see a real rationale for creating sanity checks using AI to more quickly/proactively catch pathological billing issues before they become HN nightmare stories. They wouldn’t replace billing code, but there are many ways that stupid customer mistakes can cause real costs to Amazon that either have to be refunded and absorbed by Amazon or paid by the customer causing a negative opinion of AWS. If a billing AI watching costs in realtime could detect, say, a lambda loop in the first 10 min and either alert the customer or kill it, that would make AWS feel a lot safer to use. Enumerating these conditions and fixing them individually is a task that Amazon has proven incapable of achieving. An AI watchdog layer might be the perfect shortcut to addressing all of these problems at once. Because it’s well-trodden territory that AWS has so many multi-thousand dollar foot guns that make it really scary to use as a hobbyist or small business on a tight budget.

hvb2 a day ago | parent | next [-]

> I can see a real rationale for creating sanity checks using AI to more quickly/proactively catch pathological billing issues before they become HN nightmare stories

Right, so invoicing is still a deterministic problem. You can bolt whatever on but in the end it's just product x price x units

skywhopper a day ago | parent | prev [-]

This is exactly the sort of thing that’s not possible, though. An AI will not be able to detect a “lambda loop” because it will look exactly like a “successful lambda rollout”. This sort of watchdog would just as likely shut down the wrong things and make AWS feel a lot less safe.

TJSomething a day ago | parent | prev | next [-]

This is like half of all job listings I've read recently. And it's a decent amount of fintech that's like this.

londons_explore a day ago | parent | prev [-]

Probably not actually. Transferring one kilobyte across a network link has such a low value that the billing costs of aggregating it cost more than the revenue.

So instead you take a probabilistic approach - charge the user for a megabyte of data transfer 0.1% of the time, and bill nothing 99.9% of the time.

Now the typical cost is the same, the users bill is probably accurate to the cent, but you have divided the number of billing records by 1000.

svobodovic a day ago | parent | next [-]

I don't know how cloud services count usage, but this is certainly not true for telco. I manage several fleets of hundreds/thousands of SIM cards (mostly IoT/M2M applications), and almost every provider counts the data traffic per byte. Different business and use case, I know, but still.

michaelmrose a day ago | parent | prev [-]

The way you describe requires somehow counting every bit but somehow discarding most which is obviously nonsense.

This seems statistically invalid insofar as it will tend to overbill potentially by a lot on the minority of cases.

Don't you know how much of the pipe is occupied by a given customers code at any given time or what data is being sent

londons_explore a day ago | parent [-]

You have to do it when the customer list is too big to keep a counter per customer.

fc417fc802 a day ago | parent | next [-]

A probabilistic counter per customer is also a counter per customer. Still, probabilistic billing is an amusing thought though.

michaelmrose a day ago | parent | prev [-]

No you don't

blitzar a day ago | parent | prev | next [-]

> 194,400.00 USD annually

Fuck it, im in.

TheOtherHobbes a day ago | parent | next [-]

Just wait until the same system runs payroll and you're getting paid $1.94400 annually.

blitzar a day ago | parent | next [-]

I will just tell the HR bot that I am meant to be paid 1.944 billion.

tmpz22 15 hours ago | parent [-]

And if the HR bot is anything like Facebooks customer support - itll just let you do it!

O4epegb 8 hours ago | parent | prev | next [-]

Or alternatively $1.9 billion

sgarland a day ago | parent | prev [-]

10/10, no notes.

fragmede a day ago | parent | prev [-]

That's base, don't forget bonus and stock.

sebmellen a day ago | parent | prev | next [-]

That job description feels so far beyond parody that I could scarcely believe it until opening the link! What a world.

root-parent a day ago | parent | next [-]

It gets worst:

"Senior Software Development Manager, AWS Global Bill Generation" https://www.amazon.jobs/de/jobs/10471948/senior-software-dev...

"We're transforming from monthly batch processing and manual war rooms to continuous billing, autonomous agents, and self-healing infrastructure. We believe operational burden is a technical problem, not a staffing problem"

This looks clearly...a staffing problem...

ghurtado a day ago | parent | next [-]

> This looks clearly...a staffing problem..

I think that big tech recently decided that I got 99 problems but staffing ain't one

I guess Nothing is a staffing problem when you make a rule that firing people is always the solution.

wbl a day ago | parent | prev | next [-]

If you can make the software cover the toil you save the staff for the tough cases.

quickthrowman a day ago | parent | prev | next [-]

They need to fire whoever is running AP and AP software development. Vibe invoicing is ridiculous for anyone to do, let alone Amazon.

a day ago | parent | prev | next [-]
[deleted]
iam-TJ a day ago | parent | prev [-]

The best bit of that is:

> In this role, you will own end-to-end bill run execution across all AWS partitions, drive the technical vision for autonomous billing operations, and build the team that ensures every customer receives an accurate cost estimated in minutes ...

LPisGood a day ago | parent | prev | next [-]

Seriously! If I were making a joke I would say something like

> Build AI-driven continuous validation frameworks powered by agentic workflows and large language models that autonomously manage…

But that’s word for word a 250k+ TC job in the big ‘26.

paganel a day ago | parent | prev [-]

> enabling domain experts to review in hours what previously took weeks.

This is a gold-mine. They need to get sued heavily for this incompetence.

redbell 18 hours ago | parent | prev [-]

IRE which stands for Invoice Reliability Engineer.

shawkinaw 20 hours ago | parent | prev | next [-]

Mine was just over $1 billion. I thought the email was a phish at first and almost trashed it, but it seemed otherwise very real, so I logged in to AWS just to check. Then I almost had a heart attack when I looked at my billing estimate. Eventually I saw a banner on the support page, but my question is why wasn’t there a banner on the page SAYING I OWE A BILLION DOLLARS?!

antognini a day ago | parent | prev | next [-]

To paraphrase the old joke, if you wake up with a $78,000 AWS bill, you have a problem. If you wake up with a $78 million AWS bill, Amazon has a problem.

rcleveng a day ago | parent | prev | next [-]

I did too, those awstrack.me URL's look super suspicious and I hadn't seen this alert trigger before so didn't know what to expect.

At first I was sure it was a phishing attempt. Then went to the console (not using those links) Saw there was an outage where the console was wrong (no mention of email alerts) Then I thought I was hacked - what a perfect cover up for someone to evade detection when the console was wrong. Looked at some logs, realized the incident text was just not exhaustive on the impact. Went back to my cup of coffee.

Note to self- should have looked here first.

jayanmn a day ago | parent | prev | next [-]

Enterprise account . We got - 3trillion and change

chii a day ago | parent | next [-]

-$3 trillion! That's the highest earning investment that has ever existed!

theflyingelvis a day ago | parent | prev | next [-]

3.7 billion. Offered to pay it in monthly installments. Haven’t hears back

idiotsecant a day ago | parent | prev | next [-]

Quick do your IPO before the books update

munk-a a day ago | parent [-]

It's true, if you're spending that much money you must be worth a ton! Just look at SpaceX!

rconti a day ago | parent | prev [-]

same. over 2t in one day.

jerf 18 hours ago | parent | prev | next [-]

The real fun comes later when the US Federal Government taxes us on the forgiven debt as income.

kjs3 16 hours ago | parent [-]

The finance fun noone knows about. I knew someone who did this as revenge. Sent some asshole a fake invoice for US$stupidmoney for a couple of months which said asshole ignored, then sent a certified mail 'I forgive asshole US$stupidmoney', and copied the IRS. IRS went to collect on the unpaid taxes, and if you've never been there, the IRS does not play when it comes to that sort of money. I don't know how it ended, but there were multiple lawyers involved, so it was expensive no matter what.

01284a7e a day ago | parent | prev | next [-]

Yes, I am taking legal action, no doubt.

bot403 a day ago | parent | next [-]

Why? What's the damages? They showed you a wrong number, then later acknowledged it and fixed it. Just because the number was "very big" to you doesn't mean you were actually aggrieved in some way.

amelius a day ago | parent [-]

Big numbers can lead to stress which can lead to all kinds of disorders.

TheSoftwareGuy a day ago | parent | next [-]

In order to successfully sue for emotional damages, you have to prove quantifiable damages. Usually that means if you ended up having to get therapy to deal with it, you can sue for the cost of the therapy.

Cthulhu_ 11 hours ago | parent | prev | next [-]

Can, but did they? Can you prove it to a judge? etc

mito88 a day ago | parent | prev [-]

small numbers too....

:)

fc417fc802 a day ago | parent [-]

We should just go with numbers in general in order to play it safe. The new guidance from legal is absolutely no numbers on invoices for liability reasons. Similar to the removal of actual math from math class we're going to let the experts figure out how to implement it.

dymk a day ago | parent | prev [-]

…for emotional damage?

inigyou a day ago | parent [-]

If you were a business maybe you could claim for the emergency on-call time spent diagnosing, but you'd probably still lose AND amazon would fire you as a customer.

corvad a day ago | parent | prev | next [-]

Please do not open up emails directly always login to your account.

SegfaultSeagull a day ago | parent | prev [-]

Time to get a second job buddy.