About six years ago we built a ticket classification/routing model and pipeline [1]. The initial version was trained using SVMs, it took a couple of minutes to train, and the ticket classification was just a few milliseconds. It's so fascinating to watch how the field moved over to LLMs, and then back to faster/cheaper classifiers again.

[1] "Linnaeus: A highly reusable and adaptable ML based log classification pipeline", https://arxiv.org/abs/2103.06927

NOTE: the work was actually accepted at the International Conference on Software Engineering (ICSE) in 2021.

At ~$21,700 saved per million tickets, saving millions annually would suggest at least 92 million tickets a year to reach $2 million saved. Is that about right?

I initially thought of tickets as just customer support, or internal JIRAs, but the article mentions alarms, security findings, and capacity increase requests, so that makes the scale feel more plausible. Still surprising so many tickets go through the system.

Founder of Seldon AI (www.seldon-ai.com) here. We are working on dev tools to help developers with this kind of refactoring and continual production work.

Basically there is a whole host of tarpit usage of LLMs in application backends today. This is simply another case of LLMs enabling people to build what they cannot productionize or maintain.

Today developers without any AI/ML skillset can prototype a fancy new feature with LLMs, what used to require collecting and curating data to train a pipeline of ML models then serving them in production.

The problem comes when it's time to take the new feature into production. That $20 cost with prototype/beta workload is now $30k per month in LLM costs, and those hallucinations that never happened on their 200 test samples is happening 600 times per day.

They are then stuck, because they lack the knowledge and budget to build and run the production ML pipeline properly, the infra stack necessary to handle production workload. Management also doesn't understand why they have to invest in rebuilding something that "seems to work" with "outdated methods".

These system one or decision models are a step up from LLMs when it comes to this usage. Problem is that they are still difficult to work with (limited in expression power) for many application logic. Despite the initial popularity, I hear about a lot of frictions refactoring LLM based backend with them, and multiple cases of production rollbacks to LLMs already.

I think system one models is a sensible paradigm, and we are on the precipice of something great, but there is a lot more work needed above the model layer to make this work.

This article is so confusing. For example, they say they tried the Strands Decider model and dismissed it for Gemma, then they continue to call the model they used Strands Decider? Did I miss something?

Ok, but why did they chose Sonnet in the first place? Couldn't they have chosen Haiku or GPT Luna or maybe even some small cheap Chinese model?

So why did they not even try Haiku 5.5?

Why would Amazon go to another company to let them provide an adjusted decider model?

Amazon already has all these tickets labeld, they could just have build it themselves?

> JSON key order changed the answer. Sending the same content with keys in a different order moved the output probabilities by up to 0.19

Isn't it detrimental?

You can do this even more efficiently with the rust crate bonsai. You will have more control and you can create sceneries of logic. You can use incredibly cheap models this way and get way better results.

I built a real time character story. In which players interact and integrate fictional characters. There is no delay in response and the characters are all in character. Oh and it’s all voice.

It’s shocking how effective this is.

Classifying into one of 20 groups? Did anyone try scikit learn?

People use machine learning models before and after LLM is invented.

50x cheaper, damn. I recall Chinese LLMs being similarly cheaper (and quite suitable for such simple tasks).

I think a decision model is superior here based on the task, and of course they weren't out yet when they started, but I'm wondering if they wouldn't have similar results (on price, probably not on speed) if they went with DeepSeek or something?

I don't mean send the data to china obviously — there must be an American provider that offers it?

Anyway it's moot in this case (decision model is a better fit) but it just makes me wonder who else is overpaying by 2 orders of magnitude.

What I don't understand is why are they not offering the decision model via Bedrock yet. Sagemaker / Lambda are quite expensive if you're not using the model continuously.

As a customer I don’t care what model they are using. Amazon is one of the worst e-commerce platforms (at least here in Germany) - damaged boxes, damaged items, items taking forever to ship, having to deal with rude delivery people just to name a few.

This is perfect example of how to not approach solving business problems. What do they actually need? Listening to customers.

What are they doing? Outsourcing delivery to third parties, lowering quality controls, creating worst working conditions, filling the platform with sponsored listings and ads, allowing any sort of cheap Chinese garbage with fake 5 stars on the platform, making the UI worse and worse everyday, and the list goes on and on…

This pattern is playing out a lot. Running LLM inference at scale on apps typically lacks ROI. But I increasingly see folks use LLMs to help improve classical techniques.

So there’s some LLM use to develop a new version of algo, but maybe 2% of what would have been used by putting LLMs in the path of every API call. That generally does have ROI.

Putting LLMs in the loop each time is generally a dumb move, unless you’re an AI bro trying to get everyone to use LLMs for all the things.

Are developments like that priced in Anthropic IPO?

It's funny how AI sometimes feels like it went full circle:

— "Let's build efficient classifiers"

— "Forget about the silly classifiers, LET'S BUILD A MACHINE GOD"

— "Ah, that machine god is often too expensive and underwhelming. Let's build efficient classifiers"

> One thing that surprised us was that JSON key order changed the answer. Sending the same content with keys in a different order moved the output probabilities by up to 0.19.

That's like... really basic LLM knowledge? Like how e.g. this article measures how often answers change based on order https://zachlandes.com/trolley-problem/#appendix-a:~:text=Ch...

Or is this the Claudish "One thing that suprised us" that just means "and here's another thing"?

Amazon saves millions by asking their pencil manufacture to make them 1mm shorter. At that scale anything adds up to millions.

Yea right... Paying market price of 3 usd per mil tokens on sonnet 4.6...

Sounds like a bunch of `if` statements with extra steps. Maybe throw an RNG decider too.

Meanwhile they could have just switched to Haiku 5.5 to get same savings. No need for complex orchestration involving GPU instances and would have just been a one line change in the code.

Nice marketing to sell their decision model.

yah i mean older models dosent mean they are worthless i still use cluade opus 4.6 for day to day task

counting the miliseconds until the insurance industry picks this up

why didn't they use Jev or Luna decision model? Seems odd that they used Sonnet and Strands instead of trying out of the box solutions

If their model costs 98% less than Sonnet, and Amazon is saving “millions” then these folks are raking in 2% of millions. Congrats I guess?

why didn't they upgrade to further sonnet versions?