With the business model for API based LLMs looking iffy at best it seems like we’re heading back to the “server under your desk” era of IT again.

▲

cube00 an hour ago | parent | next [-]

Considering how all the big players are playing fast [1] and loose [2] with limits, billing [3] and adding undisclosed changes that burn your tokens on autopilot [4], it can't happen soon enough.

[1]: Limits may change without notice, including due to capacity constraints. - https://support.google.com/gemini/answer/16275805?sjid=14713....

[2]: "standard limits" are never defined - https://support.google.com/gemini/answer/16275805?sjid=14713...

[3]: https://tobyonfitnesstech.com/blog/anthropic-refund-scam/

[4]: https://news.ycombinator.com/item?id=48947776

▲

xscott an hour ago | parent | next [-]

Not to mention all the other ways they can screw you:

- Middle of the day, servers busy? Swap to Sonnet while pretending it's still Opus. Many people won't notice, and nobody can prove anything if they suspect.

- Middle of the night, server load is light? Put it into extra thinky mode so it burns more tokens to ramp up the bills. Flip the switch where it gets really pedantic about writing lots of extra test cases and verifying against documentation.

- Demand increases, but don't feel like running more hardware? Switch to low bit quants, but have a monitor model swap back to quality if it can tell you're running a benchmark.

Assuming model capability plateaus (I think it will), token providers will be in a race to the bottom to maximize profits at the expense of quality that's very difficult to measure.

▲

mister_mort 23 minutes ago | parent | next [-]

It all sounds like having to rely on a dodgy housing contractor that wants to steal from you, take shortcuts AND choose the gold-plated options from their supplier friends, and will start doing this the minute you are not on site supervising. You don't do it yourself (because the contractor is faster and stronger than you in many ways) but you can't leave, so you're stuck on the worksite just watching them.

	▲	xscott 4 minutes ago \| parent [-]
		It's worse though, because you can't really watch them at all. It's very difficult to get quantitative numbers for quality. Even within the same model family, same tokenizer, and complete control over the weights and logits, perplexity and KL-divergence isn't really what you want. Now put it behind an HTTP endpoint, and it's just opaque. I've seen local models recognize when the task I'm asking them for is likely to be an artificial benchmark. And any smart company is going to use lightweight models to monitor your sessions. If their sentiment analysis suspects you're close to cancelling, they'll up the knob for a few days until you calm down. Or worse, their accounting tells them that you're getting too much value from your fixed price subscription, so they turn the knob down to encourage you to cancel. In the short term, the "frontier" models are too good to ignore. But if (when?) that plateaus, I don't see how anyone could trust a non-local model. When you pay an ISP to serve your web site, you can tell if they over-compress your images to save storage and bandwidth. With LLMs, it's just JSON with more errors and pointing to the fine print that models are not deterministic.

▲

wolttam 10 minutes ago | parent | prev | next [-]

What areas do you think model capability will plateau in, and why?

▲

dannyw an hour ago | parent | prev | next [-]

These kind of tricks will completely break API customers (although selectively applying it to consumer subs is probably beyond likely at this point).

▲

htrp an hour ago | parent | prev [-]

Ugh... didn't think about extra thinky mode in the middle of the night.

So many ways for enshittification here.

▲

12 minutes ago | parent | prev [-]

[deleted]

▲

gdhkgdhkvff 6 minutes ago | parent | prev | next [-]

Why do you say API based llms looking iffy at best? Do you just mean current profitability due to market pressures from some companies’ subsidized investor money?

Surely, even if you’re just using open weights models, it should theoretically be cheaper to use them in a highly optimized cloud architecture(even with vendor markups) rather than each person serving their own models from much less efficient (and more importantly, much less consistent volume) self-owned “server under your desk”?

▲

staplor 11 minutes ago | parent | prev | next [-]

What do you mean iffy? The major AI labs are gross profitable when selling access to inference. In addition, the best models have trillions of parameters and are most efficiently served on large, expensive clusters and served to many concurrent users.

	▲	wolttam 8 minutes ago \| parent [-]
		They make money on each token when you look at the electricity and interconnect fees, but no, I don’t think they’ve turned a profit on their Capex, even a little bit

▲

skohan an hour ago | parent | prev | next [-]

I've been coding using the LLM server in my living room for the past few weeks, and I haven't had this much fun with tech for ages

	▲	exe34 4 minutes ago \| parent [-]
		Can I ask, do you feel the pain of the level of abstraction? I haven't tried local in a few months, but last time I tried, I felt like I was directing a coding exercise - whereas with a frontier model, it feels more like directing a product building. "I need this feature", vs "write code to do this in this file".

▲

drob518 24 minutes ago | parent | prev | next [-]

That’s part of it. There’s also just a natural back-and-forth between what I call “time sharing” and “personal.” When the thing you want is expensive, you share it remotely, but as soon as costs fall, everyone wants it under their desk.

▲

Der_Einzige 31 minutes ago | parent | prev [-]

With how expensive consumer hardware is and will continue getting (due to LLM demand), good luck getting a "server under your desk" for something less than an arm, leg, and first born.

Until A100 prices are reliably under 1.70$ an hour, there is no GPU/AI bubble and Michael Burry doesn't know anything about GPUs.

	▲	xscott 18 minutes ago \| parent [-]
		There are lots of points in a spectrum of choices. DGX Sparks, Strix Halos, and the surviving Mac Studios can easily run these 30B class models, just not as fast. So maybe just the leg, but you can keep the arm and first born. And super noteworthy is that a 27B model (Qwen 3.6 27B) from this year is a huge improvement over a 120B model (gpt-oss:120b) from last year. The goal posts are moving, but at some point "good enough" is good enough for the kind programming I like to do.