Have you been using an MTP setup? I've been having pretty good luck with the Qwen models with the built in MTP heads via https://mtplx.com at around Q4. My main driver rig is an M5 Max MBP work got for me a few weeks ago. Hitting 1000-1200 TPS prefill on Qwen 3.8-Flash-Next there.

Hahaha. It's an excellent question, though!

I have not spun up my own homelab, but, I have researched it and the electricity costs can be mitigated to a large extent.

One... the GPUs can be massively clocked down during idle state, to the point where the fans can be shut off as well. The machines themselves can be shut down and wait for a magic wake-on-LAN packet if needed.

Two... even under load, the GPU cores can be significantly underclocked with very little performance loss. The GPU and VRAM/HBM clocks are independent, and the GPU is largely bottlenecked on VRAM/HBM so you can just drop the GPU speed. A commonly reported figure I saw was, basically, 350W nominal cards being underclocked to consume "only" 200W under load with ~5-10% perf loss.

This all assumes you're using discrete GPUs and not AIO systems like a Mac Studio which is going to be pretty efficient just by design; they idle at 35W or so and in practice max out at a few hundred watts. I believe DGX Spark and Strix Halo are similar.

I must stress that this is second-hand anecdata here, admittedly, but my understanding is that it can be pretty manageable.

> There is no reason to believe that equivalent level model output will be more expensive in 12 months

It's almost never a drop-in replacement, and having to check and adjust integrations and workflows with new models gets old fast. My task was perfectly solved by the old model, I don't need a newer, "better" one - especially at higher prices ("more cost-effective" my foot). Local models lets one choose a model and freeze the downstream integrations forever, without being forced on the 6/8-month upgrade treadmill by aggressively short, scarcity-driven hosted model-deprecation schedules.

The big providers are losing on average tens of billions a year on these services, so yes prices must go up. Even Moore’s won’t help in the medium-term due to shortages and difficulty/reluctance to vastly increase capacity.

I would think it's the opposite: there's increasing chatter about how long the AI labs will subsidize cheap subscriptions. I suspect that at some point one of them will bump up prices, and the rest will follow. That'll signal the end of cheap subscriptions and prices will steadily creep up.

I'm not saying that local AI is necessarily economically justified right now; but it's certainly quite reasonable to think that in 4 years the subscriptions will be significantly more expensive.

Surely you cannot possibly believe there is no reason.

Don't get me wrong: I hope you are right, and I am generally optimistic about the future of AI.

But do you really think, in a world filled with examples of big software companies repeatedly taking away or hamstringing capabilities we've taken for granted, that you can just count on a big tech company hosting cheap inference on incredibly powerful models forever? Surely we have learned by now that these companies do not exist to provide a public service to us, and the government cannot always be counted on to have the best interests of the citizens in mind.

I mean, how many times have we seen this in just the past decade or two?

- Consistent attempts to pass legislation weakening or banning the use of encryption

- Exorbitant Reddit API pricing (still salty about the death of the amazing Apollo app)

- Google fighting against sideloading on android

- US gov't issuing export control directive to suspend access to Fable/Mythos

- US lawmakers considering ways to regulate adoption of open weight models

- Chinese officials considering restricting overseas access to their most advanced models

I can absolutely see a much more restricted, closed down, and expensive future due to a combination of government regulations (regardless of which nation is doing it) and big companies rug-pulling as the check comes due on all the billions of dollars spent to get here.

I would say tech workers have very weak class consciousness and understanding of power structures. Those who care about not ceeding power is a tiny minority compared to say among MDs and lawyers and their guilds/unions.

> It's quite shocking to me how many experienced, tech-savvy people, who used to care about cookies and ad tracking - are now willingly sending their business strategies, highly confidential contracts, and intimate personal issues to a cloud provider because "it is only $0.0x per million tokens!".

Because there are more privacy guarantees there, depending on the provider. "But what if they violate their contract!" is some pretty tin-foil hat stuff.

How is this any different than a business running their website out of the cloud, assuming you are using a provider with appropriate contractual terms?

You can care about tracking and ads but still be comfortable storing your backups in the cloud, and many have been for quite awhile, even sometimes without encryption - that is totally different than e.g. Meta actively trying to track you and understand your relationship graph and your purchases etc.

i think you overestimate the number of tech-savvy people who ever cared about cookies and ad tracking.

Yeah I use it for hobby only. I would use cloud models to develop new stuff, I don't really care if it gets lost because if I were to publish it it would be FOSS anyway. I hate entrepreneurism and monetisation so I'm happiest being a salaried employee with many hobbies :)

But no way whatsoever I'm uploading my personal files, photos, emails, chats into cloud AI. No way.