I think that's overly pessimistic. Here's [1] a video of somebody running it on a ~$6000 rig and getting around 14T/s for complex prompts (about double that for simpler prompts). Payback time is going to depend on your electric cost/consumption. In most domains cloud providers end up charging a significant premium rather than a offering a scale enabled discount, relative to local at retail costs. That will almost certainly end up being the case with LLMs as well, if it isn't already.

Furthermore we continue to follow the path that image gen neural networks took. In that domain hardware requirements reached a peak and then started sharply declining to where we are today where a plain old video card can rapidly generate images that took a supercomputer not that long ago. So it's reasonable to assume that performance of such a system could potentially even increase over time.

[1] - https://www.youtube.com/watch?v=ZWS2JVN2iBI

I think the biggest reason is to own the stack so your model can't be changed out from under you, but maybe I care about that too much.

> It is absolutely not worth buying hardware to run models for purely (long term) cost reasons

This is especially true when it's trivial to have the LLM itself write you a script/tool that can rent a GPU node for you (via API calls to providers) and then download and set up an open weight model for you.

I'm actively uninspired to write high quality code when using Anthropic/OpenAI models given the high chance I'm a customer as well as used as dataset generation tool for them.

But currently cloud does beat costs of hardware ownership, particularly with ridiculously high RAM/GPU/SSD costs....again due to these same companies.

I mean, I think it depends. At home 3 of us we use AI for multiple reasons, from coding apps to asking general questions, and if we would have to pay equivalent subscriptions that would be ~1k a year on AI + submitting all your data to external services. I payed around ~8k on 2 DGX Sparks that, at the moment, serves perfectly fine as a ChatGPT/Claude replacement at home (DS4 Flash peaking at ~170 tokens per sec with 6 concurrent sequences), and even once the technology is obsolete for inference in a few years, I will still have 2 pretty powerful machines for whatever I need + some pretty fast NVME Storage. I don't think its a terribly bad idea.

And compliance.

It seems absurdly naive to rely on "oh, the cloud AI of the future will definitely be as open and priced the same way it is right now."

And not "Hey, these companies have a history of giving you something nice now, and rugpulling you either in quality or price later."

Your "absolutely" seems silly.

I think the privacy argument that keeps coming up is overrepresented. Certainly ZDR is enough for an absolute majority of use cases? I see so much talk about local inference but I doubt most of it has privacy as a valid argument (not arguing it doesn't exist). It's fun to do things locally though. I've tried it as well but cloud is just faster and cheaper.

Solar panels are produced for less than $1 per watt btw. Shame they're illegal in Texas because they compete with the governing oil industry.

Too hot and expensive to run right now but a great hedge for peace of mind against $200 subscriptions shooting up to the $4000* they should cost.

*$1000? $14,000? Who knows but everything in the middle there has been claimed.

I wish Texas would write up a regulation allowing 'balcony solar' as I could easily generate 1000-2000w of solar in my small back yard to take a bite out the sizeable cooling bill I have.

Cool! I'm thinking about a local set up. What's your usual tokens/second rate?

OpenAI have only just announced it and have every reason to hype it up.

Could be a long time till gets released

And Tenstorrent greatly exceeds it, but how will you actually get one of those cards?

> Tools vs services in my mind. There is no guarantee any provider will continue to do what they are doing for you at the price they are doing it.

with open models, there is ecosystem/market of providers, where you can easily switch to provider you like

With competition we kind of have guarantee up to what providers can do, they don't have that much control, the most radical thing they can do is to go bankrupt.

Do You have guarante any electricity price?

Or if you need stuff that APIs don't / can't provide. Or for future proofing your workflows. Running things locally gets you "the same thing" in perpetuity, while APIs might change, models can be deprecated and features can be removed.

Cybersec is also hit and miss, depending on what provider you choose, verification systems and all that jazz. Also, running locally allows you 100% data privacy, in any situation and for whatever usecase you might have. ~100k for hardware for a small team of devs to code locally is not that expensive in the grand scheme of things.

Lastly, local models allow for training / finetuning on your own data and processes. $/tok is not everything for everyone. Sometimes you can take a hit on value / speed if you get something else that matters for you.

> Part of it is knowing that whatever sort of enshittification the cloud providers do, my local programming environment won’t ever be less effective than it is today locally.

Is that true though? Many of the core LLMs need to be retrained as languages evolve to incorporate changes (language specifics, compilers, tooling, etc.). To some degree this can be handled via context injection in a variety do forms (agents looking up documentation and so on) but inevitably it’s not stationary in time, just as your OSS stack (probably) isn’t (depending on the languages, technologies, and use cases).

So your hardware is to some degree dependent on the good merit of groups like Z or Alibaba or whomever pushing out updated open weight models that dumped loads of capital into to train. You can keep using the existing models but at some point I suspect they’ll start to have more friction due to dated specs in language and so on. Again there are tuning and ways of layering this information on, and in theory you can even do some training on your own but I don’t think it’s as stationary as being portrayed here.

Those updated open weight models may not always be there (updated on new data). The usability of them is probably fairly long to be fair, but I suspect you’re going to see explosion in everything from libraries to languages etc due to LLMs so even the rate of change across your OSS stack may cause these models to be dated quite quickly, at least in the core model which will require layering fixes.

To be clear I’m on the fence thinking about much of the same issues and as close as I am to pulling the trigger, I keep thinking of very valid counter arguments as to why it’s me just wanting this thing I own. Which may be enough.