| ▲ | jacquesm 2 hours ago |
| Working on a blog post to document it all, but that will take a few more days. The short version: the big rig is probably $25K, ttft is dependent on how much prefill you have but typically a couple of seconds (2K prefill/sec in one setup, about 500 in the other), tgs 100 to 150 or so depending on what it is doing and how well the speculative decoder works with the current context. Electricity is pretty bad, full tilt it's about 2500W or so but I've power limited it to 1500 to keep it civil. It is not cheaper than using a frontier lab (or at least, I don't think it is), but the independence bit is the important part for me and I don't care if that carries a surcharge, even if it is a substantial one. |
|
| ▲ | gr_norm an hour ago | parent | next [-] |
| The benefit of the agent working firstly for you rather than for Anthropic or OpenAI will become increasingly apparent. We're already seeing signs: remember when Anthropic openly promised to have Claude sabotage people using it to create competitors? And OpenAI is looking for revenue opportunities from advertising, which can't end well for their users. Just think of how user-hostile other online services have gotten, and imagine how acutely painful the proprietary LLM providers could make theirs. Thankfully, open models provide some counterbalance. |
|
| ▲ | glassresearch 2 hours ago | parent | prev | next [-] |
| I'm not saying your approach is wrong - but for my workloads, it makes a lot more sense to just pay for a decade (!!!) of Claude Code 20x Max. |
| |
| ▲ | segmondy 41 minutes ago | parent | next [-] | | My local model never says no. I never get banned. If I want to do security stuff, reverse engine, there it goes! Do I wanna chat about politics? No problem! Do I want it to analyze medical stuff? I don't get to hear that it's not a doctor and I should call 911. Do I want to explore legal docs, no problem! There are lots of things that cloud models will say no to! I read that don't even agree to help with advanced biology and chemistry topic much. I also never run out of tokens, no 5 hr limit. I can run it 24/7. I don't run out of tokens. I can run 16 sub agents at the same time, non stop! 1-4 billion tokens a month? No problem. Any agent harness I want! No restrictions. Privacy all day. My own true system prompt, not something someone has determined, no filters. Furthermore, there are many open weight cloud models that don't have equivalent online. For a while before cloud models got good at math, the best model at math was DeepSeekv2 Math and there was no API. The only way to use it was to run local. All sorts of audio models, image/video models, etc. No multiple subscription needed. Private UI for my family, kids included (I can guard what kids infer). Got a friend that really needs some tokens? I give em access through private VPN. Telling someone that they are paying for decades of Claude for local AI is like telling someone they are paying for decades of public transport when they buy a car. Sure, you are right, but ... | | | |
| ▲ | jacquesm an hour ago | parent | prev | next [-] | | If I want to go out of business that is what I would do. Most of the data I have here could never be shipped to a service provider, I'd be in lawsuits for the next decade or so and die bankrupt. | | |
| ▲ | greesil an hour ago | parent | next [-] | | Okay man I've got to ask, what kind of data are we talking about? Legal? Medical? I've heard that there are legal software platforms like Clio that offer AI products, not sure what just eavesdropping on some lawyers at the bar. | | |
| ▲ | jacquesm an hour ago | parent [-] | | http://www.modularcompany.com/ Companies pretty much give us the full overview of what they have from a technical perspective (and often much more than that), we sign NDAs with investors that have serious teeth in them. The data they share with us would not be disclosed under any other circumstance and even our reports are shared only with the clients. It's a nice little business. I'm not going to risk offending my customers for convenience or to save some money. | | |
| |
| ▲ | an hour ago | parent | prev [-] | | [deleted] |
| |
| ▲ | calmingsolitude an hour ago | parent | prev | next [-] | | LLM subscriptions are heavily subsidized. I’d be surprised if Claude 20x costed the same after a decade. | |
| ▲ | benoau an hour ago | parent | prev | next [-] | | If you know you will pay Claude a few thousand bucks a year a decade adds up too, owning the hardware could just as well end up cheaper because you can sell it or as others have alluded the price to rent could get much worse. | | |
| ▲ | dylan604 35 minutes ago | parent | next [-] | | Would you really expect to be using the same hardware over the course of that decade? One of the biggest drawbacks to using on-prem hardware is keeping up with the upgrades. Leasing upgrade cycles are a thing for businesses, but for someone at home it seems like later in the decade the local system will be showing its age. | | |
| ▲ | jacquesm 7 minutes ago | parent [-] | | The 3090's I have would sell for more than new today... for the next couple of years or so it will be pretty bad on the hardware front. Until there is more RAM manufacturing capacity and NV has a credible competitor I don't think it will get better. |
| |
| ▲ | jacquesm an hour ago | parent | prev [-] | | Heh, the hardware I bought I could easily sell for 2x so you may actually end up making money. Though I hope that that won't last. |
| |
| ▲ | tjwebbnorfolk an hour ago | parent | prev [-] | | The bet we (self-hosters) are making is that once these companies IPO, they will be forced to do whatever they have to do to make money. i.e. price increases, more throttling of subscription non-usage-based accoutns, etc. They are subsidizing all of us to the tune of $billions in losses per month. That can't go on for a decade. Probably not even for another 18 months. I still use all the tokens Claude will give me on the max plan. But I also run deepseek v4 locally for overflow and for... contingencies. |
|
|
| ▲ | selcuka 2 hours ago | parent | prev [-] |
| > It is not cheaper than using a frontier lab Are you comparing to subscription prices or API prices? Because if it's the former it won't last forever. |
| |
| ▲ | jacquesm an hour ago | parent [-] | | True, they're subsidized, I'm not. But I don't have their utilization so I figure it would always be more expensive to run this locally. |
|
