Remix.run Logo
▲ lrvick an hour ago

I run 8x r9700s at home which lets me run up to 2x 1M context sessions of GLM 5.3 flash at ~50tps or 12x sessions of qwen 3.8 flash next at ~100tps.

I exclusively use this setup for all AI usage from fixing obscure reproducibility bugs in software that once took me weeks, to compiler bugs, to kernel hardening and hardware drivers, to building a complete unprivileged and security hardened replacement for systemd in rust. Most of that was done using Qwen 27b back when I had only 2x r9700s.

There is literally not a single problem I have faced in 25 years of systems engineering that I cannot solve now 10x+ faster using my local LLM setup.

I turned down the free OpenAI and Anthropic tokens in my YC batch because I have absolutely no interest in being reliant on third parties to do my job.

▲back_to_basics an hour ago | parent | next [-]

>I turned down the free OpenAI and Anthropic tokens in my YC batch because I have absolutely no interest in being reliant on third parties to do my job.

Which is exactly why said power and utility will soon be BANNED under a myriad of false aegis'.

An economy that almost exclusively extracts ECONOMIC RENT from the working class, won't permit any "leakage" or autonomy.

▲agcat an hour ago | parent [-]

Did you turn down this for internal use-case? OR for your core product? I am keen to learn how are people running stuff locally for their core product to be able to make it easy for their customers to use.

▲lrvick 38 minutes ago | parent [-]

I have no interest in having my business or my personal AI usage being reliant on any single third party. We currently have no customer facing LLM products, but if we did they would be only options we can run confidentially which no providers can do ATM to our standards. Current plans are just racking GPUs like I did at home.

Most of my peers do use OpenAI and Anthropic for development work, which pains me, but I will never tell someone else how to do their job if the end result meets my quality bar.

I lead core systems engineering building custom enclaves and operating systems and I do 100% of my work with hardware I own on my own property. When the internet goes down, I do not care and keep building.

▲MomsAVoxell 23 minutes ago | parent | prev | next [-]

I exclusively only use local models. I do not subscribe to anything, and I have my agents wrapped and inspected.

2xDGX Spark cluster is my current load, it is sufficient for a great deal many wonderful, wonderful things. It isn't fast as consumer AI, but I have nevertheless been able to do wonderful things with it for the past year'ish.. I'd get a couple more, but indeed they've tripled in price, so yeah .. things will just do for now, and it's fine. Pretty great, actually.

I think the way we control AI is we air gap it. My cluster is air-gapped.

▲dghlsakjg an hour ago | parent | prev | next [-]

How does this make you less reliant on third parties?

It seems like it just shifts which third parties you rely on. I think local inference is cool and your setup sounds dope, but you still have third party hardware manufacturers, and third party models in the mix.

What is the advantage of this over OpenRouter, which lets me select for, jurisdiction, quantization, ZDR, etc.

note: My use case can't even come close to having this make sense financially, but I understand if yours does. Either way, that doesn't make the third party argument better.

▲runtime_terror 38 minutes ago | parent | next [-]

Not the OP but I would assume: - no data retention policies to worry about in the first place - no monthly billing outside of power bill and capex - no jurisdiction issues - full control over your data - no network round trips or even worry about network outages in the first place

▲lrvick 36 minutes ago | parent [-]

Exactly. I own the GPUs and bought them all used on ebay. One time cost.

When the internet goes down, my productivity is not impacted, and I have total privacy on my work at all times. Also no rate limiting etc, and I can burn tokens freely.

▲bornfreddy 3 minutes ago | parent [-]

Maybe a naive question, but how do you even run 8x GPUs? Threadripper Pro? Anything special about OS / software stack?

Sounds interesting, though a bit pricey (if I'm reading the prices correctly it's 20k just for GPUs). Fingers crossed that the prices will fall.

▲drdexebtjl 37 minutes ago | parent | prev [-]

Is this some sort of gotcha? You didn’t build your computer from scratch with sand, sticks and pebbles, so you’re just as reliant on third parties?

If you own your own hardware, you don’t have to depend on and trust third parties continually, just once when you bought it.

▲lrvick 23 minutes ago | parent [-]

This feels a bit pedantic, but my scope here is third party services, like Anthropic or OpenAI subscriptions.

If the power goes out, internet goes down, I fire up my generator and keep building.

We are talking about digital sovereignty and ownership.

I own the means of my production.

▲0xbadcafebee 4 minutes ago | parent [-]

[delayed]

▲txdv an hour ago | parent | prev | next [-]

whats electricity bill and do you not fear that your hardware will be obsolete while others will have access to better models on faster hardware?

i just calculated that it should be Per month (30 days) 380euro in my region, which is a lot after an initial 20k.

I have to say, no limits stopping sounds nice though.

▲lrvick an hour ago | parent [-]

> whats electricity bill

$300/mo in the bay area for 1kwh running 24/7, at least until I get solar installed, then free.

> do you not fear that your hardware will be obsolete

If it never gets better than it is right now, I cannot think of any problems in software I cannot solve locally, so I am all set for a while.

Though also, the capabilities on my -existing- cards have gone UP dramatically in the past 6 months since as the local LLM community keeps optimizing.

Sure in 3 years all these cards will be a lot cheaper than I paid, but having them for the next 3 years is absolutely worth it for me.

This costs me less than a used car. Most people chase car and phone upgrades every 2 years. I tend to get something good enough and upgrade once a decade. Depends on your goals really.

▲wez470 an hour ago | parent | prev | next [-]

How cost effective had it been? I don't want to rely on 3rd parties either, but my experience is it's super expensive to run locally versus using a provider (both in terms of hardware costs and ongoing electricity usage).

▲lrvick an hour ago | parent [-]

No third party can ever provide the privacy and sovereignty of being able to run locally with abliterated models. I would do it this way even if the providers were much cheaper as I value those properties a -lot-.

In practice though, I can produce about 3B tokens/month of Qwen 3.8 Flash Next which is the model I crank almost 24/7. That would cost something like 1k/mo on OpenRouter. My setup costs about $300/mo in electricity, though getting solar soon to drop that to zero.

▲cortesoft 42 minutes ago | parent [-]

It doesn’t really drop the cost of electricity to zero; you have to amortize the cost of the panels over their lifetime, plus the opportunity cost of the capital outlay (or interest on the loan)

▲lrvick 36 minutes ago | parent [-]

That is fair. Still, dramatically reduced cost and reliance on one less third party.

▲random3 an hour ago | parent | prev | next [-]

Can you share more, or direct us to more resources around local setups? For starters what's the upfront cost, what are the space, weight, energy requirements, etc.

▲lrvick an hour ago | parent | next [-]

Really depends on what GPUs you go with but I hate proprietary software so I went with AMD AI PRO r9700 cards which you can still pick up used on ebay for $1500 used on auctions. Started with 1 then expanded over time as I won auctions. Though on a lower budget my $400 16gb 6800 XT can run Qwen 3.8 27b which I could absolutely get by with at significantly reduced speed.

My entire 12u rack has roughly the same capability as an H200, though at maybe 15k all in with mostly used parts vs 50k for a used H200 checking todays ebay prices and that is -just- the card you still need to build a system for.

Consumer cards are a lot more power hungry than enterprise cards and they take up more space. I run my setup next to my dryer, running on 240v for 15% better power efficiency.

Also using transparent memory encryption on a Xeon and an EPYC for confidential and private inference guarantees no third party can offer me. Of course I can run abliterated models that do any security research I want without refusals.

Strata and Radiance are the state of the art ATM for running big MoE models on small consumer GPUs at high performance.

▲Leif24 8 minutes ago | parent [-]

Re: running qwen 3.8 27b on a 16gb card - that isn't a MoE model, was it just heavily quantized? How much context window were you getting?

Just starting to dive in on local AI with my own 16gb card, and the struggling with evaluating what tradeoffs are worth it.

▲vablings an hour ago | parent | prev [-]

There are tons of resources out there on how to run a local setup. Once you have something that's somewhat functional you can then have the system self optimize your software stack to suit your needs.

As for cost a 9700 is running at about 2k USD MSRP right now and the rated power is 300w.

▲ an hour ago | parent | prev [-]
[deleted]