Remix.run Logo
▲ dghlsakjg a day ago

How does this make you less reliant on third parties?

It seems like it just shifts which third parties you rely on. I think local inference is cool and your setup sounds dope, but you still have third party hardware manufacturers, and third party models in the mix.

What is the advantage of this over OpenRouter, which lets me select for, jurisdiction, quantization, ZDR, etc.

note: My use case can't even come close to having this make sense financially, but I understand if yours does. Either way, that doesn't make the third party argument better.

▲runtime_terror a day ago | parent | next [-]

Not the OP but I would assume: - no data retention policies to worry about in the first place - no monthly billing outside of power bill and capex - no jurisdiction issues - full control over your data - no network round trips or even worry about network outages in the first place

▲lrvick a day ago | parent [-]

Exactly. I own the GPUs and bought them all used on ebay. One time cost.

When the internet goes down, my productivity is not impacted, and I have total privacy on my work at all times. Also no rate limiting etc, and I can burn tokens freely.

▲bornfreddy a day ago | parent [-]

Maybe a naive question, but how do you even run 8x GPUs? Threadripper Pro? Anything special about OS / software stack?

Sounds interesting, though a bit pricey (if I'm reading the prices correctly it's 20k just for GPUs). Fingers crossed that the prices will fall.

▲lrvick a day ago | parent [-]

Gave a full parts list as a reply to my original post

▲almondfestival a day ago | parent | prev | next [-]

it's the difference between a one-time transaction with a third party and an ongoing, perpetual third-party liability. you buy your inference hardware, you download your model, and if you wanted/needed to, you could run inference in perpetuity (or until your hardware broke down). compare that to anthropic/openai/etc being able to throttle intelligence, deprecate models, or raise prices on you whenever they felt like it.

sure, in a very general sense you're dealing with third parties in both scenarios, but if you actually own the hardware and model weights, you can control your own destiny for the lifespan of that hardware.

▲dghlsakjg a day ago | parent [-]

> it's the difference between a one-time transaction with a third party and an ongoing, perpetual third-party liability.

no ongoing liabilities except the power bill and opportunity cost of dropping $20k on hardware.

He isn't using OAI or Anthropic models. He is using models that area available for absurdly cheap from completely independent providers in multiple jurisdictions.

I get the base level argument, but from a practical point of view there are so many different dependencies that shifting one isn't really doing much.

▲lrvick 20 hours ago | parent [-]

I use exclusively open source software on hardware I own. I can generate my own electricity if needed. If there is a way to run AI more privately and sovereign than what I am doing, I am not aware of it.

▲drdexebtjl a day ago | parent | prev [-]

Is this some sort of gotcha? You didn’t build your computer from scratch with sand, sticks and pebbles, so you’re just as reliant on third parties?

If you own your own hardware, you don’t have to depend on and trust third parties continually, just once when you bought it.

▲lrvick a day ago | parent | next [-]

This feels a bit pedantic, but my scope here is third party services, like Anthropic or OpenAI subscriptions.

If the power goes out, internet goes down, I fire up my generator and keep building.

We are talking about digital sovereignty and ownership.

I own the means of my production.

▲0xbadcafebee a day ago | parent [-]

Sounds like commie bunk to me! We can no longer sit back and allow Open Weight infiltration, Open Weight indoctrination, Open Weight subversion, and the international Open Weight conspiracy, to sap and impurify, all of our precious bodily fluids.

▲dghlsakjg a day ago | parent | prev [-]

> Is this some sort of gotcha? You didn’t build your computer from scratch with sand, sticks and pebbles, so you’re just as reliant on third parties?

I'm not the one claiming to be free of third party dependencies. Third parties built my hardware, they wrote the firmware in the hardware, they wrote the OS on the hardware, they designed the network stack that you might use to connect to a local server, they provide security updates to all of that software, they ensure that I have a stable power supply to run all this, etc.

I'm incredibly reliant on third parties, some of which happen to sell me tokens. I recognize that I live in a world where I will ALWAYS have a dependency on third parties, and rather enjoy the fact that I can use GLM flash on someone else's $20k computer for a few cents.

▲drdexebtjl a day ago | parent [-]

Yes. They built the hardware. Now it’s mine, and they’re no longer involved, so I no longer depend on them. That’s how it works.