| ▲ | teaearlgraycold a day ago |
| The nice thing is unlike Cloudflare or AWS you can actually host good LLMs locally. I see a future where a non-trivial percentage of devs have an expensive workstation that runs all of the AI locally. |
|
| ▲ | breatheoften a day ago | parent | next [-] |
| I'm more and more convinced of the importance of this. There is a very interesting thing happening right now where the "llm over promisers" are incentivized to over promise for all the normal reasons -- but ALSO to create the perception that the "next/soon" breakthrough is only going to be applicable when run on huge cloud infra such that running locally is never going to be all that useful ... I tend to think that will prove wildly wrong and that we will very soon arrive at a world where state of art LLM workloads should be expected to be massively more efficiently runnable than they currently are -- to the point of not even being the bottleneck of the workflows that use these components. Additionally these workloads will be viable to run locally on common current_year consumer level hardware ... "llm is about to be general intelligence and sufficient llm can never run locally" is a highly highly temporary state that should soon be falsifiable imo. I don't think the llm part of the "ai computation" will be the perf bottleneck for long. |
| |
| ▲ | lwhi a day ago | parent [-] | | Is there any utility in thinking about LLM provision in terms of the electricity grid? I've often thought that local power generation (via solar or wind) could be (or could have been) a viable alternative to national grid supply. | | |
|
|
| ▲ | PunchyHamster a day ago | parent | prev | next [-] |
| I'd imagine at some point the companies will just... stop publishing any open models precisely to stop that and keep people paying the subscription. |
| |
|
| ▲ | lxgr a day ago | parent | prev | next [-] |
| I’m fairly sure you can also still run computers locally and connect them to the Internet. |
| |
| ▲ | irishcoffee a day ago | parent [-] | | Ah, you need to buy into this dystopia wholesale. The internet is also down because the LLMs fucked up the BGP routing table, which congress agreed (at the time) should run through the LLM interface. Imagination, either the first or last thing to die in 2075. | | |
| ▲ | lxgr a day ago | parent [-] | | Congress administrating BGP? Now we’re talking dystopia! | | |
| ▲ | irishcoffee a day ago | parent [-] | | “Hey folks, did you know in 100 years you can’t just call the town doc? Nah, you need to go get a referral. No, for real. Yeah, yeah, that is in fact a compound fracture. I can’t treat it without a referral. Congress made the rules.” Is it so different? |
|
|
|
|
| ▲ | colordrops a day ago | parent | prev | next [-] |
| What's the best you can do hosting an LLM locally for under $X dollars. Let's say $5000. Is there a reference guide online for this? Is there a straight answer or does it depend? I've looked at Nvidia spark and high end professional GPUs but they all seem to have serious drawbacks. |
| |
|
| ▲ | exe34 a day ago | parent | prev | next [-] |
| I think it's possible, but the current trend is that by the time you can run x level at home, they have 10-100x in the frontier models, so if you can run today's Claude.ai at home, then software engineering as a career is already over. |
| |
| ▲ | teaearlgraycold a day ago | parent | next [-] | | You can run quite powerful models at home on a maxed out Mac Studio. The difference between those and SoTA is more like 2x. | |
| ▲ | pstuart a day ago | parent | prev [-] | | My poorly informed hope is that that we can have mixture of experts with highly tuned models on areas of focus. If I'm coding in language Foo, I only care about a model that understands Foo and its ecosystem. I imagine that should be self-hostable now. | | |
| ▲ | tsimionescu a day ago | parent | next [-] | | A model that only understands, say, Java is useless : you need a model that understands English and some kind of reasoning and has some idea of how the human world works, and also knows Java. The vast majority of the computational effort is spent on the first two, the second is almost an afterthought. So, a model that can only program in Java is not going to be meaningfully smaller than a model that can program in ~all programming languages. | |
| ▲ | exe34 a day ago | parent | prev [-] | | my suspicion is that this is not how intelligence works. creativity comes from cross breeding ideas from many domains. | | |
| ▲ | pstuart 14 hours ago | parent [-] | | Sure, but in the context I was considering, creativity itself wasn't a concern. For coding, creativity is not necessarily a good thing. There are well established patterns, algorithms, and applications could reasonably be construed as "good enough" to assist with the coding itself. Adding a human language model over that to understand the user's intents could be considered an overlay on the coding model. I confess that this is willful projection of my hope to be able to self-host agents on affordable hardware. A frontier model on powerful hardware would always be preferable but sometimes "good enough" is just that. | | |
| ▲ | exe34 an hour ago | parent [-] | | I want to self-host too, but I've spent the last few weeks playing with Claude code on my hobby projects - it solves abstract problems with code, and gives actionable reviews, whereas qwen code with qwen3-coder-480 seems to just write simple code and gives generic feedback. |
|
|
|
|
|
| ▲ | cft a day ago | parent | prev | next [-] |
| That's the only future of open source that I can see. |
|
| ▲ | szundi a day ago | parent | prev [-] |
| Those good ones are not even close though - or are they |
