|
| ▲ | captainbland 3 days ago | parent | next [-] |
| I think the issue is that LLMs are a cash problem as much as they are a technical problem. Consumer hardware architectures are still pretty unfriendly to running models which are actually competitive to useful models so if you want to even do inference on a model that's going to reliably give you decent results you're basically in enterprise territory. Unless you want to do it really slowly. The issue that I see is that Nvidia etc. are incentivised to perpetuate that so the open source community gets the table scraps of distills, fine-tunes etc. |
| |
| ▲ | butlike 3 days ago | parent | next [-] | | You got me thinking that what's going to happen is some GPU maker is going to offer a subsidized GPU (or RAM stick, or ...whatever) if the GPU can do calculations while your computer is idle, not unlike Folding@home. This way, the company can use the distributed fleet of customer computers to do large computations, while the customer gets a reasonably priced GPU again. | | |
| ▲ | vlovich123 3 days ago | parent [-] | | The kinds of GPUs that are in use in enterprise are 30-40k and require a ~10KW system. The challenge with lower power cards is that 30 1k cards are not as powerful, especially since usually you have a few of the enterprise cards in a single unit that can be joined efficiently via high bandwidth link. But even if someone else is paying the utility bill, what happens when the person you gave the card to just doesn’t run the software? Good luck getting your GPU back. |
| |
| ▲ | cyanydeez 3 days ago | parent | prev [-] | | Consumer hardware is there. grab a mac or AMD395+ and Qwen coder and Cline or Open code and you're getting 80% of the real efficiency. | | |
| ▲ | smilekzs 2 days ago | parent [-] | | New Strix Halo (395+) user here. It is very librating to be able to "just" load the larger open-weight MoEs. At this param count class, bigger is almost always better --- my own vibe check confirms this, but obviously this is not going to be anywhere close to the leading cost-optimized closed-weight models (Flash / Sonnet). The tradeoff with these unified LPDDR machines is compute and memory throughput. You'll have to live with the ~50 token/sec rate, and compact your prefix aggressively. That said, I'd take the effortless local model capability over outright speed any day. Hope the popularity of these machines could prompt future models to offer perfect size fits: 80 GiB quantized on 128 GiB box, 480 GiB quantized on 512 GiB box, etc. |
|
|
|
| ▲ | nunez 3 days ago | parent | prev | next [-] |
| Open-source models will never be _truly_ competitive as long as obtaining quality datasets and training on them remains prohibitively expensive. Plus, most users don't want to host their own models. Most users don't care that OpenAI, Anthropic and Google have a monopoly on LLMs. ChatGPT is a household name, and most of the big businesses are forcing Copilot and/or Claude onto their employees for "real work." This is "everyone will have an email server/web server/Diaspora node/lemmy instance/Mastodon server" all over again. |
| |
| ▲ | heavyset_go 2 days ago | parent [-] | | Local models are more like browsers than servers. The user doesn't care where they're hosted, they click an icon and ask questions either way. | | |
| ▲ | fragmede 2 days ago | parent [-] | | People do care about the privacy of these things though. It's one thing to talk about encryption, but users are pouring out their heart and soul to these things, and they're not all idiots. |
|
|
|
| ▲ | fnordpiglet 3 days ago | parent | prev | next [-] |
| The problem is even if an OSS had the resources (massive data centers the size of NYC packed with top end custom GPU kits) to produce the weights, you need enormous VRAM laden farms of GPUs to do inference on a model like Opus 4.6. Unless the very math of frontier LLMs changes, don’t expect frontier OSS on par to be practical. |
| |
| ▲ | lukeschlather 3 days ago | parent | next [-] | | I feel like you're overstating the resources required by a couple orders of magnitude. You do need a GPU farm to do training, but probably only $100M, maybe $1B of GPUs. And yes, that's a lot of GPUs, but they will fit in a single datacenter, and even in dollar terms, there are many individual buildings in NYC that are cheaper. | | |
| ▲ | fnordpiglet 2 days ago | parent [-] | | I refer you to the data centers under construction roughly the size of Manhattan to do next generation model training. Granted they’re also to house inference, but my statement wasn’t hyperbole, it’s based on actual reality. To accommodate the next generation of frontier training it’s infeasible for any but the most wealthy organizations on earth to participate. OSS weights are toys. (Mind you i like toys) |
| |
| ▲ | palmotea 3 days ago | parent | prev | next [-] | | > you need enormous VRAM laden farms of GPUs to do inference on a model like Opus 4.6. It's probably a trade secret, but what's the actual per-user resource requirement to run the model? | | | |
| ▲ | supern0va 3 days ago | parent | prev [-] | | There's already an ecosystem of essentially undifferentiated infrastructure providers that sell cheap inference of open weights models that have pretty tight margins. If the open weights models are good, there are people looking to sell commodity access to it, much like a cloud provider selling you compute. |
|
|
| ▲ | runarberg 3 days ago | parent | prev | next [-] |
| That would be accepting the framing of your class enemy, there is no reason to do that. |
|
| ▲ | metalliqaz 3 days ago | parent | prev [-] |
| unless they are also pirate LLMs, I don't see how any open source project could have pockets deep enough for the datacenters needed to seriously contend |
