| Having multiple 6 year old cards doesn't seem like it's that big of burden for local LLMs. I get that a lot of people don't have them. And a single one can be VERY performant. And the smaller models like a 7B can run on much smaller hardware like a mid-range [3|4|5]060. My entire AI Dev Box cost $4500 in parts. 128GB RAM, i7-10700, 1TB and 2TB SSD, and 2x 3090s. Today's prices and inflation have definitely made that price tag seem a lot better than it was, but it was an investment in all things GPU that were happening in 2020 (crypto, blender, image gen), then LLMs exploded. |
| |
| ▲ | thayne 3 hours ago | parent | next [-] | | A single, used 3090 costs more than I have ever spent on a computer. | | |
| ▲ | 9cb14c1ec0 3 hours ago | parent | next [-] | | Yes, the tunnel vision around local models on this site is crazy. The percentage of people in the world who can afford the hardware is extremely low. | | |
| ▲ | layer8 2 hours ago | parent | next [-] | | It seems roughly similar to the pricing level of personal computers in the early eighties (i.e. IBM PC and Apple Macintosh). I’d expect prices to come down significantly over the next few years. Not so much in the next year or two, but after that. | | |
| ▲ | oblio an hour ago | parent [-] | | Just like for warships, the complexity and cost of building cutting edge hardware has grown exponentially up to a point where a significant chunk of the world's computing is dependent on 2 companies: ASML, TSMC. We shouldn't extrapolate linearly from examples from the 80s. | | |
| ▲ | layer8 a minute ago | parent [-] | | No, but I wouldn’t expect it to stagnate like with Intel in the 2010s either. |
|
| |
| ▲ | regularfry 25 minutes ago | parent | prev | next [-] | | There's a non-small contingent who lucked into the periodic games machine upgrade at the right time to snag a {3,4,5}090 rig just before everything exploded. It's a small contingent now but it was less so then. And now those people can add a second card for roughly what that whole system would have cost new originally. | | | |
| ▲ | BoxOfRain 2 hours ago | parent | prev | next [-] | | It's a decreasing pool as well I'd say, the dev machine I built last summer would make less financial sense to me now for example. | |
| ▲ | jermaustin1 2 hours ago | parent | prev | next [-] | | I don't think there is tunnel vision. I'm just saying that I have a couple 3090s I invested in a handful of years ago, and they are still going strong today as multiple GPU-needing technologies emerged. I'm not saying everyone has to run local LLMs, because the APIs are in a race to the bottom, and my $10 of OpenRouter credits I bought months ago is down to $8.94 because most models give you MILLIONS of tokens for a US Quarter. | |
| ▲ | zahlman 2 hours ago | parent | prev [-] | | I mean, I'm not rushing out to buy that kind of hardware myself, but it is a matter of perspective. People commonly spend an order of magnitude more on a car, and that's just the sticker price. |
| |
| ▲ | wafflemaker 3 hours ago | parent | prev [-] | | My single 3080 runs so hot I don't need to warm my room in winter, and have to play games in my underwear in summer. |
| |
| ▲ | zamadatix 2 hours ago | parent | prev | next [-] | | I got a great deal on ~72 TB of NVMe right before storage prices shot up, doesn't make it any less ridiculous that I have it or any more relevant to people talking about building a NAS now. 99% of people, even in tech, do not have the stupid amounts of hardware people like us hobby on. | | |
| ▲ | oceanplexian 2 hours ago | parent | next [-] | | Most people in the US have a car, and the average new car is $40,000. Hell where I live a middle class consumer will spend double that on a Boat or an RV and think nothing of it. These aren’t elite tech workers. It’s not unfathomable that if a personal, generally intelligent local AI provides enough utility and doesn’t require you to tweak CLI flags millions of Americans would want one. | |
| ▲ | sroussey 2 hours ago | parent | prev [-] | | where? i would love that. |
| |
| ▲ | xnx 2 hours ago | parent | prev [-] | | > 2x 3090s You could sell those and have enough money to pay for hosted inference for years. | | |
| ▲ | jermaustin1 2 hours ago | parent | next [-] | | They cost more to run than hosted anyway. But that isn't the point of having them. They are a playground, a backup when the internet is down, or claude is down. They can render Blender scenes pretty well. They play any game I want. You can do each of those at various hosts and own nothing. Or own a couple "over priced" cards and do it all at home on battery power for a few hours while the power is out. | | |
| ▲ | robotresearcher an hour ago | parent [-] | | For me it’s more that you can show them your financial and medical data without BigCo looking over your shoulder. |
| |
| ▲ | Gecko4072 2 hours ago | parent | prev | next [-] | | But after all those years you’d still have 2 3090s, which are now about 6 years old and still holding value. | |
| ▲ | irishcoffee an hour ago | parent | prev [-] | | I keep seeing this comment. This is _hacker news_ where, back in the day, people just hacked on things, because it was a hobby. They weren't "moneymaxxing" or desperately trying to be as insanely efficient as possible. They hacked on stuff with a can of surge at 3am because it was fun. Your comment is like a meta comment of "LLMs are generating everything, after a while the ouroboros will eat itself. (Which I agree with)" If people aren't hacking on this shit just because, you have completely conceded control of software to a handful of sociopaths, and open source software is dead. |
|
|
