| ▲ | aliljet 7 hours ago |
| What I really want to know for models like this is if I can have them run on really lame consumer hardware (consider 128gb of ram, 2080ti) with extremely low TPS, but with reasonable enough intelligence that you can walk away and expect something to bake really nicely while you sleep. Perhaps I'm losing touch with the edge, but it doesn't seem like this is quite possible with the hardware baseline I'm talking about... |
|
| ▲ | MaxikCZ 7 hours ago | parent | next [-] |
| "Model like this" not sute about ds v4.1 but qwen3.8 flash next is currently running on 10gb 3080 + 64 gb ddr5 ram at respectable 40-70t/s with 1200pp at max context |
| |
| ▲ | Iolaum 6 hours ago | parent | next [-] | | Indeed, I m surprised how little attention qwen-3.8.-flash-next seems to be getting while it's so well fitted for 128gb unified machines. | |
| ▲ | 72deluxe 2 hours ago | parent | prev [-] | | I have a 16GB 5060 and 32GB system. RAM - are there any recommended Qwen 3.8 Flash models worth running on this ? I am using Nemotron 30B A3B at the moment | | |
| ▲ | user_7832 an hour ago | parent | next [-] | | Fwiw I think you can run 3.8 flash (next?) on your setup with SSD streaming. There's this video (1) that I found, and my understanding is that flash next can be very efficient wrt ram needed, by using a "dictionary" (or something similar). 1 - https://youtu.be/IH8XmxiwliQ?si=lJPvYzi3mHNsiw8r | |
| ▲ | MaxikCZ an hour ago | parent | prev [-] | | 32 gb of ram is gonna be your blocker for iq3, u will have to go iq2 and thus the output will suffer. At the plus side, it will spew faster than iq3. Iq3xxs fits barely on my 10+64gb, so much that I am agresivelly cashing checkpoints to disk instead of keeping them in memory, moved vision from ram to disk too etc, I am sitting at few gigs left RAM and few hundred MB free on gpu. |
|
|
|
| ▲ | davedx 7 hours ago | parent | prev | next [-] |
| I use DwarfStar DeekSeek V4 Flash on my Mac Studio with 128gb of RAM and it's pretty good - decent TPS and coding ability. Apparently the V4.1 model can also be run by streaming weights from the SSD - I've not tried this yet because it results in a lower TPS, but eventually this version of 4.1 should also be usable like this. |
|
| ▲ | XMPPwocky 6 hours ago | parent | prev | next [-] |
| expert streaming with cache-conditional routing is very powerful (you can run 4.1 Flash on two Strix Halos and a 9070XT at around 50-70TPS decode in ~full quality), and that setup might be able to at least do a few tok/s with it. |
|
| ▲ | akmarinov 7 hours ago | parent | prev | next [-] |
| You sure can, https://github.com/lxe/dsx gets you this deepseek on 2x3090s and 128 gb ram at about 30 t/s At around 75% of the full model intelligence |
| |
| ▲ | veber-alex 7 hours ago | parent [-] | | About half the t/s of an online provider, heat, noise, power consumption and on top of that you only get 75% of the intelligence? Sounds like a waste of time to me. | | |
| ▲ | akmarinov 6 hours ago | parent | next [-] | | Half the token/s - sure, no way around beating big ass data centers. Heat, noise and power consumption are actually quite ok - i'm running 2x3090s, capped at 250W and i haven't seen heat and noise being an issue, sitting right next to it. Might be helping that it's getting colder. 75% of the intelligence with 2x3090s, with a 5090 if you have it it's around 90% and these are first day builds too, typically people hone in and improve these all the time. It's true local doesn't make sense value wise against online, but if you have the hardware, having "unlimited" tokens with no 5h limit, no weekly limit, is super nice. | | |
| ▲ | origini 2 hours ago | parent [-] | | I have not capped my 2x4090s so my room gets cosy warm. But here my price per kwh is 40 cents (euro). But if you integrated it with solar system and a heat exchange, this will be feasable |
| |
| ▲ | nirvdrum 6 hours ago | parent | prev | next [-] | | As luck would have it, you can just pay the provider and use their service. Others may want to run locally for privacy reasons or just for the fun of it. I think at-home AI is incredibly awesome and I applaud the enthusiasts pushing it forward. Just because that's where it's at today doesn't mean it's the best we can do. Plus, you know, this thread was about running AI on "lame consumer hardware". There are more power efficient options available. | | |
| ▲ | egeozcan 5 hours ago | parent [-] | | I use only providers because I don't have the hardware, but if, and that's a big if, the only moat left for humans is our ability to steer and make bigger picture decisions with our personality, I think it makes sense not to want to send all that conversation data to a 3rd party. |
| |
| ▲ | celrod 6 hours ago | parent | prev [-] | | [dead] |
|
|
|
| ▲ | lhl 7 hours ago | parent | prev | next [-] |
| You can get something relatively capable and usable with your configuration and Strata: https://github.com/Niko1221/Strata |
| |
| ▲ | Balinares 6 hours ago | parent [-] | | Strata is intriguing and it definitely looks like there's something useful happening there with weight streaming at usable speeds, but I've not managed to have it serve a full answer without inference crashing before long. Pity. | | |
| ▲ | MaxikCZ 3 hours ago | parent [-] | | I dont think its strata problem, its been running on my system for days without any crash. |
|
|
|
| ▲ | system2 6 hours ago | parent | prev | next [-] |
| Nvidia stopped manufacturing the 5090, so you and people like us cannot run these models on our computers and are forced to use American SOTA so they don't lose the game... |
|
| ▲ | szundi 7 hours ago | parent | prev [-] |
| [dead] |
