| ▲ | 72deluxe 2 hours ago | |
I have a 16GB 5060 and 32GB system. RAM - are there any recommended Qwen 3.8 Flash models worth running on this ? I am using Nemotron 30B A3B at the moment | ||
| ▲ | user_7832 an hour ago | parent | next [-] | |
Fwiw I think you can run 3.8 flash (next?) on your setup with SSD streaming. There's this video (1) that I found, and my understanding is that flash next can be very efficient wrt ram needed, by using a "dictionary" (or something similar). | ||
| ▲ | MaxikCZ an hour ago | parent | prev [-] | |
32 gb of ram is gonna be your blocker for iq3, u will have to go iq2 and thus the output will suffer. At the plus side, it will spew faster than iq3. Iq3xxs fits barely on my 10+64gb, so much that I am agresivelly cashing checkpoints to disk instead of keeping them in memory, moved vision from ram to disk too etc, I am sitting at few gigs left RAM and few hundred MB free on gpu. | ||