| ▲ | jadbox an hour ago | |
Nemotron 3.5 Lightning runs on how little GPU vram? Can q4 run on 16gb? | ||
| ▲ | sleepyeldrazi 36 minutes ago | parent [-] | |
Not by the looks of it, but it got me thinking, currently in the middle of Level1Techs coverage on the model and switchyard and he mentions "how easy it is to customize it". Fully admitting that I haven't yet read the docs, my issue with that is "we can train LORAs for 35B as well, why use this (according to benchmarks) worse model for customization instead of a slightly bigger better one?" Assuming I eat my words after going through the docs and this is actually a more efficient model / loras adapt better, I don't see as much value in it as is, as a REAP of it (remove least-important experts, domain-locked tests show ~98% retained accuracy) to something like 20B-A3B (rouhgly matching gpt oss, which while a good model, is outdated knowledge-wise and not as good with tool in my xp). Having a 20B-A3B model at q4 that has a lora to be your local orchestrator (delegating coding to server/cloud models) and ci/cd runner does start sounding like an appealing proposition to me, as that would fit in 16gb vram easily (fitting many consumer gpus and 24gb macs). | ||