| ▲ | danielhanchen 2 days ago | |||||||||||||||||||||||||
Thank you! I presume 24B is somewhat faster since it's only 4B activated - 31B is quite a large dense model so more accurate! | ||||||||||||||||||||||||||
| ▲ | ryandrake 2 days ago | parent [-] | |||||||||||||||||||||||||
This is one of the more confusing aspects of experimenting with local models as a noob. Given my GPU, which model should I use, which quantization of that model should I pick (unsloth tends to offer over a dozen!) and what context size should I use? Overestimate any of these, and the model just won't load and you have to trial-and-error your way to finding a good combination. The red/yellow/green indicators on huggingface.co are kind of nice, but you only know for sure when you try to load the model and allocate context. | ||||||||||||||||||||||||||
| ||||||||||||||||||||||||||