|
| ▲ | bozdemir 4 hours ago | parent | next [-] |
| For example you might add Strata & Qwen 3.8 Flash Next with some quantizations. Nowadays it is a go to selection and very sought after MoE model. |
| |
|
| ▲ | cyanydeez 4 hours ago | parent | prev [-] |
| He's exagerating, I hope. The only bottleneck a harness attachs is pre-context fill and whatever tooling is installed. A proper harness reduces these to negligable startup costs. Unless of course he's driving some weird lambo that dumps 200k of context into their chats. Who knows, I feel like we're on a yellow submarine. |
| |
| ▲ | bozdemir 4 hours ago | parent [-] | | Exagerating? I setup llm endpoints on corporate level in a company on H200 gpus, that serves hundreds of software engineers, I have observed differences, even on different versions of nvidia drivers. I shared my day to day experience. It is more and more important if you have very limited VRAM like 24gb on a 4090 RTX desktop or RTX 5090 Laptop gpu. I set up and use llms on my personal devices also, when I can't use cloud models on classified company materials. Here is an example problem what I daily report and deal with. https://github.com/ggml-org/llama.cpp/issues/27330 | | |
| ▲ | jimnotgym 4 hours ago | parent [-] | | I can believe different drivers make a difference, but that wasn't the point under debate | | |
|
|
