|
| ▲ | kristianp 2 days ago | parent | next [-] |
| Which card are you using? I was getting about 40 with an UD q3 quant with MTP (prediction) enabled and llama.cpp compiled for my compute capability, but was very limited in the context size. I have an 4060 ti 16GB. Wouldn't recommend it as there's a tradeoff between larger context without MTP and about 18 tokens/s. |
| |
| ▲ | Pragmata 2 days ago | parent [-] | | I got a RTX5080 as primary and a RTX3050 (8gb) as secondary. I always max out on context size so that means i need to split the model between both gpus + offload some into system ram when using LM studio. I'm guessing that is what accounts for most of the slowdown when i use that. With this recipe i don't have to worry about it. It's pretty fast, and since its all running in the 5080 with max context it's stable (though i got to look out not to load anything big on the same gpu) |
|
|
| ▲ | falsaberN1 2 days ago | parent | prev | next [-] |
| With llama.cpp (CUDA) and a 5060ti (16GB) I get 60t/s with 128K token space. Odd you got 7t/s, did you verify all the model was loaded in VRAM? (--gpu-layers all) |
| |
| ▲ | Pragmata 2 days ago | parent [-] | | it was but i max out on context so it doesn't all fit with kv cache etc... |
|
|
| ▲ | blagui 2 days ago | parent | prev [-] |
| 7 t/s not sure how you can use this?
And if it start thinking a bit. It works yes but that's really very slow.
On top it's very low quant that impact the model quality. |
| |
| ▲ | supermatt 2 days ago | parent [-] | | > 7 t/s not sure how you can use this 80 t/s - maybe you need to finish reading a sentence before you comment on it? |
|
