I've been using it for the past few days, and it runs really well!

I usually get 7 token/s using llama or lm studio, but this inference recipe runs at a smooth 80 tokens per second.

Genuinely very usable, and fully local!

kristianp 2 days ago | parent | next [-]

Which card are you using? I was getting about 40 with an UD q3 quant with MTP (prediction) enabled and llama.cpp compiled for my compute capability, but was very limited in the context size. I have an 4060 ti 16GB. Wouldn't recommend it as there's a tradeoff between larger context without MTP and about 18 tokens/s.

	▲	Pragmata 2 days ago \| parent [-]
		I got a RTX5080 as primary and a RTX3050 (8gb) as secondary. I always max out on context size so that means i need to split the model between both gpus + offload some into system ram when using LM studio. I'm guessing that is what accounts for most of the slowdown when i use that. With this recipe i don't have to worry about it. It's pretty fast, and since its all running in the 5080 with max context it's stable (though i got to look out not to load anything big on the same gpu)

▲

falsaberN1 2 days ago | parent | prev | next [-]

With llama.cpp (CUDA) and a 5060ti (16GB) I get 60t/s with 128K token space. Odd you got 7t/s, did you verify all the model was loaded in VRAM? (--gpu-layers all)

	▲	Pragmata 2 days ago \| parent [-]
		it was but i max out on context so it doesn't all fit with kv cache etc...

▲

blagui 2 days ago | parent | prev [-]

7 t/s not sure how you can use this? And if it start thinking a bit. It works yes but that's really very slow. On top it's very low quant that impact the model quality.

	▲	supermatt 2 days ago \| parent [-]
		> 7 t/s not sure how you can use this 80 t/s - maybe you need to finish reading a sentence before you comment on it?