| ▲ | kamranjon a day ago |
| "Ternary Bonsai 2 27B reaches up to 143 tokens/second on NVIDIA GeForce RTX 5090 and 46.8 tokens/second on M5 Max. On an RTX 4090, Ternary Bonsai 2 27B consumes just 0.714 mWh/token, making it 40% more energy-efficient than an 8B model running in full-precision." |
|
| ▲ | pizza234 a day ago | parent | next [-] |
| Their mention of the 5090 is bit odd, since on 32 GB GPUs, Q6 fits while having better quality. Very interesting model for 16 GB GPUs though! |
| |
| ▲ | pwython a day ago | parent | next [-] | | Sometimes you want a decent model running in the background that doesn't take up all the VRAM. | | | |
| ▲ | sisve a day ago | parent | prev [-] | | They mention 5090 with regards to speed, Q6 will not have that speed? And speed matters a lot for many use cases | | |
| ▲ | selectodude a day ago | parent | next [-] | | 150 tokens per second on a ternary model implies that it’s GPU bound, I’d bet a Q6 model is even faster because it’s existed longer and seen more optimization. You’d have to be insane to not run an NVFP4 quant over a ternary quant on Blackwell if they both fit. | |
| ▲ | wincy a day ago | parent | prev [-] | | With Ninfer and Qwen 3.8 27b it uses a groupwise int mixed quant, and it gets 160 tokens/sec. The mixed quant is between 4 and 6 bits. | | |
| ▲ | Foobar8568 17 hours ago | parent [-] | | Ninfer is compatible with a nvfp4 model for the 27b. Also nowadays I prefer to use the byteshape one, I get less loops, and I am not sure if I really see a difference in speed or quality. Pure vibe agentic coding on a C++ codebase or ocaml one, ocaml one has codex as reviewer as I am more interested by that project, the other is more for fun. |
|
|
|
|
| ▲ | azatom a day ago | parent | prev [-] |
| it is like "my fridge is 2mkm (millikilometer) from my desk"
m=0.001 h=3600 it should be just Ws or just J |
| |
