| ▲ | Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update(unsloth.ai) |
| 89 points by Retro_Dev 7 hours ago | 14 comments |
| |
|
| ▲ | dajonker 24 minutes ago | parent | next [-] |
| This is a great area of research with many players. See also ByteShape https://byteshape.com/blogs/Qwen3.8-27B/ (I'm not affiliated) |
|
| ▲ | aliljet 2 hours ago | parent | prev | next [-] |
| There are so many of us with weak GPUs and these updates are massive wins. Still waiting for the 2080 moment where this 27B param model and a reasonable kv cache can exist and function at slow, but overnight-burning speeds .. |
| |
| ▲ | Incipient 28 minutes ago | parent | next [-] | | This would depend hugely on the price of electricity, however I'm pretty sure where I am (at roughly 25c/kwh in usd) the price of running qwen costs more than 6.1-luna per token. So other than privacy (which doesn't matter for my work) there isn't any justification for running locally. Is privacy purely what people justify local models on? | | | |
| ▲ | 3eb7988a1663 an hour ago | parent | prev [-] | | As an experiment I took, not this current quant, of Qwen 3.8 27B and had it analyze some of my writing in CPU-only mode with llamacpp. Tried a Q4 and Q5. It was dreadfully slow, but you fire that off before bed, and you can get results the next day. This was a ~6000 word markdown file, took maybe 10 minutes before it started generating anything at all, but then it just cranks away. I would need to tune down the default thinking intensity, but it otherwise did the job. Not compatible with vibe coding, but if you want a local tool to review content, it is possible today. | | |
| ▲ | jedbrooke 41 minutes ago | parent [-] | | I’ve been using Bonsai 2 (also based off of Qwen3.8 27B) on my mac mini in this way (using pizza-bot for the harness). In my case I’ve been using more for deep web search type stuff. It’s been a nice workflow to kick off a request in the evening and come back to a 1 page report on it the next morning. |
|
|
|
| ▲ | nout 3 hours ago | parent | prev | next [-] |
| How does it compare to the 3.8-27B versions from TensorFold? I.e. does it make sense to run unsloth via TensorFold engine? |
|
| ▲ | hacker_88 2 hours ago | parent | prev | next [-] |
| Figuring out local llms , but it did built me single page html . On 12 GB VRAM and 16 GB ram windows . |
| |
| ▲ | aliljet an hour ago | parent [-] | | Give us more details. What kind of hardware? What kind of setup? |
|
|
| ▲ | quaddoggy an hour ago | parent | prev | next [-] |
| Just installed Qwen 3.8 Splash tonight (doesn’t appear on linked page). Super impressed. Easily averaging 45 tps, even with thinking on. Totally acceptable. Running M4 Mac mini, 48GB, LM Studio. |
|
| ▲ | ilaksh 4 hours ago | parent | prev | next [-] |
| How do the smallest quants of this compare to Bonsai 2? |
| |
| ▲ | jedbrooke an hour ago | parent | next [-] | | Bonsai 2 does some additional tuning in addition to just quantizing (mostly around the fp16 block weights per 32 params), so they’re able to pack more in to just so few GB. The tradeoff with unsloth seems to be that it’s more directly applicable for other off the shelf open models, whereas with Bonsai you’d have to wait for them to release another one | |
| ▲ | DiabloD3 4 hours ago | parent | prev [-] | | Bonsai 2 doesn't work at all, Unsloth's smallest... also doesn't work, but is still better. Bonsai's technology would be better suited for 100s of B models. |
|
|
| ▲ | jan_Sate 4 hours ago | parent | prev [-] |
| Awesome! I wonder how it does at Q5+ compared with v3.0 |
