| ▲ | jonplackett 2 hours ago |
| I just got qwen 3.8 27b mlx running on my Macbook Pro and honestly I’m pretty blown away by how not-dumb it is. |
|
| ▲ | prettyblocks 2 hours ago | parent | next [-] |
| My problem is how hot they run. I'm on an m4 pro. Do you have the same issue? |
| |
| ▲ | jonplackett an hour ago | parent | next [-] | | It’s hot and also LOUD and runs the battery down quick. But I’m having a lot of luck just running things when I’m away from the computer and can leave it plugged in. It starts going weird (unreliable and slow) with context over 80k so you have to pick tasks one at a time and baby sit a lot more than Claude. But it really is very capable and feels like there’s an intelligence there to talk to. Maybe gpt-4 level clever? I have an m5 max 64gb and I think anything slower would be quite painful. | | |
| ▲ | seanmcdirmid 3 minutes ago | parent [-] | | I have an m3 max 64GB and it runs ok, I was playing around with it last night. Its reasoning is better, but the token/s still makes an MoE model (35B A3B) preferable so I switched back. Try using Goose instead of Claude's harness? Goose doesn't load as much context in at the start, so it might be more usable. I've definitely been able to get it past 128K, although I typically don't go beyond 70-80k. MoE might also be better at this. |
| |
| ▲ | lukan 2 hours ago | parent | prev | next [-] | | I don't have the hardware but a often mentioned advice is to put your mac into energy saving mode - it still will work, a bit slower, but stays cool. | |
| ▲ | an hour ago | parent | prev | next [-] | | [deleted] | |
| ▲ | downrightmike 2 hours ago | parent | prev [-] | | Mineral oil bath? | | |
|
|
| ▲ | alexchantavy 2 hours ago | parent | prev | next [-] |
| How many tok/s are you getting? What gen mbp? |
| |
| ▲ | mattdw 25 minutes ago | parent | next [-] | | My M4 Pro 48GB gets about 13tok/s, in both 3.6 and 3.8 27b Qwens. Qwen A3B and Gemma get closer to 100tok/s from memory but the results are pretty poor for coding tasks. Edited to add: for agentic workflow I’m running omlx which tells me it has about a 90% cache hit rate (tradeoff is some disk and mem space) - that noticeably changes the felt speed. | |
| ▲ | MrScruff 22 minutes ago | parent | prev | next [-] | | I get around 20 tok/s, 4 bit quant, MTP, 4 bit KV cache quantisation. On an M4 Pro 48Gb. | |
| ▲ | dominotw an hour ago | parent | prev [-] | | i suspect ppl dropping generic "its awesome" comments are not actually using it and prbly just managed to get it running for a prompt or two. |
|
|
| ▲ | StarlaAtNight 2 hours ago | parent | prev | next [-] |
| how quick does it respond? what are specs of your laptop? |
| |
| ▲ | Gareth321 an hour ago | parent | next [-] | | I tried it on my M1 MacBook Pro. It's slow but surprisingly smart as a general purpose LLM. Maybe GPT-5.3 level. I gave it a bunch of tools and it can search the internet, make product recommendations, document, code, etc. | | |
| ▲ | alexpotato an hour ago | parent [-] | | Had the same reaction so had Grok create a script to: - find a free GPU droplet on digital ocean - fire it up - pull in a snapshot of the model + extra files/packages etc - set up a ssh tunnel so that the localhost:8000 routes to the above Then I just configured OpenCode to use the above and was off to the races. Works out to be about ~$2/hr all said and done which isn't bad as I only pay when I'm using it (but could get expensive with 24/7 running) | | |
| ▲ | embedding-shape a few seconds ago | parent | next [-] | | [delayed] | |
| ▲ | b112 16 minutes ago | parent | prev [-] | | Awesome, I used Claude to write a small python script to do the same with Linode's API. The only difference is I setup a persistent drive, and with Linode you can boot off of it. So my biggest start up lag is ~ 2 minutes to deploy + boot, then maybe 2 more to warm the model. I actually dislike LLMs. But I'm a realist, and on-demand compute like this is massive cost saving measure. |
|
| |
| ▲ | chorlton2080 2 hours ago | parent | prev [-] | | Does it need to respond fast? For important applications, I'm sure we'd all be fine waiting 20 minutes for a high quality, usable answer. Or is it the need for interative refinements that make speed relevant? | | |
| ▲ | jonplackett an hour ago | parent | next [-] | | It requires patience but it’s more like waiting 5 mins for it to do tasks. You need to be much more involved though and do things slower than Claude where you can trust it to do a lot of tasks at once. It doesn’t have the context for that | |
| ▲ | dominotw 22 minutes ago | parent | prev [-] | | if you are so sure about what the final shape of your output is then its prbly not a common use of ai |
|
|
|
| ▲ | applicative an hour ago | parent | prev [-] |
| Did you read even the title? |
| |
