I also expect local LLMs to catch up to the cloud providers.

I spent last weekend experimenting with Ollama and LM studio. I was impressed at how good Qwen3-Coder is. Not as good as Claude, but close - maybe even better in some ways.

As I understand it, the latest Macs are good for local LLMs due to their unified memory. 32GB of RAM in one of the newer M-series seems to be the "sweet spot" for price versus performance.

That's out of touch for 90% of developers worldwide

You need to leave much more room for context if you want to do useful work besides entertainment. Luckily there are _several_ PCIe slots on a motherboard. New Nvidia cards at retail(or above) are not the only choice for building a cluster; I threw a pile of Intel Battlemage cards on it and got away with ~30% of the nvidia cost for same capacity (setup was _not_ easy in early 2025 though).

You can gain a lot of performance by using optimal quantization techniques for your setup(ix, awq etc), different llamacpp builds do different between each other and very different compared to something like vLLM

Honestly though how many people reading this do you think have that setup vs. 85% of us being on a MBx?

> The Qwen3-Coder-480B-A35B model could also be run on a 4090 or 5090 by splitting the active 35B parameters across VRAM and RAM.

Reminds me of running Doom when I had to hack config.sys to forage 640KB of memory.

Less than 0.1% of the people reading this are doing that. Me, I gave $20 to some cloud service and I can do whatever the hell I want from this M1 MBA in a hotel room in Japan.

Yes but they are really less performant than claude code or codex. I really cried with the 20-25GB models ( 30b Qwen, Devstral etc). They really don't hold a candle, I didn't think the gap was this large or maybe Claude code and GPT performs much better than I imagined.

How much context do you get with 2GB of leftover VRAM on Nvidia GPU?

you need a couple RTX 6000 pros to come close to matching cloud capability

But… they do all the time. Almost everybody uses some mix of Office, Slack, Notion, random email providers, random “security” solutions etc. The exception is the opposite. The only thing prevents info leaking is ToS, and there are options for that even with LLMs. Nothing changed regarding that.

This is a poor take imo. Depends on the industry but the worlds businesses run on the shoulders of companies like Microsoft and heavily use OneDrive/Sharepoint. Most entities, even those with sensitive information are legally comfortable with that arrangement. Using a LLM does not change much so long as the MSA is similar.

I agree with you (I have a 32G M2Pro) and I like to mix using local models running with Ollama and LM Studio with using gemini-cli (used to also occasionally use codex but I just cancelled my $20/month OpenAI subscription - I like their products but I don’t like their business model, so I lose out now on that option).

Running smaller models on Apple Silicon is kinder on the environment/energy use and has privacy benefits for corporate use.

Using a hybrid approach makes sense for many use cases. Everyone gets to make their own decisions; for me, I like to factor in externalities like social benefit, environment, and wanting the economy to do as well as it can in our new post-mono polar world.