That makes sense. The title initially sounded like a general llama.cpp speedup on Apple Silicon, but if the improvement comes from fixing kernel selection inside Virtualization.framework VMs, that distinction is pretty important.

It looks to me like this won't speed up llama.cpp for everyone, just for users running it in this particular kind of Virtualization.framework VM.

The fix here works around a problem where the VM was causing llama.cpp to select the wrong kernels.

I recall there was another YC startup that was working on Mac-specific ML optimizations for local inference (and perhaps fine-tuning).

I wonder if their work is related?

> 11.08× faster and generated tokens 16.36× faster than the same workload in the same stock VM.

So this was the comparison, for me the title was a bit confusing

I don’t understand what Apple 1-9 are. At first I thought it was M series chips but there is no M9 (yet)

Still no easy way to get up and running performantly with Llama.cpp

Nobody tells you how, they just act like you're an idiot. So I always say fuck this and install Ollama.

Then everyone goes BLASPHEMY "just use llama.cpp"

Yeah I did, and it's slow as hell. It doesn't work well. Idk why.

"You aren't doing it right"

Okay tell me how

"NO"

-----

For this reason, Ollama is the superior solution. I know, downvote, everyone hates Ollama here but until llama.cpp gets their shit together on developer experience it doesn't exist as far as I'm concerned