Most consumers aren't running LLMs locally. Most people's on-device AI is likely whatever Windows 11 is doing, and Windows 11 AI functionality is going over like a lead balloon. The only open-weight models that can come close to major frontier models require hundreds of gigabytes of high bandwidth RAM/VRAM. Still, your average PC buyer isn't interested in running their own local LLM. The AMD AI Max and Apple M chips are good for that audience. Consumer dedicated GPUs just don't have enough VRAM to load most modern open-weight LLMs.

I remember when LLMs were taking off, and open-weight were nipping at the heels of frontier models, people would say there's no moat. The new moat is high bandwidth RAM as we can see from the recent RAM pricing madness.

> your average PC buyer isn't interested in running their own local LLM.

This does not fit my observation. It's rather that running one's local LLM is currently far too complicated for the average PC user.