I believe it is much closer than you realize

https://huggingface.co/Qwen/Qwen-Image-2.1-Turbo (7B)

Yes, which I say as a subscriber and donor to 404media (they do very good journalism work imho) and someone who leans more towards AI skeptic while being a heavy user in specific domains. "It's not a business, it's a feature."

Durable storage + sufficiently performant local LLM hardware + appropriate models [last two are wildcards and hard to predict the forward curve imho] = this experience locally. Remember when mobile video hardware acceleration wasn't good? And now, it is very good.

I think we're a breakthrough away. I don't think processing power is an issue. A 7B model has a lot of cruft (like knowledge of code syntax) that a character sim doesn't need. It should be possible to extract just the relevant parts of a larger and more powerful model and run it with the same resources you would run a general 7B model. We just don't have the ability to isolate and extract weights like that yet.

Yes, current gen hardware was not built to run such models, and are obviously inefficient and bad at it.

Significant investment is being spent on producing hardware that is more suitable for running mid-sized (100B-ish) models. Starting with flash chips with wide enough interfaces to provide the weights (NAND flash can be ~5x more energy-efficient for reads than DRAM), and going into more exotic solutions like compute-in-memory.

Yes, iphone, ipad, mac. Maybe your phone makes the inference requests to either Apple servers (requiring iCloud+ and credit based?) or your Mac (Tailscale equivalent).

My assumptions are that inference costs continue to decline (and/or the difficulty level of routing requests declines) and hardware acceleration continues into the future. These assumptions might be wrong, but that is the idea the thesis is based around.

(have built workflows and harnesses that currently target LLM provider APIs under the assumption I'll be able to point "locally-ish" eventually, my career had a strong focus on distributed computing about 10-15 years ago, think managing data and work clusters for LHC detector data processing, and I grew up on hobbyist distributed.net and BOINC projects, all the lenses I see this problem through; queues, jobs, sync vs async performance optimization, etc; recovered sysadmin|infra/devops engineer a lifetime ago)

It's more like saying 'I don't care if monsanto goes under because I will save my own seeds to plant next year's crops'.

You are correct, but I think it's much more like consumer culture doesn't and never will care about any of these details.

Specifically with regard to AI, the consumers aren't even aware of the "grocery store" either. They're being served from a silver platter. What matters most is the quality of results, not the methods. You can disagree all you want for whatever (good and compelling) reasons, but it doesn't change the fact that many people do not even buy groceries now and doordash everything.

If it turns out that you can put a mobile kitchen in their backyard that cooks up everything without them realizing, it makes no difference to the consumer.

Realistically, yes there's a finite amount of training data, hence the plateau in capability we've seen for at least a year. I think we're likely to see a consolidation and compression of all of this into local models. The economics don't support the expected growth of the current arrangement either. There simply isn't that much distinct data out there as initially assumed.