As someone who does a lot of work with local LLM's, today's systems feel woefully under-powered. I'm looking forward to a future where my laptop has 10x the memory, 100x the memory bandwidth, and optimized cores to make inference workflows that currently take minutes or hours go down to seconds or milliseconds.

While we're at the point where traditional software is pretty much fast enough for all but extreme use-cases, with LLM's it feels like we're back to the days where you press compile and go have a coffee or chat to your colleague.

▲

simonask 3 hours ago | parent [-]

Right now there isn’t really a convincing use case for local LLMs outside of enthusiast or specific niches.

It takes a lot of expensive hardware, and most people don’t have extreme enough requirements to recoup that investment.

That might change, but it would require either that hardware gets much cheaper or everyone’s demands for LLMs increase significantly.

	▲	skohan 9 minutes ago \| parent [-]
		That's what I mean though. The comment I was responding to was talking about "forever laptops" - my point is there's plenty of room for new capabilities which will make current hardware obsolete. Just like how GPU's didn't exist at all, and became a standard part of computing. And given how fast the hardware and software is evolving, I can easily imagine a future where we all have very capable models running on our own devices for an embedded intelligence layer that's doing most of the day-to-day tasks, and only have to outsource to a super-smart cloud model for specific things.