$10k is well outside my budget for frivolous computer purchases.

zozbot234 3 hours ago | parent | next [-]

It would be plenty in-budget if the software part of local AI was a bit more full-featured than it is at present. I want stuff like SSD offload for cold expert weights and/or for saved/cached KV-context, dynamic context sizing, NPU use for prefill, distributed inference over the network, etc. etc. to all be things that just work for most users, without them having to set anything up in an overly error-prone way. The system should not just explode when someone tries to run something slightly larger; it should undergo graceful degradation and let them figure out where the reasonable limits are.

▲

stefs 3 hours ago | parent | prev | next [-]

yeah, but if you really really wanted to and/or your livelyhood depended on it, you probably could afford it.

▲

bdangubic 6 hours ago | parent | prev [-]

99.97% of HN users are nodding… :)

▲

hparadiz 5 hours ago | parent [-]

There are way too many good uses of these models for local that I fully expect a standard workstation 10 years from now to start at 128GB of RAM and have at least a workstation inference device.

▲

bdangubic 4 hours ago | parent [-]

or if you believe a lot of HN crowd we are in AI bubble and in 10 years inference will be dirt cheap when all of this crashes and we have all this hardware in data centers and it won't make any sense to run monster workstations at home (I work 128GB M4 but not run inference, just too many electron apps running at the same time...) :)

	▲	bigiain an hour ago \| parent \| next [-]
		> I work 128GB M4 but not run inference, just too many electron apps running at the same time. This is somewhat depressing - needing a couple of thousand bucks worth of ram just to run your chat app and code/text editor and API doco tool and forum app and notetaking app all at the same time...
	▲	hparadiz 3 hours ago \| parent \| prev [-]
		Inference will be dirt cheap for things like coding but you'll want much more compute for architectural planning, personal assistants with persistent real time "thinking / memory", as well as real time multimedia. I could put 10 M4s to work right now and it won't be enough for what I've been cooking.