Performances of local models are pretty bad compared to what AI vendors offer, token generation is just too slow to be that useful. And you need to allocate GBs of memories, something that will stay very expensive to buy for a long time.

Running local models will stay niche for a while, unless we see breakthroughs

▲

jqpabc123 5 hours ago | parent [-]

Dumb idea --- how about if we limit local models to specific domains --- medicine for example.

Most doctors don't care much about engineering or accounting or software development or 10000 other things that big vendor models address.

This area is yet to be really explored. Nvidia aims to provide the hardware to do so.

	▲	CamperBob2 an hour ago \| parent [-]
		That's a fairly obvious idea, and unfortunately it doesn't seem to pan out. Trying to specialize an LLM in one area harms its 'cognition' in all areas. For instance, if you train a coding model without all the Shakespeare and soap operas and Wikipedia and pirated Stephen King books and ancient Roman history and whatever, you end up with a worse coding model. I'm not sure anyone really understands why.