Unrelated to this slowdown, I could imagine this sort of thing being the right application for things like Taalas, although I'm not quite sure whether it can actually offer what I imagine that it can (i.e. ability to make a cheap, physically small card that can run a large model).

A laptop can't run the latest big models, because memory is too expensive. This thing probably can't run models larger than Mistral Small 4. But it'd be a completely different thing to have a hardcoded Mistral Large 4, than a choice of tiny models.

▲

RealityVoid 3 hours ago | parent [-]

You probably won't be able to run a sota model on a single Talaas card. You probably need a set. It should theoretically be cheaper, faster, more power efficient than Nvidia at running that model. But then you are stuck with a chip that can run only that model. And I understand that context size will also be a problem.

	▲	impossiblefork 3 hours ago \| parent [-]
		Yeah, that was why I felt that, while Taalas might not be the right thing in general, it could be really appealing in a laptop, that would otherwise not be able to run any big model at all. I think I misunderstood what the technology could do a little bit, because I imagined these fixed matrix multiplies encoded into hardware as small-ish, as using routing space etc., not just fast. I think what I really imagined was useful would be more easily achieved by having a big ROM with model weights connected to an ordinary GPU.