| ▲ | ipdashc 10 hours ago | ||||||||||||||||
> 17k tps taalas chip It's odd to me that I haven't heard anything about this approach (baking LLMs/weights into silicon directly) since. It seems almost common-sense that we're going to end up there eventually. And it feels like that point is drawing ever closer now that model capabilities, if not quite plateauing out, are at least getting to a "good enough" point for a LOT of use cases. I wonder if it's being worked on in secret, if there's something about it that makes it infeasible, or if companies are really too nervous to lock in one model like that because the next one down the line could be a huge improvement. Re. infeasability, I have heard that the Taalas demonstration chip ran Llama 3.1 8B (a pretty horrible model) and that even that took a massive amount of transistors / die area. So it might just be the case that the good models are too big to fit on silicon? | |||||||||||||||||
| ▲ | coder543 33 minutes ago | parent | next [-] | ||||||||||||||||
> It's odd to me that I haven't heard anything about this approach since. It has only been four months since they unveiled their first prototype. I don't understand your confusion. Chip development does not happen overnight...? Their initial blog post laid out a roadmap, so theoretically they should have another thing to demonstrate this summer. | |||||||||||||||||
| |||||||||||||||||
| ▲ | mdp2021 an hour ago | parent | prev | next [-] | ||||||||||||||||
> It's odd to me that I haven't heard anything about this approach ... I wonder if it's being worked on in secret, if there's something about it that makes it infeasible The studies and efforts are ongoing and public, and there are technical hurdles to be faced - but the relevant works go back in time quite a lot and there is heightened interest in it now. It seems that you simply took the "hyped headlines" for the whole of the work. | |||||||||||||||||
| ▲ | topspin 9 hours ago | parent | prev | next [-] | ||||||||||||||||
I have also been thinking about this a lot, and share your belief that this is inevitable. Taalas has a running demo here: https://chatjimmy.ai/ It's eye opening: generated an AVX-512 optimized Mersenne Twister in C in 0.076s, 13,706 tok/s. Too fast for the tok/s to be terribly accurate. | |||||||||||||||||
| ▲ | wmf 10 hours ago | parent | prev [-] | ||||||||||||||||
Good models will require multiple Taalas chips but Groq and Cerebras also require a lot of chips and that hasn't stopped them. | |||||||||||||||||