Yeah Im surprised nobody is talking about this. When everyone first saw Taalas I looked at the design and it had a big legup in physical cache availale compared to most chips. Makes you wonder how much of a benefit there is to the actual "baking" of the model vs just having a large chip with a ton of SRAM (or whatever) soldered close to the edge physically.

I feel like what we really need is the ability to solder computer cache on all sides of the chip Meaning above and below as well. If you can only attach it to the edges you will be inherently physically limited on the amount you can put (and maybe even have latency benefits as well)

Taalas does not have cache so...

I agree that Groq with multilayer hybrid bonding could be a good idea.

What you're describing is what Cerberas does.

Talaas is different, it's a true compute-in-memory architecture where the weights are stored in the connections between the transistors that perform the matrix multiply, rather than in seperate memory cells.

Most of the benefit comes from this architecture; hardwiring the weights into the silicon is just the easiest way to implement it. SRAM requires too many transistors, DRAM requires an incompatible manufacturing process, and exotic phase-change memories aren't readily available.