You'd need hundreds of GB alone for the KV cache of each user. For something like LLama 3 405B you need ~67GB at ~130k tokens. A single CS-3 has 44GB on-chip sram.

So, afaik, Cerebras are optimizing for ultra-low latency batch=1 inference.

https://newsletter.semianalysis.com/p/cerebras-faster-tokens... goes quite in-depth.

LLama 3 405B had the most unoptimized kv cache usage by far. Deepseek v4 pro uses 2.4GB for the same context length[1].

[1]: https://vllm.ai/blog/2026-04-24-deepseek-v4