> In terms of long context, accurate attention retrieval from early tokens is just impossible, given the expansion of RoPE encoding for the positions, or in case of Kimi that don't use it anymore, as well as deepseek, early context is heavily compressed you lose accurate information.

"Every gambler knows the secret to survivin' is knowing what to throw away, and knowing what to keep." - Kenny Rogers

Humans have limited context, just like AI. The difference is that humans - at least some of the time - can figure out which pieces are more likely to be important, and therefore prioritize keeping those in the context.