The issue is that once you pollute your context window with the “wrong” information even after you have guided the LLM to the right path, it is still more likely to go off the rails.

https://research.trychroma.com/context-rot