If you inspect the Chain of Thought summaries, the LLM often knows full well what it is doing.

That's not knowing. That's just parotting in smaller chunks.