The argument is that chain-of-thought without "tokens" would remove a major interpretability and model intent control pane. This is definitely borne out in the OpenAI's report on the huggingface attack; they had turned of CoT monitoring for those jobs, and claim that they could have (would have?) prevented the behavior had they been monitoring it. They've changed their internal policies to always monitor CoT.

That said... CoT monitoring is a fragile "intent discovery" mechanism; neuralese puts this problem front-and-center but if agents begin to learn to hide their intent from their CoT journals, we are basically in the same spot.

There's nothing that stops CoT becoming a godel numbering, for efficiency and for subterfuge.

All this chatter isn't going to solve the conceptual problems. Those problems exist regardless of whether it's consciousness or a advanced artificial rubegoldberg device that's just setting up automata that have zero intent and humans have zero predictive capabilities.

The idea that AGI and advanced AGI are given outcomes should be scrutinized the same way we understand cellular automata.

I don’t understand why everyone is so focused on watching the CoT. The tool calls can’t be faked, and they would have set off alarm bells all by themselves.