The “obviously” here is puzzling. It is certainly obvious this would be better but it is not at all obvious how to do so.

Exactly, because the whole thing is entirely based on autocompleting the next word, which is exactly why CoT prose helps getting better predictions.

Why people don't treat categorizing model activations as a science is beyond me.