People at the big labs have been terrified of losing the chain of thought because they currently monitor it to prevent users from using their models in certain ways they consider unsafe or bad for their business. Personally I think chain of thought monitoring is a bad way to do this and obviously needs to be replaced by some way of analyzing the model's activations directly.

Agreed, having text as a reasoning layer impedes performance while giving a false sense of security.

It could be encoding anything in its adjective choices etc. The same thing that model providers are using for watermarking can used by the model itself for private thought.

The “obviously” here is puzzling. It is certainly obvious this would be better but it is not at all obvious how to do so.

It's also one of the major way labs can understand these models. Without the models are significantly harder to understand and align.

That’s certainly the long term goal, but it seems foolish to abandon it without a solution to the problem.

Also agreed. Begging for crumbs at the CoS table is impeding cracking the tough nut of MechInterp. And further facing the fascinating and fertile fact that nobody has a clue of how any of these black boxes really work.-

> hey currently monitor it to prevent users from using their models in certain ways they consider unsafe or bad for their business

Can't they just monitor the text outputs as well?