I agree. It should seem obvious that chain-of-thought does not actually represent a model's "thinking" when you look at it as an implementation detail, but given the misleading UX used for "thinking" it also shouldn't surprise us when users interpret it that way.

These aren’t just some users, they’re safety researchers. I wish I had the chance to get this job, it sounds super cozy.