> If so, the thinking trace can be sort of nonsensical for a reader, though whether this is an idiosyncrasy of the model or a property of LLMs in general isn't clear to me yet.

Yes, several models think in weird jargon. Here is an example of Mythos's thinking while playing solitaire: https://www.lesswrong.com/posts/wCSEpT3dTGz4N86Wi/even-illeg...

> 7♣-removal-IS-the-prerequisite-for-10♠/9♥!!)-⟹-OVERLAP-(ii)+(iv):-{6♠ J♦ 9♥ 2♣}-=-FOUR--—-UNLESS-7♣'s-seat-8♥-...-and-2♣-drains-only-at-crack-:-⟹-2♣-celled-+-9♥-celled-simultaneously-UNAVOIDABLE-in-t8-dig--—-BREAK:-9♥

This is a small step in the direction of something called "neuralese", where the model has stopped thinking in English and is thinking in internal vector spaces. Since this gets serialized through text, it isn't quite true neuralese, but it's moving in that direction.

I mean, I'm sympathetic towards the models. My internal thought process when writing code uses lots of intermediate steps that would be hard to write out in English.

> My internal thought process when writing code uses lots of intermediate steps that would be hard to write out in English.

This is something really interesting to me. It turns out there's far more diversity in thinking than you'd imagine given that we're all largely similar meat-in-a-box. I'm on the visio-spatial-tacit wing and speaking my thoughts outloud can be very awkward, whereas one of my former coworkers is on the "all thinking is in words and visual/spatial information comes in the form of words describing the scene" wing, so he can literally narrate his thought process out loud, very interesting conversations can be had discussing the subjective differences.