This is actually discussed in the interview: https://www.dwarkesh.com/i/176425744/llm-cognitive-deficits

It seems to be more nuanced than what people have assumed. The best I can summarize it as is that he was doing rather non-standard things that confused the LLMs which have been trained on vast amounts of very standard code and hence kept defaulting to those assumptions.

Maybe a rough analogy is that he was trying to "code golf" this repo while LLMs kept trying to write "enterprise" code because that is overwhelmingly what they have been trained on.

In my experience, LLMs are really good at retrieving obscure knowledge/algorithms that are locked behind journal paywalls.