I disagree, because you can represent the constituent parts of any AI model as code and data. We can reliably build AI with this knowledge, but not brains.

Understanding does have layers, and that's why "poorly understood" is a meaningless goalpost. A book can be well understood without researching the gematria behind character's the names when you write them in reverse. An LLM can be well-understood even if you don't comprehensively test each quantization for miraculous unexpected behavior at the FFN level.

Your DNA is merely data, and humans can reliably make more of it too.

A book can be poorly understood if you know it's made out of paper and ink, but can't read. The contents would be are meaningless symbols and numbers. You might note that some patterns of symbols repeat, but be completely clueless why or what it represents.

Notably, you could still print them all day.