Yeah, that was pretty much my experience. The models weren’t lacking knowledge as much as discipline. Without a good workflow, they will most likely spend thousands of tokens exploring dead ends.

Sounds like an energetic developer that just graduated from college…

(Without the learning and growth potential)

I think 'exploring the dead ends' is *possibly* very fruitful. Various disciplines have a lot of appearing dead ends that someone in another field did solve. And they don't talk with each other.

What I've seen with lots of the breathless 'OMG SCIENCE ADVANCEMENT' articles including the one yesterday, is that the LLMs are quite extraordinary about linking a dozen different fields together, and delivering an answer combined from all of them. Basically, the solutions to a lot of current scientific problems are partially solved, but partial from a lot of fields that don't talk with each other. And also, they don't use the same nomenclature, so simple searches don't suffice.

An LLM seems to use Chomsky's Universal Grammar, and thus on training, does normalize all training data, including different words for the same thing.

A lot (perhaps all?) Of these BIG scientific advancements were solved by humans, partially. An LLM jigsawed them together and appeared it solved the thing.