My concern with most of these visual benchmarks, popular as they are, is that they are likely more indicative of knowledge (i.e. how comprehensive the training data is and how well it can be retrieved from the model) than of reasoning ability. I don't see in particular how a model would construct a CoT that mapped somehow to a representation of the cube geometry and its animations in latent space without a large chunk of that being pre-existing information.

▲

nomel 2 hours ago | parent | next [-]

> without a large chunk of that being pre-existing information.

Is there any evidence that novel reasoning is present in LLM? I've never been able to make that work, and I believe Apple's paper some time ago was good evidence that it doesn't exist. In my experience, sparse latent spaces result in a complete, comical, failure in reasoning.

▲

drivebyhooting 2 hours ago | parent [-]

See the new mathematical proof published by OpenAI.

I’m not very valiant to verify its veracity. But even if the math is merely derivative it merits mention.

	▲	nomel 2 hours ago \| parent [-]
		True, but that's an unknown internal model, without details of the architecture. We'll have to see if the LLM model, itself, was responsible for the "novel" bits, or if it was stuffs bolted onto the LLM that made it possible. I suppose "LLM" is maybe no longer sufficient to describe the systems that LLM are being integrated into, so maybe my point is pedantic/semantic.

▲

Chu4eeno an hour ago | parent | prev [-]

Yeah, Anthropic likely is gaining an edge in tests like this from the data they got from Canva.