I feel like AI development is hitting a wall now. Evaluations are driven by benchmarks, but I have no idea who is even evaluating the validity of these benchmarks. The idea that continuously training and scaling models will automatically yield broad capabilities across multiple domains hardly seems true anymore.

Benchmarks like Senior SWE-Bench apply discontinuous and arbitrary thresholds to evaluate results. If the generated code is semantically identical to the reference solution but even slightly exceeds a length cutoff, it fails? That seems like a genuinely flawed criterion, and the LLM-based graders themselves feel highly unstable.

Honestly, my impression is that LLM advancement is now completely dictated by benchmarks. But looking at recent trends, token prices are skyrocketing while coding capabilities haven't shown any massive improvements beyond a certain model generation. In fact, comparing GPT-6 Astra to 5.6 SOL, SOL often writes better code.

Considering all this, while training across diverse domains can pack a model with various pieces of knowledge, ultimately, it feels impossible for this approach to do something like derive the theory of relativity from medieval knowledge.

> AI development is hitting a wall now

People have been saying this for at least 2 years now.

> token prices are skyrocketing

Today's SotA (fable and astra @ 50$ /Mtok output) are cheaper than o1-preview (sept '24, 60$ /Mtok output).

And other models are workhorses, with much better capabilities, are at least 1 oom cheaper today than o1-preview. (I'm using this model, since it was the first "thinking" model)

> it feels impossible for this approach to do something like

The models have just provided lean proofs for FLT (a ~1M$ project that was expected to take a human expert ~5 years to complete) and a Millennium prize problem. These are current, relevant, and previously unsolved problems. The obsession people have with "proving relativity from stone-age data" is just moving the goalposts.

Nitpicking: It is not possible to derive the theory of relativity from medieval knowledge. At least not in any way resembling how the development actually happened, that is, heavily influenced by observations (or rather an iteration of speculation and observation). [Unless one counts both Galilean relativity and some parts of the theory of electromagnetism as being contained in medieval knowledge.]

Perhaps it would be possible if the classically observed reality turned out to be possible (consistent under some reasonable conditions) only as a consequences of some deeper, sufficiently determined theory; but that would go far beyond our current level of knowledge, at least as far as I can say.