> More likely (without careful vetting by the folks aggregating these models) is that the quality will go down as more and more AI-generated output gets subsumed into these models.

This assumes that there aren't algorithmic breakthroughs which reduce training/inference costs by several OOMs.

How much do these models need to do before people throw their hands in the air and say, ok this is happening. The Erdos unit distance problem, which as far as I understand was approached by multiple competent mathematicians was solved by a frontier model. Sure people argue there was no novelty there (I cannot comment as a non-mathematician) but it feels like they can draw lines laterally from deep knowledge in different fields (in this case combinatorics and algebraic number theory I believe) and solve problems.

Now if you have millions of instances running in parallel, all "probabilistic", working on frontier AI research I really don't see the blocker (and believe me I wish I did).

> How much do these models need to do before people throw their hands in the air and say, ok this is happening

What is "this"? Most people arguing against some of the more fervent predictions and promises of "inevitability" are people who are using these models in day to day - they see what the models can do, and what they struggle at.

> Now if you have millions of instances running in parallel, all "probabilistic", working on frontier AI research I really don't see the blocker (and believe me I wish I did).

My genuine prediction is that you'll get a lot of early results simply because you're applying attention to some low hanging fruit of problems, but then it will drop off due to the cost of tokens and the low rate of return. This doesn't mean that the models are especially capable of novel thought, just that we haven't algorithmically brute forced a problem with known solutions.

We would be seeing more success cases if the promises were true, setting aside AGI, human replacement, etc. We would see more, better products with more features that people would use. We wouldn't be having any arguments. The human replacement presupposes the models work in ways that they don't, and until proven otherwise, can't. I've watched those who embrace it fully flounder around on projects, some have lost their mind from the constant LLM validation, and I've seen companies go all in and then pull back based on both cost and efficacy over the last year.

I'm still waiting for the success case examples applied on a scale that would make any of the predictions come true.

> This assumes that there aren't algorithmic breakthroughs which reduce training/inference costs by several OOMs.

Yes, if one must assume something it is generally fair to assume that things will continue as they are. Research breakthroughs do happen, but they are not something for which you can predict the timing.