This is a recurring sentiment but flawed, I think.

First of all, neural nets do nit return averages per se. They construct space between the points and extrapolate outside of the points. So even if a point was not in their training data, they will be ok, in many situations, to acknowledge it.

Or in other words - LLMs don’t average. They construct world models. A novel thing that fits their world model will be accepted no prob. A thing that doesn’t may still be accepted but with challenges.

The same is true though for humans, including scientists. There is a saying that science moves one grave at a time - because often prev gen of scientists needs to die off for a new idea to take root.

Or in yet other words - even if llms produced averages, an average of a discontinuous set can lie outside of that set. And the set of all human ideas is very much discontinuous.

▲

krona 8 hours ago | parent [-]

> They construct space between the points and extrapolate outside of the points.

They don't. They interpolate between the points on a manifold.

▲

kolinko 7 hours ago | parent | next [-]

Manifold vs space - yes, I meant manifold, english is not my first language and I missed this word.

The argument stands.

	▲	krona 5 hours ago \| parent [-]
		Interpolation vs extrapolation? It's very simple distinction and fundamental. Look at ARC-AGI-3 for empirical evidence, if that's important to you. My point however is purely an a priori one.

▲

memoriyato3 8 hours ago | parent | prev [-]

so the proof to the unit-distance problem was on the manifold, given it was outputed by a LLM?

is the proof to the Riehmann hypothesis also somewhere on the manifold and we just need to prod the LLM with the right prompt so it locates the point?

▲

krona 7 hours ago | parent [-]

> so the proof to the unit-distance problem was on the manifold, given it was outputed by a LLM?

By design this must be the case, even when you account for stochastic sampling i.e. 'temperature'. All it's outputs are a highly-dimensional combinatorial interpolation (I'm talking about GPTs here)

That's probably why Claude is very good at producing plausible nonsense rather than the often correct response of 'I don't know'.

	▲	kolinko 7 hours ago \| parent [-]
		What you’re saying here makes no sense.