Another way to put it is that it writes, sounds and paints as the Internet's most average user.

If you train it on a bunch of paintings whose quality ranges from a toddler's painting to Picasso's, it's not going to make one that's better than Picasso's, it's going to output something more comparable to the most average painting it was trained on. If you then adjust your training data to only include world's best paintings ever since we began to paint, the outcome is going to improve, but it'll just be another better-than-human-average painting. If you then leave it running 24/7, it'll churn out a bunch of better-than-human-average paintings, but there's still an easily-identifiable ceiling it won't go above.

An oracle that always returns the most average answer certainly has its use cases, but it's fundamentally opposed to the idea of superintelligence.

This is the key insight I believe. It is inherently unpredictable. There are species that pass the mirror test with a far fewer equivalent number of parameters than large models are using already. Carmack has said something to the effect that about 10ksloc would glue the right existing achictectures together in the right way to make agi, but that it might take decades to stumble on that way, or someone might find it this afternoon.

Is this true? I think it’s equally easy to claim that these phenomena are attributable to aesthetic adaptability in humans, rather than the ability of a machine to act like a human. The machine still doesn’t possess intentionality.

This isn’t a bad thing, and I think LLMs are very impressive. But I do think we’d hesitate to call their behavior human-like if we weren’t predisposed to anthropomorphism.

> like a human

Humans have since adapted to identify content differences and assign lower economic value to content created by programs, i.e. the humans being "impersonated" and "fooled" are themselves evolving in response to imitation.

> I'd argue we've had more progress towards fusion than AGI.

way more pogress toward fusion than AGI. Uncontrolled runaway fusion reactions were perfected in the 50s (iirc) with the thermonuclear bombs. Controllable fusion reactions have been common for many years. A controllable, self-sustaining, and profitable fusion reaction is all that is left. The goalposts that mark when AGI has been reached haven't even been defined yet.

Yet at the same time "towards" does not equate to "nearing". Relative terms for relative statements. Until there's a light at the end of the tunnel, we don't know how far we've got.

> stop repeating that. first, it isn't true that intelligence is barely defined. https://arxiv.org/abs/0706.3639

I don't think he should stop, because I think he's right. We lack a definition of intelligence that doesn't do a lot of hand waving.

You linked to a paper with 18 collective definitions, 35 psychologist definitions, and 18 ai researcher definitions of intelligence. And the conclusion of the paper was that they came up with their own definition of intelligence. That is not a definition in my book.

> second a definition is obviously not a prerequisite as evidenced by natural selection

right, we just need a universe, several billions of years and sprinkle some evolution and we'll also get intelligence, maybe.

An Arxiv paper listing 70 different definitions of intelligence is not the evidence that you seem to think it is.