Remix.run Logo
▲ LudwigNagasena 3 hours ago

Minimizing perplexity may be enough to in some sense match human-level intelligence that, combined with a computer’s ability to do the grunt work, leads to super-human results. But actually going beyond human intelligence would almost by definition imply that perplexity on human texts should increase.

If RL is merely a way to elicit latent capabilities acquired in pretraining, as the article claims, that could mean that currently there is a wall that separates human-level AI from superhuman-level AI. And no one has figured out how to break it yet.

▲mitthrowaway2 an hour ago | parent | next [-]

> But actually going beyond human intelligence would almost by definition imply that perplexity on human texts should increase.

It doesn't imply that.

Imagine you meet a very confused ant one day. They are looking for food, but they seem to keep walking in circles, checking the same places. They've been doing this for hours. You figure out that they're following a chemical trail that goes in a loop. You successfully predict the ant's activity over the next hour. Then you get bored and hungry and go look for food. You probably manage to buy a sandwich without getting stuck walking in circles for hours. Your ability to model an ant's mind has not turned your mind into an ant's.

▲LudwigNagasena an hour ago | parent [-]

I can figure out how an ant thinks. But unless I am trying to do it on purpose, I won't copy an ant. What I’ll do when I'm hungry is a different question.

Transforming your analogy back into the topic at hand, that would be like adding a prompt to the whole human pre-train dataset in the style of "How would a human continue that text?" And that should mean that even super-human intelligence gets the same perplexity. But my point is not that a super-human intelligence wouldn't be able to mimic a human, but that if it is not trying to mimic a human, it should diverge. If you don't pre-prompt it to answer like a human on a question that 90% of humans get wrong, it should answer like 10% without regressing to the mean, which means getting higher perplexity. But if your whole measure is perplexity, you wouldn't be able to train super-intelligence in the first place.

The question is not how to get super intelligent model to pretend to be a human to minmize perplexity, but how to get to super intelligence in the first place and whether minimizing perplexity makes sense in such case.

PS Funny that ChatGPT actually raised a similar concern when I asked it to challenge my comment but I decided not to elaborate on it in my original comment because I am not sure how to succinctly explain my thoughts on it and I am not sure whether I did it well in this comment.

▲ACCount39 2 hours ago | parent | prev | next [-]

Why would perplexity on human texts have to increase?

That would imply that "beyond human intelligence" is closer to a sidegrade than to an upgrade - becoming "superhuman" by necessity poisons your ability to predict human text.

In reality, what would preclude a "superhuman" AI from being a superset, a direct upgrade - both "very good at predicting human text" and "very good at predicting things no human is able to predict"?

▲plasticchris 2 hours ago | parent [-]

because presumably the ai would do a better job than any human, and so it should doubt human outputs.

it doesn't mean it poisons your ability to predict human text, but that it knows if it were to generate output it would likely be more correct than the human inputs...

▲ACCount39 2 hours ago | parent | next [-]

Off the top of my head: lowering perplexity on human-dominant data might mean that we're leaving behind "the model is innately inept and can't predict some things well" and getting closer to "the model is innately capable, but knows human idiosyncrasies and is holding its full capabilities back when predicting human texts."

Thus, if RL-trained for pure performance/correctness, the "innately inept" model would typically underperform the "innately capable" one. The second model has started from a better foundation, so when trained to push itself past the usual limits of human-dominated data, it would attain better task performance.

▲robotresearcher 2 hours ago | parent | prev [-]

If the task is to predict what a human would say, then a perfect oracle would produce what a human would say, even if it was factually incorrect.

▲plasticchris an hour ago | parent [-]

But if the task is to be correct and useful in the maximum likelihood then this is unlikely to be the metric used

▲dist-epoch 2 hours ago | parent | prev [-]

You include the Riemann Hypothesis proof in the dataset. This is how you break the human-level wall.

When AlphaGo moved away from human dataset and became AlphaZero, which trained only on it's own output, performance got another big jump.