Remix.run Logo
▲ codewithcheese 8 hours ago

RL is not scalable to super intelligence. Super intelligence should be able to learn continuosly without verifiable rewards.

▲StevenWaterman 8 hours ago | parent [-]

You don't need RL all the way to ASI. You just need RL far enough that the model becomes good enough at AI research that it discovers something better than RL.

▲codewithcheese 7 hours ago | parent [-]

> becomes good enough at AI research that it discovers something better than RL

That might require continuous learning without verifiable rewards. RL models might help humans join the dots faster, but we don't know how many years that will take.

If its not possible with neural nets as we know them now, and requires a number of completely novel breakthroughs, there could be far from exponential progress even with the best RL models.

▲StevenWaterman 6 hours ago | parent [-]

Yep, I agree. Improving AI might get harder, faster than the AI gets better.

But it also might not, which is the scary part.