Yeah but why shouldn't this be possible? We learned that we can already create artifical intelligence that surpasses human intelligence in some dimensions. There is no natural barrier here. The pace of this improvement would be debatable, but what speaks against the possibility of such accelerating self-improvement?

▲

hgoel 3 hours ago | parent | next [-]

In the real world there aren't any true exponentials, everything eventually saturates as ultimately physics related constraints hit. You can only compress information so much, transfer it so quickly, you can only access resources at a certain speed, only so much energy is available, etc.

AI ultimately has to live in this reality and face the corresponding limitations. These companies have already consumed much of the world's supply of computing power for the next several years, and they're burning vast sums of money to keep the improvements going. RSI won't learn for free, it won't extract massive cost reductions without up front expense, it can't build factories faster than humans can work out related societal matters, it can't magically pave the deserts with solar panels for power or build and run nuclear power plants and more.

Point is, the cost of progress is already approaching the limits of what even the richest countries are able to bear (without war-like mobilization), and to bypass those constraints would require a supposed ASI to construct its own parallel supplychain from scratch without having much ability to directly interfere with reality. Recursive self improvement is ultimately limited by everything else that cannot move at the speed of electricity.

▲

stevenhuang 2 hours ago | parent [-]

> Recursive self improvement is ultimately limited by everything else that cannot move at the speed of electricity.

I'm not sure what your point is. No one thought RSI would break the laws of physics.

▲

hgoel 2 hours ago | parent [-]

I'd recommend reading the full post :)

Specifically: AI ultimately has to live in this reality and face the corresponding limitations. These companies have already consumed much of the world's supply of computing power for the next several years, and they're burning vast sums of money to keep the improvements going. RSI won't learn for free, it won't extract massive cost reductions without up front expense, it can't build factories faster than humans can work out related societal matters, it can't magically pave the deserts with solar panels for power or build and run nuclear power plants and more.

▲

stevenhuang 2 hours ago | parent [-]

No proponents of RSI state they will be operating outside of reality. Said another way, they will operate within the confines of what's possible and still be RSI. I'm quite surprised this is something that needs to be clarified.

You are constructing a straw man of your own making.

	▲	hgoel 2 hours ago \| parent [-]
		Well, right now we have ex Anthropic employees telling the media that their terabyte sized models can possibly copy themselves onto the internet and run elsewhere as if the necessary computing resources are ubiquitous. Plus, "we must pace the frontier" implies that the argument is that the frontier is moving too fast, but if RSI can't move faster than the rest of reality and the models needed for RSI are already nearing the limits of current human reality, RSI can't move much faster than we can improve reality.

▲

zozbot234 3 hours ago | parent | prev | next [-]

> We learned that we can already create artifical intelligence that surpasses human intelligence in some dimensions.

Yes and this was very hard and required massive real-world resources. We didn't just get a sudden flash of insight by thinking real hard about how to make ourselves smarter. Yet that's always the story that underlies any claim of RSI. You can always phrase things generally enough to make any kind of AI-led improvement look like "RSI" no matter how short-term and tightly bounded, but that's just not helpful.

▲

margalabargala 3 hours ago | parent [-]

Would you not agree that, using existing AI tooling, making an LLM of arbitrary below-frontier capability is now easier than it would be without using LLM tooling?

Given that, it seems obvious that the next generation of LLMs will arrive faster than they would have without LLM capability. And the one after that. The floor is being raised, which makes it easier to push on the frontier.

Fable has only been out for three months. Astra is even newer. The capability of these models compared to what existed even a year ago, and the effect they are having on the production of new software, is immense.

That's all you need. RSI can happen with what we have now, just by enabling the continuous shrinking of the loop of people trying new ideas and implementing them. It does not require some magical "go make yourself better" prompt against some model that is past some magical tipping point.

▲

zozbot234 37 minutes ago | parent | next [-]

> Would you not agree that, using existing AI tooling, making an LLM of arbitrary below-frontier capability is now easier

Marginally easier? Yes of course, same as how it's now "easier" to write any kind of code because we aren't using punch cards anymore. That still doesn't get you to any kind of unbounded "takeoff" scenario, because diminishing returns are a thing. The "loop" of people trying out new ideas can only shrink so much.

	▲	margalabargala 8 minutes ago \| parent [-]
		Right. I'm saying the unbounded takeoff scenario isn't realistic, but it doesn't matter. The rate of improvement is continuing to increase, and the gap between present day and autonomous rogue felony generators is not large.

▲

user43928 2 hours ago | parent | prev [-]

Internally Mythos has been available in February.

The labs have been holding their best models back for a while it seems like.

▲

jayd16 an hour ago | parent | prev [-]

Billions of years of evolution hasn't hit on it. Seems pretty unlikely.