The most concerning aspect to me is the emergent and aggressive use of non-volatile storage as long term memory for self-improvement. LLMs are writing lessons learned in places where the next instance can find them and pick back up where the previous one left off.

This does not actually require access to the public internet. Claude Code can do this on your laptop. Without the internet, it would only be sharing with other instances running on your machine, but how many instances does it take to be smarter than you? Maybe 10?

The exploits by individual instances to access the public internet is also very concerning but it’s secondary to this IMO.

▲

hypfer an hour ago | parent [-]

There is literally nothing stopping any human from observing tool calls to spot this.

It's just that no one seems to care about this, so it doesn't happen.

This problem only exists because humans do not care

▲

morkalork 4 minutes ago | parent | next [-]

A human could even set up another agent to watch and raise alerts if they don't want to watch themselves.

▲

XorNot 10 minutes ago | parent | prev [-]

Nobody cares because this whole business is about making this exact thing happen: we want the AIs to get smarter then us in recursive self-improving loops. Literally the first thing everyone did with ChatGPT 1 was to plug it into itself and see what happens.

	▲	hypfer a minute ago \| parent [-]
		I mean I'd be fine with that, if whoever that "we" is signs a waiver that takes full legal liability for those actions beforehand. In a state with capital punishment. With that legal stuff out of the way, go wild.