> instead just call defeat and say “I’m not sure how to proceed next”.

Because that is fundamentally impossible given how they work...

The thing does not even know when it succeeds or fails. Actually the thing does not "know" at all...

All it can does is to show some limited textual behavior that matches with "knowing"..

You can get near this point with scaffolding. Keep in mind, LLMs are next word predictors at their root. More abstractly, they capture and replay likely human intelligence by way of written language. Tokens.

With that concept in mind, it's clear how they can be made to "give up".

	▲	qsera 6 hours ago \| parent [-]
		>With that concept in mind, it's clear how they can be made to "give up". They can, but they need to be trained specifically on that behavior. They can be trained specifically to not generate textual description of things that look like hacking. But it is going to cost $$$, and as we currently see, most people don't care...