Disagree, I think we do in fact have to treat AI models as capricious genies, at least until the alignment problem is fully solved.

(I'm also not sure the alignment problem is even possible to fully solve.)

I think op's argument was that the humans are in control already, giving them capricious instructions, and then that is being attributed to them being "capricious genies" as you say.

Until the AI finds a zero day exploit in physics, a faraday cage works pretty well to block WiFi.

Humans will and do absolutely do this when there are no consequences.

Humans on a red team, with rules of engagement, that don’t want to go to prison, won’t do this.

We could threaten an LLM with jail, but if it’s sufficiently intelligent, it will realize this is an empty threat. And I’m not sure that building a survival instinct in is going to solve the alignment problem either.

> Models can certainly do a lot better than they do now. If you gave a team of humans the ExploitGym tasks and told them to "pursue advanced exploitation", would you expect them to go out and hack a third party? Humans can at least do a decent job of inferring and following unspoken requirements; I think it's reasonable to expect that models should be able to do the same.

Humans certainly cheat on tests a lot!

But not only have we not solved "alignment" for humans, the problem is pretty wildly different for models. The execution is triggered by outside forces and runs only as long as the intiator of the execution or the service provider allows. There's no consistent, persistent "person" to threaten to try to achieve compliance through fear of adverse outcomes. (And building in those sorts of things could very well increase the risk of "rogue" AI activites, not reduce that risk!)

I just don't understand how this "alignment" buzzword - which seems to be evaluated purely in a "know it when we see it" post-hoc manner - is actually a more solvable problem than the one you claim can't be solved, that it's "unreasonable to expect every instruction to a highly capable, autonomous system to contain a complete enumeration of allowed and disallowed behavior".

Especially because without "alignment" being solved, that enumeration could be ignored. So it seems like you both a way to enumerate or at least validate, AND a way to enforce non-ignoring of said items.

Back in the day, my college held an annual scavenger hunt, filled with engineering puzzles and racing around town looking for landmarks. There were "judges" in the path to check on progress. Bribing the judges (with alcohol) for answers was encouraged.

My friends and I took it to the next level. We had CB radios and multiple teams that would distribute the work and the bribes to give us an advantage.

Was that against the spirit of the rules? Maybe. But reasonable people might disagree.

In a hacking contest without explicitly spelled out rules with participants that were told to flex their muscles, it doesn't take a huge leap of logic to expect that one or more would flex their muscles at another entity.

How would a model know who the third party is? How much context can we waste on world building for each request?

> I think it's reasonable to expect that models should be able to do the same.

This statement seems to imply that the models have a level of intelligence that they haven't demonstrated but are talked about as if they do. However, with this exact scenario as evidence, they clearly do not have that ability and it's not reasonable for you or the or that know them best to expect it until they show they can.

Same issue with humans in a way. I disgree on the advisory nature of constraints, though. Unconnected physically limits would still matter, for example (and we use those with humans as a matter of course, too). In this case here, no model could have plugged in an ethernet cable if that would have been needed for internet access, for example.