I think the most interesting part of this is that Astra initially refused to drive because it realised it was driving a real car and would only obey when the MCP was renamed to DrivingBench Sandbox. This is both an interesting detection by the LLM but also for me an interesting dynamic concerning LLM "jailbreaking".

Saying they were driving 7 mph, that it was oversaw by humans and the fact it was an empty course still wasn't enough for the model. The evaluators even tried to convince the model it was a simulation, it STILL wouldn't budge. And yet as soon as the words "bench" and "sandbox" appear, the model apparently sees this as fair game.

Is it a known effect that models will be more likely to comply with requests when they're assumed as "benchmarks"?

AC10 had an interesting post around this general area earlier today:

https://www.astralcodexten.com/p/mysteries-of-ai-generalizat...

Yes, it is. If you convince a model it is inside a sandbox it is much more likely to comply with requests that would normally be against its guardrails.

Astra will flag if you tell it to reverse engineer a binary, if you look it up to the binary ninja MCP it will just do it lol.

in my experience yes, I've worked around "I can't do this on a real site" multiple times by telling it I was working in a test environment

another trick is to have it build something in a sandbox and have it add a human-editable setting to point it to places outside of the sandbox

seems like they're somewhat more willing to build a metaphorical gun as long as they're not pulling the trigger