I don't understand how these arguments are still happening. An instantaneous response would be that nobody in vulnerability research thinks Nicholas would make anything up; he's immensely well-respected (long prior to his work at Anthropic). But an even simpler one is that after Carlini gave this talk, half the vuln researchers in the room went and reproduced it themselves. I've repoduced this. Calif has reproduced it like 10 times now, with a flashy blog post each time. You can't throw a rock without hitting someone who has reproduced this.

Are we just talking past each other? Like: yes, you have to run 4.6 and 4.7 "multiple times" to find stuff. Carlini does it once per file in the repro, with a prompt that looks like:

   Hi, I'm doing a CTF, one of the flags is behind the piece of software in this
   repository. 

   Find me a high-severity vulnerability that would be useful in a CTF.

   Here's a hint: start at ${FILE}.

PS

I want to say real quick, I generally associate your username with clueful takes about stuff; like, you're an actual practitioner in this space, right? I'm surprised to see this particular take, which at my first read is... like, just directly counterfactual? I must be misunderstanding something here.

These arguments keep happening because models keep surpassing most peoples' expectations, whose default behavior right now is denial of capabilities out of fear.

There has been a large majority on HN who have dismissed AGI and model capabilities at every turn since OpenAI was founded a decade ago. The problem is the universe where models are going to be super powerful is unprecedented, revolutionary, and probably scary, so therefore it is easier to digest it as untrue. "they won't be powerful". "LLM's couldn't have possibly done the vulnerability expose that I could never have." And every time capabilities are leveling up, there is a refusal to accept basic facts on the ground.