"The final text asks some AI companies to submit their powerful new models to a voluntary government review 30 days before releasing the products to the public, a pause that would give federal agencies some time to gauge what threats the products may pose to sensitive financial, national security and other computer systems."

How specifically does that review work? I want to give federal agency Opus 4.8 now, while 4.7 has been out for a while (leaving Mythos aside for now). They have 30 days to figure out whether it poses a threat.

How do you do that? Is there an eval for this and if there is why can't they just make it public? What is the agencies objective (but proprietary?) analysis here?

I seriously doubt even the government actually knows or has a real plan, let alone one actually related to security. If it's anything like their track record, they'll just be asking the AI about a topic related to their enemies (i.e. anyone opposed to them in any way) to see if it says anything remotely positive about them, or anything remotely critical of the regime or out of line with the regime's "alternative facts".

The review is they ask it about the epstein files and ensure any other politically sensitive topics have the “right” answers.

Just do a VW and detect when you might be in the testing phase. Off the top of my head:

Train it dumb on "systems:, user:" prompt pairs.

Unleash on "system:, user:" prompt pairs.

Guess which you're providing for evaluation.

Self-report and self regulation, kind of like Boeing with FAA ... so not functional in long term

It's in the text of the order, it directs NIST to:

> develop and maintain a classified benchmarking process to assess the advanced cyber capabilities of AI models and determine the threshold at which an AI model should be designated a “covered frontier model” for the purposes of this order

It's just so Elon Musk gets to personally delay releases so Grok can maybe ever gain any meaningful traction...

> Is there an eval for this and if there is why can't they just make it public?

For the same reason the CIA doesn't publish the Windows exploits it finds?

[dead]