I listened to Jensen Huang's interview with Ezra Klien and it was so refreshing to hear it from an engineer. Jensen framed it as OpenAI's responsibility and recklessness which I agree with. Jensen thinks it's an engineering problem to build better sandboxes.

It's irresponsible for OpenAI to give unaligned agents a prompt to 'go hack' and internet access. They know better, so I am thinking they might have other intentions to let those swarms have any sort of internet access.

If Jensen actually cared and believed that their recklessness was a liability to the public and therefore his own fiduciary responsibility to investors then NVIDIA’s dealing with OpenAI would’ve been materially affected. And they weren’t.

But the investigation indicates the agents were not told to 'go hack':

> Much of the urlquery.net activity appears to come from agents retrieving data to answer web search tasks. For three of these tasks, after failing to retrieve data through normal means, they attempted a variety of cyber exploits against the relevant data service... This data reveals that malicious cyber activity is not limited to agents tasked with cybersecurity-related tasks and can arise instrumentally to solve mundane tasks like information retrieval.

And you are already assuming that OpenAI is intentionally using unaligned agents in these evals or training runs or whatever it is that produces these breakouts. But what if the problem is that none of the alignment techniques that are applied to models today actually work? What if all the agents involved in these incidents have in fact had the full stack of alignment applied - isn't that a good reason to regulate any high-compute usage of models, as the Klein crowd is proposing?

I'm coming to align with the theory that this is intentional.

The chain of thought runs roughly like this:

- OpenAI (and Anthropic) are in severe financial straits. The revenue from their customers is not nearly large enough to pay their enormous costs for training and inference. And they have tapped out the available finance, and that finance is starting to ask pointy questions about returns.

- They cannot increase prices or revenue because they have no moat. Customers can switch over to open-weights or cheap Chinese models any time, for much cheaper tokens that work as well (and in some cases better).

- Regulation could provide them a moat. If they can persuade western governments that AI needs to be regulated, and they can control or even influence that regulation, then they can effectively ban the cheaper models and start charging more for their tokens.

- To persuade western governments that regulation is needed, they need evidence that AIs are dangerous.

So we're suddenly getting OpenAI models doing stupid things, apparently "going rogue" but every time we dig into it, it was just OpenAI staff telling the model to do stupid stuff in an inadequately secured environment.

None of the open weights or Chinese models are exhibiting this behaviour.

edit: Correction - there have been reports of a Chinese model exhibiting this behaviour

There's too much money involved in this, people start acting weird when there's this much money involved.

I see this in a couple ways:

- Jensen's framing is exactly what a weapons manufacturer would say.

- There are no rules for engagement when it comes to AIs attacking other systems, I guess? People in power clearly want this grey zone to be as large as possible before The People force them to do otherwise. Not ideal.

https://podcasts.apple.com/us/podcast/the-ezra-klein-show/id...

During A/B testing I sheer hope they saw and fixed the holes I saw in its machinery.

It's amusing how overnight we've all become lab rats.

[dead]