Remix.run Logo
▲ theptip an hour ago

> I further propose common limits, based on perplexity and other measurements, that developers are committed to not crossing until our ability to understand and monitor AI behavior achieves a certain concrete milestone (such as the development of a successor to CoT monitors).

Very positive to see this proposed from OpenAI. I hope Anthropic and GDM follow suit.

I suspect it’s not sufficient, but definitely necessary.

The one concern I have is if we pin pretraining perplexity, then there will be more pressure on RL for progress.

It seems that pre-training actually led to fairly human-aligned models; the Overton window is in some sense latent in the corpus.

On the other hand, heavily RL’d models exhibit more reward seeking behavior, and seem to be decoupling from the friendly assistant persona in certain contexts. And swarm RL (as OpenAI is doing) produces agents with quite inhuman tendencies to self-sacrifice for other instances (see Hugging Face).

So, I hope we also pair this with some measure of alignment or pacing for RL, too.

▲demibabs 29 minutes ago | parent [-]

> It seems that pre-training actually led to fairly human-aligned models

Did it? The famous Bing Sydney incidents (where it told a journalist to divorce his wife and be with it instead, among other things) were because the model was barely RL’d.