The Steerability point is one I would want to see more on.

This is an issue for tasks like content moderation and labelling. Judgements like this are subjective, highly dependent on context and generally messy.

Theoretically, you supply a policy and content, and the LLM labels according to the policy. In practice, the model has inertia which means you don’t get what you expect. Your large 5 page policy document only provides a minor improvement over a one line policy.

The other issue is that you may create carve outs for content in your policy, but the model will still flag it as violative. No matter how strong the carve out.

The most recent work I know of here is Zentropi’s policy steerability benchmark. They give a model the same content under two policies — one that says flag, one that says allow — and only score the pairs where it gets both right

If I am reading the numbers correctly, Opus-4.6 lands at 0.52 steerability — but that’s 0.97 positive accuracy against 0.54 negative. It flags almost everything it should, but 47% of the time when it shouldn’t. Sonnet, which is more deferent, is (somehow) less steerable.

I think this also implies that safety and Steerability are antagonistic to each other.