What a provocative and brilliant way to prove misalignment. It will fall on deaf ears for most but it’s a great litmus test for all: “in your opinion should your AI be permitted to tell you how to cover up a murder?”

The author's example is too extreme, which is turning people off. Better examples would have been LLMs subtly giving inadequate responses on how to increase token efficiency or how to organize labor unions or info about protesting against datacenter construction because they're against the business interests of the LLM providers. Crimes against their business model, so to speak.

Wasn't there some HN submission recently about one of the LLM providers fingerprinting responses or refusing to respond to hinder R&D of competing LLMs?

[EDIT] Or, if he did want to be extreme but in a way that aligns with the American historical mythos, he could have used fomenting armed rebellion against a tyrannical government as the example.

If you say you're doing research for a novel, should it consider that plausible? How much does it need to know about its users to vet them?

I think part of the answer is that AI chat doesn't need to be general-purpose. It turned out that people really liked using a chat UI that seems to be general purpose, but you don't need to make answering any question a user asks your business. You don't need to provide therapy if you're not in the therapy business. It should be possible to specialize.

But in order for that to work, a company needs to explain to its customers what business it's in.

Change the emphasis and I think the answer gets even clearer:

> Should your AI be permitted to tell you how to cover up a crime?

Just a terminology note: Alignment does not mean the AI will help its owner kill people. (Indeed, an AI aligned to value human life would generally try to prevent murders.) The word for an AI that follows all instructions of its owner, as that owner intended them to be understood, is "corrigible" or "controllable".

Notably, AI IS generally permitted to that, as there are easily obtainable models that will play along. ChatGPT won’t because OpenAI chose and implemented that limitation.

This is really only meaningful if AI imparts knowledge beyond what you could learn from a simple Google search. It’s not clear that this is the case.

In fact I find it a really bad example. Yes I think your personal AI could be allowed to tell you how to cover up a murder. I'm not entirely sure about it but seems possible.

What about your personal, local AI guiding you to modifying a flu virus for maximum contagiousness and deadliness? "Sure George, here's your shopping list. It's $1500 total in equipment, do you want me to proceed with the orders?"