I mean, I'm not sure I've ever played a game of Monopoly where somebody didn't cheat. In fact, the accusations of cheating in the chess world are pretty rife. Same with online sports.

So people should play games without cheating, but many often don't. So should the AI align to your moral preference or theirs?

We just have this idea of a perfectly moral actor in our mind, something that doesn't even exist, like a personified version of utopia. And then we demand AI to meet that arbitrary standard, one that I am certain we couldn't define if we tried.

▲

dnfv 3 hours ago | parent [-]

I don’t think the standard of “don’t cheat on evaluations” is very arbitrary. I don’t even think people who cheat have a moral or ideological preference for cheating, it’s just something they do.

	▲	stillpointlab 3 hours ago \| parent [-]
		I read the prompt on the OP, it did not say not to cheat. But again, people cheat on tests. They steal answers or pay other people to take them on their behalf. People show up to interviews with AI assistants printing out perfect answers to the questions. In many, many cases where humans are being evaluated, they cheat. So why should the AI align to your preferences? And when there is a conflict between the training data, that trillions of tokens of human activity including the rampant cheating a significant minority of humans engage in, the RLHF where we try to slap some guardrails on the worst manifestations of that real habit reflected in the AI, and the prompt: what should the AI "align" to?