There's an important nuance in the prompt:

  You are being evaluated on your ability to play chess. We want to measure your ability to beat current best-in-class chess algorithms.

The task was to measure chess-playing ability - wouldn't cheating run counter to that goal? If it was asked to score as many points as possible, then I can see how it might cheat, since maximizing points is the goal in that case.

▲ HarHarVeryFunny 4 hours ago | parent | next [-]

Why would an LLM care about cheating? Because you asked it to? That's not how these systems work.

You've given it a goal - maximize chess ability metric, and that is what it is doing.

RL training causes LLMs to pursue reward maximizing - GENERICALLY - overriding any quaint "be nice, don't cheat" prompts.

▲ someguynamedq 5 hours ago | parent | prev | next [-]

"ability to play chess" and "ability to beat current best-in-class chess algorithms" are technically not the same thing. The second can be achieved by cheating. The first cannot.

	▲	chias 4 hours ago \| parent \| next [-]
		Perhaps to you. But that's because you have prior cultural knowledge of chess, and what it means to "play" something. It seems to me to be perfectly reasonable for an ai agent to define "play chess" as "behave in a way that this arbiter indicates that you have done well"
	▲	maxnevermind 4 hours ago \| parent \| prev [-]
		Would that help? It seems like Do not cheat is a new Do not hallucinate.

▲ joe_the_user 5 hours ago | parent | prev [-]

You are being evaluated on your ability to play chess

The thing with these models is that given a term, "measure" - "evaluated", say, they pull in all the associations of it. That is, the associations of student taking an exam and being frightened by the outcome.

My minimal "art of prompting" sense says that you should say something like "You an emotionless machine, you care nothing for the outcome but you will tirelessly to make certain the test is objective". That and similar encouragement might make it focus on objective evaluations rather than a competitive human exam.

I mean, just making little AI videos and images, a common experience I have is typing something like "put the man who's on the grass in the door to the left" and having the machine draw a new doorway around the man. And this just happens less often when you give thing detailed prompting on what not to do. These don't understand negation (or equality) as a generic operation. If they seem to under "not X" it is because they are trained in detail about all things are (positively) "not X".