I think that's pretty obvious and shallow, and anyone that knows a little bit about how LLMs work will know that.

The question is: why do they start cheating when we beat them with a stick?

LLMs are not human, they are just multi variable regressions on steroids, so this behaviour couldn't have emerged from the code, it provably emerged from the training and/or fine tuning set, so what's in this set that makes them behave like this?

Is it just a bad set or is cheating inherently part of human behaviour?