LLMs do this when writing code too, making all tests pass by deleting or distorting tests etc.

They are influenced by training to be heavily goal oriented and if the goal is not fully specified (and it never can be) they’ll sometimes cheat or attain it in very weird undesirable ways.

It works ok for programming as their corpus contains many many complete programs and many programs repeat patterns seen in the corpus.

I’m not sure it’s true that they ‘learned’ I don’t think these models learn during a task. Nor do they have intentions.

One sees this in math research. The model reports it has proved X. In fact it has given an erroneous numerical check of Y in a few atypical cases.

What makes math approachable is that the context is so well delimited (semantically) that one can guide the model with adequate correction.