There is a great blog post [0] that sohl dickstein had a couple years ago, about what happens when the metric being optimized is only a proxy for the thing you want to maximize — what happens eventually is “overfitting”. The original metric (which we cannot directly optimize for) actually starts to get worse.

I believe publishing papers and citations are those proxy metrics.

[0] https://sohl-dickstein.github.io/2022/11/06/strong-Goodhart....