I think you're both right, but Neumann much more so.

It's practically impossible to create a benchmark that can't be gamed when the input space to your system is so very large.

What is the alternative? Test all the queries on all datasets?

I find his argument aligns perfectly with my experiences and expectations moving forward. You can't give these reward-seeking models a broad goal that they can solve narrowly and leave them unattended. The problem is that the rigor is being applied so late that it is ruining the system. I disagree that further improving rigor and decision making is the right path forward because it, in this case, places that burden on the party of last resort. It should be the first human in the loop, if not the model itself, that triages these issues.

Peer review isn't intended to be the first and last stage of review. These people are lazy and their actions deplorable.

A system is what it does, unfortunately.