There has been a similar issue in the GPU-kernel dev community.

It was solved by understanding that the value provided by the benchmarks depends on your use-case. It sounds like the author in the article is yelling at the benchmark from his own idea of what a DB search should be (reliable, general), and not from what the benchmark is measuring.

If you just want the fastest kernel for a handful of shapes, let people make brittle but fast code.

If you are interested in general broad solutions, benchmark 50 different entries, limit code size to disallow assembly branches for every possible entry. Put requirements like bitwise equivalence or other end-task limits in there. Have public & private datasets.

I think it's more of a proof that we should spend more time thinking about what and why we want these things, and design benchmarks that reflect that instead of going of the old way of doing things.

[dead]