It still proves something much narrower. Wherein AI is insufficient to answer, people are apt to rely on it anyway.

A good real-world issue is health, where the issues are very complicated with many things poorly defined even at the state of the art where practitioners are relying on personal judgement and lots of data but patients are apt to feed ai very little data compared to what their doctor has.

Real frontier models can remain confidently incorrect in these cases.

It would be interesting to measure not just the accuracy, but how many people actually decided to double check the answer