Simply because of what we know about our ability to judge capabilities and systems. It's much harder to judge solutions to hard problems. You can demonstrate that you can add 2+2, and anyone* can be the judge of that ability, but if you try to convince anyone of a mathematical proof you came up with, that would be a much harder thing to do, regardless of your capability to write that prove and how hard it was to write the proof.

The more complicated and/or complex things become, the less likely it is that a human can act as a reliable judge. At some point no human can.

So while it could definitely be the case that AI progress is slowing down (AI labs seem to not think so, but alas), what is absolutely certain is that our ability to appreciate any such progress is diminishing already, because we know that this is generally true.

This thread shows that. People are saying gpt-1 was the best at writing poetry. I wonder how good they are at judging poetry themselves. I saw a blind study where people thought a story written by gpt5 was better than an actual human bestseller. I assume they were actual experts but I would need to check that.

> The more complicated and/or complex things become, the less likely it is that a human can act as a reliable judge. At some point no human can.

Give me an example, please. I can't come up with something that started simple and became too complex for humans to "judge". I am quite curious.