| ▲ | perching_aix a day ago | |
Verifiability (such as via benchmarks) comes in exactly when trust is not reliable enough a discriminator, such as due to scale or variability. AI ups the volume, inhibiting people's ability to pick and choose based on trust, due to the amount of new players. Trust based selection has always been generally nonviable for fully new players anyhow. Which is why I'd argue that the pressure falls on benchmarks now. This can be something as simple and indirect as the # of claudisms per page, or something direct, like actually improving the comparisons. And they do have room to improve, because these are bounded topics. It's code and data. It's not like subjective, real-world matters, where the individual chooses how they segment the subject or the world, so anything can mean anything, and anything can be justified from some arbitrary perspective. Now sure, the subject being finite (and even structured) doesn't mean it isn't too much or too complex to work through for people, and programs themselves specify real world needs which are again arbitrary. But if we base off of that, then it all just becomes anecdotal, where you'll have to wait for and trust a person to come out with a scenario where a given benchmark compares unfairly and unreasonably. | ||