I've seen you comment along those lines before, and I think you're both right, and terribly wrong.

By all measures, the way you describe any hiring processes you've designed and had input on designing the metric and the work sample, probably is both high in signal, and not very arbitrary in scoring. They sound genuinely and refreshingly good. Maybe one day I'll interact with one :)

But man, that is very much not how work sample tests work in _a lot_ of places out there.

Places with actual, formalized rubrics for what constitutes a "pass" or a "good" score are very rare, it's almost always just based on vibes of whatever person is reading your code. If there _are_ formalized rubrics, then they have "suggested allowed time" that is incongruous with what the requirements for a "pass" actually are.

All of that is downstream, fundamentally, from the design of the tests not being taken very seriously by the people doing that. And because they're not actually thinking about it very deeply (or they're not allowed to because of time constraints or whatever); "sit down and redesign this task from first principles thinking about what AI enables now" is just not something that happens, and you just get people increasing the scopes of the project blindly.

I agree that there aren't many places that are serious about work-sample testing and far too many that using them as a hiring hazing ritual, but that doesn't make me "terribly wrong", just more distinctively right. We've hired resume-blind at Fly.io for 6 years using work samples with fixed rubrics.