It's weird that people think AI breaks the concept of work sample testing. Work sample testing isn't "about" programming, and predates the profession of programming. You can (and some companies do) work-sample test sales account managers, customer support, accountants, whatever.

AI changes the underlying job you're testing for. So, obviously, the tests you might have been using pre-AI won't work anymore; they're testing something that isn't really the job anymore. Update your tests so they're about the real work again, that's all. For coding, that probably means assuming (or requiring) candidates use AI to do your assessment.

What AI really does mess with is conversational/interactive interviewing. We do all our interactive scripted interview on Slack, but I can imagine us having to end that practice and return to face-to-face.

▲

klausa 2 hours ago | parent [-]

I've seen you comment along those lines before, and I think you're both right, and terribly wrong.

By all measures, the way you describe any hiring processes you've designed and had input on designing the metric and the work sample, probably is both high in signal, and not very arbitrary in scoring. They sound genuinely and refreshingly good. Maybe one day I'll interact with one :)

But man, that is very much not how work sample tests work in _a lot_ of places out there.

Places with actual, formalized rubrics for what constitutes a "pass" or a "good" score are very rare, it's almost always just based on vibes of whatever person is reading your code. If there _are_ formalized rubrics, then they have "suggested allowed time" that is incongruous with what the requirements for a "pass" actually are.

All of that is downstream, fundamentally, from the design of the tests not being taken very seriously by the people doing that. And because they're not actually thinking about it very deeply (or they're not allowed to because of time constraints or whatever); "sit down and redesign this task from first principles thinking about what AI enables now" is just not something that happens, and you just get people increasing the scopes of the project blindly.

▲

tptacek 2 hours ago | parent [-]

I agree that there aren't many places that are serious about work-sample testing and far too many that using them as a hiring hazing ritual, but that doesn't make me "terribly wrong", just more distinctively right. We've hired resume-blind at Fly.io for 6 years using work samples with fixed rubrics.

	▲	klausa 2 hours ago \| parent [-]
		Sure, it was a rhetorical flourish that maybe didn't land :)