Prompts, prompts, prompts.

Give them a time-constrained challenging problem that is wide in scope and see how well a candidate can decompose it before feeding it to AI. You can get pretty high signal within the first 2 prompts. Are they able to effectively steer the model or do they just ride with the flow and accept every AI suggestion?

How well do they know their models and their limitations? Are they able to switch tools/models effectively on the fly depending on the task? Or are they just using cursor auto mode and copy-pasting the task description into their IDE? Do they have a custom harness/workflow? What skills are they using, if any?

Judge them on quality of the output first. And pay attention to their taste.