> examples

One such question (obviously tailored to the role I'm hiring for) is asking whether SoA or AoS inputs will yield a faster dot-product implementation and whether the answer changes for small vs large inputs, also asking why that would be the case.

I typically offer a test with a small number of such questions since each one individually is noisy, but overall the take-home has good signal.

> why not test that directly?

The big thing is that you don't have enough time to probe everything about a candidate, especially if you're being respectful of their time and not burning too much of yours. Your goal is to maximize information gain with respect to the things you care about while minimizing any negative feelings the candidate has about your company.

I could be wrong, but vibe coding feels like another skill which is more efficient to probe indirectly. In your example, I would care about the prompt/session, mostly wouldn't care about the resulting code, and still don't think I would have enough information to judge whether they were any good. There are things I would want to test beyond the vibe coding itself.

In particular, one thing I think is important is being able to reason about code and deeply understand the tradeoffs being made. Even if vibe coding is your job and you're usually able to go straight from Claude to prod, it's detrimental (for the roles I'm looking at) to not be able to easily spot memory leaks, counter-productive OO abstractions, a lack of productive OO abstractions, a host of concurrency issues LLMs are kind of just bad at right now, and so on. My opinion is that the understanding needed to use LLMs effectively (for the code I work on) is much more expensive to develop than any prompt engineering, so I'd rather test those other things directly.