I've been considering different ways to measure code understanding and fluency. I'm still using traditional 'easy' problems on a coding pad to probe this, but I'm finding a new generation candidates that don't prepare for this, and may be able to have a full career with just a higher level of code understanding. But they might not ever develop the ride-a-bike with one hand level of fluency unless they grind pointless leetcodes.

So I'm increasingly uncertain about what and how to test. My default for now is still to rely on ability to write basic code fluently, but I'm open to changing this perspective.

I really want to know how this existing repo AI-assisted live coding test works, with example problems.

It seems the standard data structure puzzle type thing won't be feasible if you are using an LLM.

Also the latency for these agentic coding/prompts seems like it would make the interview a bit awkward.

Anyone been conducting or taking interviews with this kind of thing with thoughts to share?

▲

mbil 2 hours ago | parent [-]

I’ve been conducting AI Coding interviews at my job. Candidate will screen share and use AI-assisted coding environment and tools of their choice.

I ask them to implement xyz thing. What I’m looking for is how they interact with the AI agent. Do they ask the agent to plan first? Do they review the plan? Etc. It’s pretty typical stuff that you might expect an experienced engineer to do if they effectively use such tools daily.

There are a series of follow-up questions about how to productionize the toy system, which gives some additional signal about how well they understand what they’re making. I sprinkle these in when there’s dead air waiting for the bot to think.

I think we’ll need to evolve and refine this problem and process as the models continue to improve.

▲

Muromec 2 hours ago | parent [-]

>It’s pretty typical stuff that you might expect an experienced engineer to do if they effectively use such tools daily.

It's also the kind of stuff that somebody can learn in a week, so not hiring the right person who just didn't spend this week of time yet for whatever reason is a loss.

▲

pmg101 2 hours ago | parent [-]

Yeah it tells you nothing of value. It's the Recency Bias taken to its ultimate conclusion isn't it. Anything I learned about AI I learned in the last few months but somehow that's as important as all the experience of my decades long software career? Come on.

▲

Muromec 2 hours ago | parent [-]

It signals you don't have enough time or will to chase the latest stuff, otherwise you would have learned it with other cool kids half a year earlier. It carries some signal, but I would not choose based on this alone. I would even consider selecting against this to a certain point, but I also work in a place that COBOL on a mainframe. YMMV.

	▲	dan_q 2 hours ago \| parent [-]
		> I would even consider selecting against this to a certain point I agree. A technology professional who views themselves as "one of the cool kids" (read: easily manipulated by social media) is a legitimate security threat, as are many of the popularly promoted approaches to "LLM-assisted software developement".