At work we had a dispute over if AI should be allowed in the technical interview, we resolved it by both running an AI allowed and not allowed interview. Something interesting we found is that every candidate either passed or failed both. People who could not program manually without AI were not able to get the agent to complete the tasks either.

I've seen people type questions in to the LLM and get the answer they asked for but not the one they needed/wanted because they didn't have the correct terminology.