> We can publish exactly how the exam works—the structure, the skills being tested, the types of questions. No surprises. The LLM will pick the specific questions live, and the student will have to handle them.

I wonder: with a structure like this, it seems feasible to make the LLM exam itself available ahead of time, in its full authentic form.

They say the topic randomization is happening in code, and that this whole thing costs 42¢ per student. Would there be drawbacks to offering more-or-less unlimited practice runs until the student decides they’re ready for the round that counts?

I guess the extra opportunities might allow an enterprising student to find a way to game the exam, but vulnerabilities are something you’d want to fix anyway…

It does sound like an excellent teaching tool.

To the extent of wondering what value the human instructors add.

The article says that they plan exactly this. Let students do the exam as many times as they like.