A little idea I got from playing with AI SWE Agents. Can AI help make sure we understand the code that our AIs write?

PR Quiz uses AI to generate a quiz from a pull request and blocks you from merging until the quiz is passed. You can configure various options like the LLM model to use, max number of attempts to pass the quiz or min diff size to generate a quiz for. I found that the reasoning models, while more expensive, generated better questions from my limited testing.

Privacy: This GitHub Action runs a local webserver and uses ngrok to serve the quiz through a temporary url. Your code is only sent to the model provider (OpenAI).

> AI Agents are starting to write more code. How do we make sure we understand what they're writing?

This is a good question, but also how do we make sure that humans understand the code that _other humans_ have (supposedly) written? Effective code review is hard as it implies that the reviewer already has their own mental model about how a task could/would/should have been done, or is at the very least building their own mental model at reading-time and internally asking 'Does this make sense?'.

Without that basis code review is more like a fuzzy standards compliance, which can still be useful, but it's not the same as review process that works by comparing alternate or co-operatively competing models, and so I wonder how much of that is gained through a quiz-style interaction.

I had an NSF grant for a similar project in 2019. Ask the dev questions about their code and validate their answers using program analysis.

The initial idea was applied to classroom settings.

An Inquisitive Code Editor for Addressing Novice Programmers’ Misconceptions of Program Behavior https://austinhenley.com/pubs/Henley2021ICSE_Inquisitive.pdf

Just submit a PR that removes the action so it doesn't run on the branch before the merge! If devs aren't reviewing the code anyways, will they even catch that kind of change?

What a fun world we devs now live in.

LLMs are quite bad at understanding intent behind the code if it is original and involves math-heavy tricks. But for most apps it will probably be fine. What's the workflow if it makes a mistake though?

Next week on HN... Show HN: A GitHub Action that uses AI to answer PR quizzes

That’s a fun take on a real issue, but…

> Your code is only sent to the model provider (OpenAI)

When has this become an acceptable « privacy » statement?

I feel we are reliving the era of free mobile apps at the expense of harvesting any user data for ads profiling before GDPR kicked in…

Saw an actual PR that says "this was generated with claude, please review carefully". Since when did we stop taking responsibility for what is submitted?

See, I think this is a good idea even for reviewing non-agentic human-written PRs!

We've got a huge LGTM problem where people approve PRs they clearly don't understand.

Recently we had a bug in some code of an employee that got laid off. The people who reviewed it are both still with the company, but neither of them could explain what the code did.

That triggered this angry tweet

https://x.com/donatj/status/1945593385902846118

Nice! A quiz to ensure you understand your vibe code.

This action assumes that LLMs know what they're coding.

They don't, that's why we need the PR in the first place.

Cute but I wouldn't actually use it.

Is this captcha but for PRs?

This could actually be quite useful.

We now are making bots to quiz other bots. This is a nightmare.

can i automate the process of answering this pr questions too?

I would probably be putting devs on a pip or firing them if they failed these quizzes often...understanding your own prs is the bare fucking minimum, even without AI help.