| ▲ | a20eac1d 9 hours ago | ||||||||||||||||||||||
Can you give more info on how you use/prompt those LLMs for code review and what kind of prompts you use? I've had worse experiences doing it because the quality of answer has been quite bad, and I'm wondering if my methods are the reason. | |||||||||||||||||||||||
| ▲ | lionkor 9 hours ago | parent [-] | ||||||||||||||||||||||
Yes, gladly! I have not yet open-sourced my skills etc., but I can give some insight and share a couple here. Review is a skill, as in, a SKILL.md with a folder full of references: - SKILL.md: https://gist.github.com/lionkor/161525be858d1d75db4c13c0f093... - references/output-contract.md: https://gist.github.com/lionkor/8c68e33becef7a21f8408c7dc119... - references/review-lenses.md: https://gist.github.com/lionkor/0a8b080fe45306213efddf3ebb75... - references/review-workflow.md: https://gist.github.com/lionkor/d2d374b133ceb7e3660bd530ee72... - references/section-rules.md: https://gist.github.com/lionkor/8a9e503adc7fd3697410cf021f27... I'm aware that almost all of this is prompt voodoo, and there's no guarantee for the review to find anything or everything, but making it a dedicated skill and thoroughly observing the output thinking, tool calls, and result, lets me adjust these over time and fill the weak spots with even more prompting. I use this skill by simply telling the agent something like "Review the changes on the current branch against origin/main, take special care with backwards-incompatible changes to the public API" or something like that. I use `pi` (pi.dev) with a subagents extension, so that I can ask the agent to invoke a subagent to do the review, on work that the agent did. For models, I use the highest possible reasoning on whatever model I feel like makes sense, usually this is GPT-5.5 or deepseek flash/pro, depending on the confidentiality of the codebase, on the highest reasoning always (for reviews). I've also had success with a review checklist, though it doesn't produce an easy to parse (for humans) output: https://gist.github.com/lionkor/054ac2cf241e0765eee2383f0dba... This is why my review skill mandates a very strict output contract. I need the output to be very easy to parse, and the output contract I've specified there does that. In general I let <whatever the latest model of OpenAI's ChatGPT is> author and review SKILL.md and similar large prompts, usually with a ruleset like this, which is a 1600 line research artifact from a long GPT 5.5 "Pro" research session on prompt engineering: https://gist.github.com/lionkor/71498794d0a7d72173fc58766f25... Does the review catch all issues? Not at all. Does it catch, usually more than one, important issue, across large changesets? Absolutely, and that's the point! :) Feel free to ask me any questions, I'm also happy to share more about my setup via email or add you or anyone else to my private repos with more of these. | |||||||||||||||||||||||
| |||||||||||||||||||||||