Karpathys LLM wiki with QMD.

I have a small gatekeeper.py script the agent has to run to complete a round or begin a new one. One of the things it does (in addition to record keeping, linting, etc) is validate that the wiki has been updated with all the findings, lessons learned, and documentation for the round. The agent can't proceed until it's done.

Then in the AGENTS.md they're told they can use QMD to perform semantic search of the wiki , and should before starting any significant work.

It's expensive, in terms of tokens, but well worth it. I could reproduce any stage of my project if I wanted. I always know why they did what they did (if they do), and the agent's can't really forget anything important.

▲

spockz 2 days ago | parent [-]

I’ve been trying to get something similar to work with local markdowns instead. Getting them to write something is easy. But I cannot reliably trigger them to write when they make “sufficiently important” decisions. There is little self reflection (or actual intelligence).

How do you get them to trigger writing to the log in the right moment and manner?

	▲	mapontosevenths 2 days ago \| parent \| next [-]
		The karpathy wiki is really just local markdown at its core. The semantic search qmd provides is really the fancy bit. My system is fairly elaborate, but the simple trick is to build the loop so they have mechanical checks that prevent anything from counting as done until it meets standards. There are many ways to do that. A lot of folks just hook specific tool calls and make it output relèvent instructions. In my loop I use an Orchestrator and a Builder sub agent. The Orchestrator assigns the Builder work from the spec one phase at a time. The Builder has to build each phase with red then green unit tests as specified by the Orchestrator. The Builder sub agent only knows that one task. It can't consider the task finished until gatekeeper passes. Gatekeeper is a simple python script that the Orchestrator is allowed to update whenever it sees an anti-pattern. It validates that all the unit tests pass, that it HAS sufficient test coverage, that it lints properly (including custom rules the Orchestrator can add for repeated failure), that the docs exist for that phase and are long enough, and finally does some ast grep searches for common goofs, etc. Sometimes the builder will have to try many times before it passes, but each time it fails gatekeeper outputs an instruction telling them which rule they violated and the full requirements. Eventually gatekeeper will pass and the Orchestrator does a final review before assigning the next phase to a new Builder. The most common failure at this point is poor unit tests that don't test for the right thing. The Orchestrator has explicit instructions to check for that and does catch it most of the time.
	▲	matthiasforstr 2 days ago \| parent \| prev [-]
		[flagged]