> You can get the LLM to run a script which checks for all of these

Most of the time. Except for when it forgets to do it.

I think it’s funny that the solution is to use something that is not LLM driven to enforce it.

Also - pre commit hooks aren’t enforced, people will not set them up. You have to run this stuff in CI (which is incredibly annoying given that machines are writing the code in the first place)

▲

Yokolos 2 hours ago | parent [-]

This is the most frustrating part. You do everything you can to ensure there are clear instructions, you can keep the agent MD as concise and clear and short as possible. It still feels like it's all just a suggestion, and of course it is, because it's all just another part of the prompt.

▲

maccard an hour ago | parent [-]

I’m in the “AI can be great, but it’s not right now” camp. I think that pulling the verification into the harness and having the harness execute it rather than the agent would genuinely make AI go to usable for me. But even prototyping a custom harness requires API billing which is just so expensive…

	▲	DCKing 20 minutes ago \| parent [-]
		You can put many agent constraints in precommit hooks if they're static checks. I ask agents to make commits, and e.g. in a Python project have the precommit hook fire off type checks, linting and even architectural things like import boundaries (using `tach`). When an agent is prepped to make commits themselves, it will catch pre-commit failing and correct itself. The existence of static checks themselves might also help agents gain awareness of the overall verification flow including larger things like tests, but that's hard to say for certain. Putting structural code checks in a precommit hook is arguably better than pulling it into the harness, as it will enforce those constraints no matter whether an agent or human is making the commit.