Remix.run Logo
jaggederest 7 hours ago

You stop relying on the agents following instructions exactly.

You need two pieces:

a) prompts, that tell the agents what to do and how to do it (and ideally, the why, where, etc, the full picture) - that's the positive half, that drives behavior the way you want it.

b) deterministic tooling that prevents negative outcomes, like linters, compilers, static analysis, fuzzing, testing, the more the better. This side should either be firewalled off from the AI or very carefully watched so that it doesn't drift.

The part that you put in the deterministic side is the "never do x" stuff - I have lint for long comments (which AI hits every single time it commits), all my dev scripts are in typescript, precommit hooks, massive CI, and I lint even for things like redirecting error to standard out, tiny stuff, and also e.g. static migration analysis so the AI never ships an exclusive full table lock in a migration, for example.

sarchertech 7 hours ago | parent | next [-]

You can’t deterministically keep them from making even a tiny fraction of all the possible errors they can and do make though.

You can’t keep humans from making those errors either but you also don’t let an error prone human crank out 20k LOC per day without forcing other humans to understand it.

CharlieDigital 6 hours ago | parent | next [-]

This may be platform dependent.

C# Roslyn Analyzers[0], for example, are quite powerful and can identify complex patterns in code. One approach to deterministic enforcement would be to ensure that the project is set up with an analyzers library and mistakes that can be deterministically flagged are

[0] https://learn.microsoft.com/en-us/visualstudio/code-quality/...

jaggederest 6 hours ago | parent | prev [-]

I don't know if that's necessarily true in practice the way it is in theory. If you knock down 95% of the issues they create over a certain period of time, in practice you'll have better code than a human would write, and it's all upside from there.

I've been using languages with stronger type systems and that's also a huge boon.

monknomo 7 hours ago | parent | prev [-]

yes, this is a very helpful approach. I take a similar approach, partly because I find bringing up negative directions seem to bring the undesired action/direction into the agent's view and makes it more likely that they will go there