Remix.run Logo
▲ ajb a day ago

The problem is not the input (documentation) it's the evaluation of the output. Managing an AI is like managing a team of specialists, whose subjects you are not a master of. Most programmers did not have the problem of managing experts - we were the experts. There are those that do, but several of the levers you have over a team of humans (incentives, responsibility, even hiring and firing) are not available to control an AI, and the new levers ( fine tuning, training, reinforcement learning) are not in the hands of those relying on it. This is not solved by any amount of documentation.

▲CharlieDigital a day ago | parent [-]

You hand the same documents over to AI agents that evaluate the output.

The same rules that you, an expert would apply, should be documented and used by an agent to apply those heuristics.

There are obviously some areas where there's spot judgement, deviations, and exceptions. But by and large, you have some heuristic that you use when evaluating output. Document it and give it to an agent.

▲ajb a day ago | parent [-]

This suffices for merely random error (as long as the error is demonstrable) but it's not a general solution to the problem.