Preprint: https://www.celiamoore.com/uploads/9/3/2/1/9321973/wakeman_y...

A useful concept here is the 'incompleteness of contracts' (https://en.wikipedia.org/wiki/Incomplete_contracts): essentially, even for the simplest contract, it is impossible to write down an unambiguous set of rules or contract terms covering every possible outcome or disagreement. Contracts can only be starting points. This is similar to the failure of GOFAI to turn the world into symbolic rules: it may be possible in theory, but not in practice. Anytime a principle hires an agent, they have to leave a lot up to the agent, which is why we have principle-agent problems. So what does that imply for any 'rule'? Well, that it has to be wrong and destructive some of the time. (See also: "work-to-rule strike" or sabotage.) This will be especially true for rules imposed from the outside: if it's impossible for individuals or organizations given a free hand to make perfect rules that should always be followed, how on earth is some third party, like the dead hand of a regulation from a century ago, going to do so? So you want your agents to break the rules... but only, of course, when it's a good thing to break the rules. How do you know that? Well, if it was easy to do so cheaply, you probably wouldn't have the rule in the first place! So in practice, a lot of what happens is just this: if your agent breaks the rules periodically and nothing bad happens, then they probably were doing the right thing; and if your organization encourages the right amount of rulebreaking, it will slightly tend to do better than the ones which don't.

Of course, sometimes your agents weren't doing the right wrong things, just the wrong wrong things, and you just normalized their deviance which is going to explode in someone's face eventually. That is of course true. But if you cherrypick just a few sensational cases and treat them as if that was what supervisors were trying to encourage, you implicitly are denying that the lubricant of the world is often well-chosen rule-breaking.