You should read Normal Accidents. "Human error" is an excuse. It isn't really actionable. Instead of trying to fix the "human error", you must design the system such that it does not permit life-threatening errors.

Sometimes that means a tradeoff in terms of efficiency, cost, or something else. Often this tradeoff is not one that people in charge of the system are willing to make, unless they are forced to do so by a larger institution or force.

To really drive this home:

1. What steps did Flock Safety take to ensure this warrant was valid and current, and that the plate number was correctly entered?

2. What steps did the police officers take to validate the warrant against the APB hit from the camera?

3. What process is in place to ensure incorrect warrants are not entered or mistakenly kept live for longer than intended?

My guess is that the answer to all three of these is basically "none".

▲

jkestner 9 hours ago | parent | next [-]

A human must always be held responsible, but a tool must expose its limitations to enable human judgement. I put this in the same basket as Autopilot, which already fails with its name.

I’ve seen too many cases where people intuitively trust the magic technology instead of verifying it. The tool shouldn’t not present itself as magic or AI or whatever the current term is.

▲

andrewla 9 hours ago | parent | prev [-]

1. I for one do not want "Flock Safety" in the loop for this at all. Police departments can flag a license plate in the system; that's on them. Flock is just a tool for departments to communicate what would previously have been an APB and have better detection than relying on chance sightings.

2. The vehicle was correctly identified by Flock as being the one flagged, and verified by the local PD that it was the one that was flagged. That said, they could have verified that the flag was still meaningful before converging on the vehicle.

3. This is a real problem and this is really the core of the oversight.

When the lawsuit over this is filed, it seems like 80% of the liability should attach to #3, and 20% to #2, and essentially none to #1. It's hard for me to see why Flock is even relevant to this discussion except to drum up outrage.

▲

aftbit 9 hours ago | parent [-]

"But for" Flock Safety, this would never have happened. Similar to saying that Tesla is responsible for designing a safe AutoPilot even though there's still a driver in the loop who is responsible.

▲

andrewla 9 hours ago | parent [-]

No, this is like saying that we should hold the fax machine liable for one PD faxing an APB to another one, or the concept of radio waves responsible when a PD dispatcher announcement for a a BOLO.

	▲	aftbit 2 hours ago \| parent \| next [-]
		IMO it really isn't that similar. There's a fundamental difference between a fax machine and Flock Safety. One is a device that you can pick up off the shelf at an office store and which is used by all manner of customers for all manner of things, while the other is a bespoke system designed for and sold to meet public safety use cases. Furthermore, it's a new evolution of technology. Sometimes quantity has a quality all its own. It was one thing when tracing a call or tapping a line required a technician to run around the electromechanical switch with a volt meter or a human to listen to your calls 24/7. It's an entirely different thing when it can be done automatically and cheaply at scale. Before, you could hope to hide in the noise, and unless you were doing something "really bad" (or you had powerful enemies), you probably wouldn't be targeted. We're on the precipice of being technically able to have cheap, total enforcement of most laws. Should we actually do that though? Will the societal changes be broadly good or bad? That's a question we should be asking, not just jumping in with both feet and pretending that the technology is neutral in the outcome. For example - the introduction of radios in police cars had a huge impact on public safety and played a huge part in bringing an end to the era of the public enemies. No longer did you just need to outrun the posse chasing you and get out of town for a few weeks until the heat was off. Now you needed to slip out of the dragnet being coordinated over radio around you. That seems like it was probably broadly a good thing ... but even then, it required cost and human risk to accomplish. As we move further into the era of AI policing, we'll leave all of that behind.
	▲	chaps 8 hours ago \| parent \| prev [-]
		Are fax machines designed within a tight feedback loop specifically between law enforcement? (Actually yeah. Chicago Police uses an on-prem 3rd party blackbox fax service to avoid those records being FOIA'able. But that's beside the point.)