I'm really torn on this.

This particular situation, at face value at least, sounds like a good outcome. Kids had a gun, someone noticed, called police, situation handled without escalation or violence or something tragic happening.

But this doesn't generalize. Another incident described in the article was when some kids were drinking in the car and shooting toy guns. What if they hadn't been drinking? What if they were just holding toy guns, innocently going from one kid's house to another? What if Waymo thought the guns were real, called the cops, and the cops screwed up and shot one of the kids? These sorts of escalations aren't imaginary; they happen, and they end in tragedy.

The article is also a little cagey on how this "detection" happened. Was it a human spot checking cameras in vehicles? I doubt it. Most likely it was some sort of computer vision algorithm that flagged something. Perhaps a human was in the loop to verify and be the one to call the cops, but humans seem to be increasingly shutting off their brains and trusting the machine. (I recall a recent incident where someone was wrongfully arrested because one of the companies that run the license-plate reader machines flagged their car as stolen. The machine screwed up the image recognition, but the cops blindly trusted it and arrested the guy.)

What happens when cops are allowed real-time access to Waymo's in-car video footage? Maybe they even have it already. This is just more pervasive, always-on surveillance, and this will only get worse as autonomous vehicles become more popular in the taxi scene.