> With single modality sensors, you have no way of truly detecting failures in that modality, other than hacks like time-series normalizing (aka expected scenarios).

"A man with a watch always knows what time it is. If he gains another, he is never sure"

Most safety critical systems actually need at least three redundant sensors. Two is kinda useless: if they disagree, which is right?

EDIT:

> If multiple sensor modalities disagree, even without sensor fusion, you can at least assume something might be awry and drop into a maximum safety operation mode.

This is not always possible. You're on a two lane road. Your vision system tells you there's a pedestrian in your lane. Your LIDAR says the pedestrian is actually in the other lane. There's enough time for a lane change, but not to stop.

What do you do?

▲

esafak 5 days ago | parent | next [-]

> Two is kinda useless: if they disagree, which is right?

They don't work by merely taking a straw poll. They effectively build the joint probability distribution, which improves accuracy with any number of sensors, including two.

> You're on a two lane road. Your vision system tells you there's a pedestrian in your lane. Your LIDAR says the pedestrian is actually in the other lane. There's enough time for a lane change, but not to stop.

Any realistic system would see them long before your eyes do. If you are so worried, override the AI in the moment.

▲

AnIrishDuck 5 days ago | parent | next [-]

> They don't work by merely taking a straw poll. They effectively build the joint probability distribution, which improves accuracy with any number of sensors, including two.

Lots of safety critical systems actually do operate by "voting". The space shuttle control computers are one famous example [1], but there are plenty of others in aerospace. I have personally worked on a few such systems.

It's the simplest thing that can obviously work. Simplicity is a virtue when safety is involved.

You can of course do sensor fusion and other more complicated things, but the core problem I outlined remains.

> If you are so worried, override the AI in the moment.

This is sneakily inserting a third set of sensors (your own). It can be a valid solution to the problem, but Waymo famously does not have a steering wheel you can just hop behind.

This might seem like an edge case, but edge cases matter when failure might kill somebody.

1. https://space.stackexchange.com/questions/9827/if-the-space-...

▲

mafuy 4 days ago | parent | next [-]

Voting is used when the systems are equivalent, e.g. 3 identical computers, where one might have a bit flip.

This is completely different from systems that cover different domains, like vision and lidar.

▲

sfifs 4 days ago | parent | prev [-]

Isn't the historical voting pattern something more of a legacy thing dictated by limited edge compute of the past vs necessarily a best practice.

I see in many domains a tendency to oversimplify decision making algorithms for human understanding convenience (eg vote rather that develop a joint probability distribution in this case, supply chain and manufacturing in particular seem to love rules of thumb) rather than use better algorithms that modern compute enables higher performance, safety etc

▲

AnIrishDuck 4 days ago | parent [-]

This is an interesting question where I do not know the answer.

I will not pretend to be an expert. I would suggest that "human understanding convenience" is pretty important in safety domains. The famous Brian Kernighan quote comes to mind:

> Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?

When it comes to obscure corner cases, it seems to me that simpler is better. But Waymo does seem to have chosen a different path! They employ a lot of smart folk, and appear to be the state of the art for autonomous driving. I wouldn't bet against them.

	▲	ImPostingOnHN 2 days ago \| parent [-]
		Seatbelt mechanisms are complicated, airbag timing is complicated, let's just do away with them entirely in the name of simplicity? No, when it comes to not killing people, I'd say that safer is usually better. Remember the core function of the system is safety, simplicity is nice to have, but explicitly not as important. That said, beware of calling something 'complicated' just because you don't understand it, especially if you don't have training and experience in that thing. What's more relevant is whether the people building the systems think it is too complicated.

▲

qingcharles 5 days ago | parent | prev [-]

We're trying to build vehicles that are totally autonomous, though. How do you grab the wheel of the new Waymos without steering wheels? Especially if you're in the back seat staring at Candy Crush.

	▲	esafak 5 days ago \| parent [-]
		Waymos are safer, and drive more defensively than humans. There is no way a Waymo is going to drive aggressively enough to get itself into the trolley problem.

▲

terribleperson 5 days ago | parent | prev | next [-]

This situation isn't going to happen unless the vehicle was traveling at unsafe speeds to begin with.

Cars can stop in quite a short distance. The only way this could happen is if the pedestrian was obscured behind an object until the car was dangerously close. A safe system will recognize potential hiding spots and slow down preemptively - good human drivers do this.

▲

AnIrishDuck 5 days ago | parent [-]

> Cars can stop in quite a short distance.

"Quite a short distance" is doing a lot of lifting. It's been a while since I've been to driver's school, but I remember them making a point of how long it could take to stop, and how your senses could trick you to the contrary. Especially at highway speeds.

I can personally recall a couple (fortunately low stakes) situations where I had to change lanes to avoid an obstacle that I was pretty certain I would hit if I had to stop.

▲

terribleperson 4 days ago | parent [-]

At the driving school I attended, they had us accelerate to 50 mph and then slam on the brakes so we'd have a feel for the distance (and the feel).

While it's true they don't stop instantaneously at highway speeds, cars shouldn't be driving highway speeds when a pedestrian suddenly being in front of you is a realistic risk.

	▲	AnIrishDuck 4 days ago \| parent [-]
		What if the obstacle is not a person? What if something falls off a truck in front of the vehicle? What if wildlife spontaneously decides to cross the road (a common occurrence where I live)? I don't think these problems can just be assumed away.

▲

cameldrv 4 days ago | parent | prev | next [-]

You don't really ever have "two sensors" in the sense that it's two measurements. You have multiple measurements from each sensor every second. Then you accumulate that information over time to get a reliable picture. If the probability of failure on each frame were independent, it would be a relatively simple problem, but of course you're generally going to get a fairly high correlation from one frame to the next about whether or not there's a pedestrian in a certain location. The nice thing about having multiple sensing modalities is that the failure correlation between them is a lot lower.

For example, say you have a pedestrian that's partially obscured by a car or another object, and maybe they're wearing a hat or a mask or wearing a backpack or carrying a kid or something, it may look unusual enough that either the camera or the lidar isn't going to recognize it as a person reliably. However, since the camera is generally looking at color, texture, etc in 2D, and the Lidar is looking at 3D shapes, they'll tend to fail in different situations. If the car thinks there's a substantial probability of a human in the driving path, it's going to swerve or hit the brakes.

▲

consumer451 5 days ago | parent | prev | next [-]

> > If multiple sensor modalities disagree, even without sensor fusion, you can at least assume something might be awry and drop into a maximum safety operation mode.

> This is not always possible. You're on a two lane road. Your vision system tells you there's a pedestrian in your lane. Your LIDAR says the pedestrian is actually in the other lane. There's enough time for a lane change, but not to stop.

> What do you do?

Go into your failure mode. At least you have a check to indicate a possible issue with 2 signals.

	▲	Mentlo 4 days ago \| parent \| next [-]
		I came here to write the same comment you did. What I’d suspect (I don’t work in self driving but I do in AI) is the issue is that this mode of operation would happen more often than not as the sensors disagree in critical ways more often than you’d think. So going “safety first” every time likely critically diminishes UX. The issue is not recognising that optimising for Ux at the expense of safety here is the wrong call, motivated likely by optimism and a desire for autonomous cars, more than reasonable system design. I.e. if the sensors disagree so often that it makes the system unusable, maybe the solution is “we’re not ready for this kind of technology and we should slow down” rather than “let’s figure out non-UX breaking edge case heuristics to maintain the illusion of autonomous driving being behind the corner”. Part of this problem is not even technological - human drivers tradeoff safety for UX all the time - so the expectation for self driving is unrealistic and your system has to have the ethically unacceptable system configuration in order to have any chance of competing. Which is why - in my mind - it’s a fools endeavour in personal car space, but not in public transport space. So go waymo, boo tesla.
	▲	ethbr1 4 days ago \| parent \| prev [-]
		Exactly my point. That you know the systems disagree is a benefit, compared to a single system. People are underweighting the alternative single system hypothetical -- what does a Tesla do when its vision-only system erroneously thinks a pedestrian is one lane over?

▲

ranger_danger 5 days ago | parent | prev | next [-]

This is why good redundant systems have at least 3... in your scenario, without a tie-breaker, all you can do is guess at random which one to trust.

	▲	Someone1234 5 days ago \| parent [-]
		That's a good point, but people do need to keep in mind that many engineered systems with three points of reference have three identical points of reference. That's why it works so well, a common frame of reference (i.e. you can compare via simple voting). For example jet aircraft commonly have three pitot static tubes, and you can just compare/contrast the data to look for the outlier. It works, and it works well. If you tried to do that with e.g. LIDAR, vision, and radar with no common point of reference, solving for trust/resolving disagreements is an incredibly difficult technical challenge. Other variations (e.g. two vision + one LIDAR), does not really make it much easier either. Tie-breaking during sensor fusion is a billion+ dollar problem, and will always be.

▲

abraae 5 days ago | parent | prev [-]

> Never go to sea with two chronometers; take one or three.