At a certain point you do need to test in real road conditions. However, there is absolutely no need to jump straight from testing in lab conditions and “testing” using unmonitored, untrained end users.

You use professional trained operators with knowledge of the system design and operation using a designed safety plan to minimize prototype risks. At no point should your test plan increase danger to members of the public. Only when you fix problems faster than that test procedure can find do you expand scope.

If you follow the standard automotive pattern, you then expand scope to your untrained, but informed employees using monitored systems. Then untrained employees, informed employees using production systems. Then informed early release customers. Then once you stop being able to find problems regularly at all of those levels do you do a careful monitored release to the general public verifying the safety properties are maintained. Then you finally have a fully released “safe” product.