I know enough about Machine Learning and statistics to understand that errors are always there. It just needs to be small enough to not matter in the decisions that need to be taken (hopefully). But the thing is that computers can't differentiate errors from correct behavior. Anything in the code is true and if the result is catastrophic, so be it.

As software engineers, it's very often easy to specify what the system should do. But ensuring that it doesn't do what he shouldn't do is the tiresome part of the job. And most tools we created is to ensure the latter.