Yeah, I try to err on the side of not using them unless the accuracy obtained through more robust methods is just a no-go, because there are so many ways they can suddenly and irrecoverably fail if some sensor randomly produces something weird that wasn't accounted for. Which happens all the time in practice.

It is always a good idea to include outliers treatment in KF algorithm to filter out weird measurements.