Remix.run Logo
0xbadcafebee 2 days ago

If we had a software building code, it could mandate the testing procedures for consumer devices, like a car's headunit firmware. This building code could be backed by an industry body that could revoke its certification from manufacturers if they don't comply with the code. Super-advanced-testing-procedure #1: plug a phone into a test car and check it works before release.

(This software building code is more necessary for software used in critical infrastructure. But it should also be applied to consumer devices as basic protection for consumers against manufacturers breaking functionality the consumer paid for)

taneq 2 days ago | parent [-]

Plug one of every combination of vendor, model, OS, and config into the car and check if everything works. That’s what would be required to actually ensure functionality.

2 days ago | parent | next [-]
[deleted]
0xbadcafebee 2 days ago | parent | prev [-]

Don't let perfection be the enemy of the good. In industrial testing automation, you often don't test every feature completely. You calculate the cost-benefit of the complexity, time and cost it takes to test features, and target the most valuable ones. (I used to write automation test suites for industrial appliances with hundreds of thousands of permutations)

At the very least, you can test one single feature end to end, just to get a baseline that nothing is massively, obviously broken (like it was in this case!). This lack of fundamental testing is what caused the CrowdStrike bug that brought down Delta Airlines. This kind of test is extremely easy, quick, cheap, and simple, and catches the worst bugs. In other words, it's the most valuable test.

Companies will continue to let these bugs happen until we mandate a minimum amount of fundamental testing before release.