Remix.run Logo
jstanley 4 hours ago

Only if it becomes obvious why it is flaky. If it's just sometimes broken but really hard to reproduce then it just gets piled on to the background level of flakiness and never gets fixed.

nomel 3 hours ago | parent [-]

To get around this, I have it log the relevant inputs, so it can be reproduced.

The whole concept of allowing a flaky unit test to exist is wild and dangerous to me. It makes a culture of ignoring real failures in what, should be, deterministic code.

marcosdumay an hour ago | parent [-]

Well, if people can't reproduce the failures, people won't fix them.

So, yes, logging the inputs is extremely important. So is minimizing any IO dependency in your tests.

But then that runs against another important rule, that integration tests should test the entire system, IO included. So, your error handling must always log very clearly the cause of any IO error it finds.