Sounds like it's not actually well written in that case. Either you're testing the wrong output if it's non-deterministic or you have a consistency bug that's corrupting data in production.

Exactly; a very occasionally flakey test may be tolerable but is almost by definition not well written.

The commonest type I see is one where instead of waiting until expected behaviour is exhibited with a suitable timeout, the test sleeps for some shorter period and then checks to see if the behaviour was exhibited.

These tests not only flake occasionally when the CI server or dev laptop is under unusual load, but worse, accumulate until the test suite is so full of "short" sleeps that the full set of test takes half an hour to run.

Often the sleeps were seen as being acceptable because the plan was to run the tests in parallel, but then the increased load results in the tests becoming flakey.

Once you have dozens of these flaking tests for this or other reasons, it becomes a project in itself to refactor them back to something sane.

Flakey tests should always be fixed immediately unless you're in the middle of an incident or something.