| ▲ | pfdietz 2 days ago | |||||||||||||||||||||||||||||||||||||||||||||||||
I had a test suite with thousands of tests. One way of running it was to take all the passing tests, and then run them repeatedly in random order. This found new bugs involving unintended persistent state. | ||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | natbennett 2 days ago | parent | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||
After a bunch of experiences like this I will now basically only use test runners that automatically run the tests in randomized order. Gotta catch that stuff early. | ||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | crabbone 2 days ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||
While I don't doubt the veracity of your report, I don't think this is an efficient testing strategy. Ideally, you don't want a very large number of tests, no matter how big the system is. Tests are, effectively, an interface to the program that assesses the system quality / readiness for use. Any interface with thousands of individual pieces is difficult to use. Random combinations of tests also don't spark joy because this means both repetition (i.e. waste of resources) and testing potentially useless (unreachable or invalid) system states (both wastes resources and creates false alarms). Ideally, the tests should be able to compose only in desired ways and rather than combining them randomly, there should be some deterministic process that creates a unique sequence or a tree of individual tests on subsequent runs. Ideally, such a test runner could also be configured to start with an existing system in a known state s.t. the tester can apply a patch and resume testing. | ||||||||||||||||||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | sholladay 2 days ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||
Shout out to AVA for running all tests async by default, which not only finds order-dependent issues, but also race condition and other timing related issues. | ||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | neuronexmachina 2 days ago | parent | prev [-] | |||||||||||||||||||||||||||||||||||||||||||||||||
Reversing the test order is also pretty effective: https://pypi.org/project/pytest-reverse/ | ||||||||||||||||||||||||||||||||||||||||||||||||||