While I don't doubt the veracity of your report, I don't think this is an efficient testing strategy.

Ideally, you don't want a very large number of tests, no matter how big the system is. Tests are, effectively, an interface to the program that assesses the system quality / readiness for use. Any interface with thousands of individual pieces is difficult to use.

Random combinations of tests also don't spark joy because this means both repetition (i.e. waste of resources) and testing potentially useless (unreachable or invalid) system states (both wastes resources and creates false alarms).

Ideally, the tests should be able to compose only in desired ways and rather than combining them randomly, there should be some deterministic process that creates a unique sequence or a tree of individual tests on subsequent runs. Ideally, such a test runner could also be configured to start with an existing system in a known state s.t. the tester can apply a patch and resume testing.

▲

pfdietz 2 days ago | parent | next [-]

> While I don't doubt the veracity of your report, I don't think this is an efficient testing strategy.

The marginal cost of running the tests like that was very small, and bugs were found. The cost per found bug seemed very reasonable. Every test failure in this strategy indicates a bug.

(This was the standard compliance test suite for Common Lisp, btw.)

▲

crabbone 2 days ago | parent [-]

Now imagine someone who needs to test something on eg. P6e-GB200 series VM in EC2 (this is one of the newest Nvidia accelerator models: GB200 multiplied by something like 60 iirc).

I don't know what the price is and I'm not even sure you can just create this kind of a VM with any regular account, but I'm sure you can imagine a price of running such a test.

The price doesn't have to be the limiting factor. It may be time (if a test takes a very long time to set up), or it could be labor (imagine needing to flash the ROM of a PCB for each test or similar).

This is what I mean when I say that your experience is likely not transferable to many real-life testing applications. But, even if you don't pay that much for an individual test, still, why waste time and resources, if you could do better?

▲

pfdietz 2 days ago | parent [-]

If these tests are being run just once, the cost of creating them will likely vastly outweigh the cost of running them.

If they are run more than once, they can be reordered randomly each time at zero marginal cost.

	▲	crabbone 10 hours ago \| parent [-]
		This is not at all how it works. But you missed the important distinction: random is bad because it repeats unnecessarily. One of the typically used examples in statistics 101 class to drive attention to this fact is the question posed to students to guess, without actually doing any math: what is the chance of two people in a room with (however many students are there in the class) to have a birthday on the same day? People consistently estimate the chance significantly lower than what it actually is. And this error is supposed to encourage students to dedicate more attention to the subject because they will have found their intuition to be so woefully inadequate. See this for the exact math: https://en.wikipedia.org/wiki/Birthday_problem . * * * Now, if you are interested in economics of writing tests and running them (but you don't have to be, unless it's your job): First of all, for this problem, it doesn't matter how much it costs to write the tests. We are talking about the strategy to cover more feature combinations in the test, so, no matter how much it costs to write a test, the cost (per test) will be the same regardless of the strategy chosen. The combination is where the difference will manifest itself. Second, tests are normally written by salaried employees. This makes the cost of writing an individual test very difficult to divine (and even if you could, it would be a useless metric). It's difficult to calculate because a lot of factors that contribute to writing a test happen in aggregates (eg. the tester needs to explore and familiarize themselves with the system before writing a test, but this effort contributes to multiple tests). Salaried employees are paid a fixed amount regardless of the intensity of work: if they produce one, ten or a hundred tests per month, the salary is the same. Tests can be vastly different in scope and in an effort necessary to implement them, however the effort necessary to implement a test isn't indicative of the value of the test (it's possible that a very simple test will be very effective at detecting problems where a very complex test isn't and the other way around). All this said, suppose you did come up with a way of calculating the cost of writing an individual test: it still doesn't help you to estimate the cost of testing the product, it doesn't even move the needle on this question. It changes nothing about your QA department budget. It changes nothing about delivery schedule. It's just a number you can marvel at in your spare time...

▲

pkhuong 2 days ago | parent | prev [-]

> Any interface with thousands of individual pieces is difficult to use.

Standard libraries considered harmful: 100 interfaces with 10 pieces each are obviously easier to use.

	▲	crabbone 2 days ago \| parent [-]
		If they are organized into meaningful groups, then, of course: the organization makes it easier to use. Compare: have thousands of name-phone number pairings in any order vs same pairings, but now they are grouped by the first letter of the name? Don't you think that the later is easier to use?