Remix.run Logo
pfdietz 2 days ago

If these tests are being run just once, the cost of creating them will likely vastly outweigh the cost of running them.

If they are run more than once, they can be reordered randomly each time at zero marginal cost.

crabbone 10 hours ago | parent [-]

This is not at all how it works.

But you missed the important distinction: random is bad because it repeats unnecessarily. One of the typically used examples in statistics 101 class to drive attention to this fact is the question posed to students to guess, without actually doing any math: what is the chance of two people in a room with (however many students are there in the class) to have a birthday on the same day?

People consistently estimate the chance significantly lower than what it actually is. And this error is supposed to encourage students to dedicate more attention to the subject because they will have found their intuition to be so woefully inadequate.

See this for the exact math: https://en.wikipedia.org/wiki/Birthday_problem .

* * *

Now, if you are interested in economics of writing tests and running them (but you don't have to be, unless it's your job):

First of all, for this problem, it doesn't matter how much it costs to write the tests. We are talking about the strategy to cover more feature combinations in the test, so, no matter how much it costs to write a test, the cost (per test) will be the same regardless of the strategy chosen. The combination is where the difference will manifest itself.

Second, tests are normally written by salaried employees. This makes the cost of writing an individual test very difficult to divine (and even if you could, it would be a useless metric). It's difficult to calculate because a lot of factors that contribute to writing a test happen in aggregates (eg. the tester needs to explore and familiarize themselves with the system before writing a test, but this effort contributes to multiple tests). Salaried employees are paid a fixed amount regardless of the intensity of work: if they produce one, ten or a hundred tests per month, the salary is the same. Tests can be vastly different in scope and in an effort necessary to implement them, however the effort necessary to implement a test isn't indicative of the value of the test (it's possible that a very simple test will be very effective at detecting problems where a very complex test isn't and the other way around).

All this said, suppose you did come up with a way of calculating the cost of writing an individual test: it still doesn't help you to estimate the cost of testing the product, it doesn't even move the needle on this question. It changes nothing about your QA department budget. It changes nothing about delivery schedule. It's just a number you can marvel at in your spare time...