One issue with this that we ran into is that it costs actual countable money to run the test suite, which is distinct from anything else I’m used to, so the notion that we’d do enough testing to generate a statistically significant gauge of performance - man, I know it’s correct, but I’m not sure my company will survive the process.

▲

dvogel 3 hours ago | parent [-]

For these tests, why not tune the temperature and such to reduce the randomness and convert them to almost-always-succeeds vs almost-always-fails? Is it not the iteration count that drives up the cost?

▲

sarchertech 2 hours ago | parent [-]

If turn the temperature down for tests, they won’t match production behaviors. If you turn it down too far in production, the output will just be bad.

▲

dvogel 2 hours ago | parent [-]

Isn't the goal of the author to get reproducible behavior out of the agent though? I would thinking turning the temperature down would serve that production goal too.

	▲	roughly 33 minutes ago \| parent [-]
		right, this is the whole problem - the stochastic behavior is both the goal and the problem. If you want your tests to match production, you need to get a reasonable sample size, which costs real money.