One solution is to gradually move instances to you most likely solution.

But continue a percentage of A/B/n testing as well.

This allows for a balancing of speed vs. certainty

do you use any tool for this, or simply crunk up slightly the dial each day

	▲	hruk an hour ago \| parent \| next [-]
		We've used this Python package to do this: https://github.com/bayesianbandits/bayesianbandits
	▲	travisjungroth 15 hours ago \| parent \| prev [-]
		There are multi armed bandit algorithms for this. I don’t know the names of the public tools. This is especially useful for something where the value of the choice is front loaded, like headlines.