It does, if you assume you care about the validity of the results or about making changes that improve your outcomes.

The degree of care can be different in less critical contexts, but then you shouldn’t lie to yourself about how much you care.

▲

renjimen 18 hours ago | parent [-]

But there’s an opportunity cost that needs to be factored in when waiting for a stronger signal.

▲

Nevermark 18 hours ago | parent | next [-]

One solution is to gradually move instances to you most likely solution.

But continue a percentage of A/B/n testing as well.

This allows for a balancing of speed vs. certainty

	▲	imachine1980_ 17 hours ago \| parent [-]
		do you use any tool for this, or simply crunk up slightly the dial each day

▲

scott_w 12 hours ago | parent | prev | next [-]

There is but you can decide that up front. There’s tools that will show you how long it’ll take to get statistical significance. You can then decide if you want to wait that long or have a softer p-value.

▲

epgui 16 hours ago | parent | prev [-]

Even if you have to be honest with yourself about how much you care about being right, there’s still a place for balancing priorities. Two things can be true at once.

Sometimes someone just has to make imperfect decisions based on incomplete information, or make arbitrary judgment calls. And that’s totally fine… But it shouldn’t be confused with data-driven decisions.

The two kinds of decisions need to happen. They can both happen honestly.