do you use any tool for this, or simply crunk up slightly the dial each day

We've used this Python package to do this: https://github.com/bayesianbandits/bayesianbandits

There are multi armed bandit algorithms for this. I don’t know the names of the public tools.

This is especially useful for something where the value of the choice is front loaded, like headlines.