There are multi armed bandit algorithms for this. I don’t know the names of the public tools.
This is especially useful for something where the value of the choice is front loaded, like headlines.