Just for reference, in case you find yourself in an optimization under uncertainty situation again: The decision-theoretic right way to do this is generate a bayesian posterior over true ranking given ranking count and a prior on true rankings, add a loss function (it can just be the difference between the true rating of the selected item and the true rating of the non-selected item for simplicity) then choose your option to minimize the expected loss. This produces exactly the correct answer.

Can you please provide an example or link to read more? Seems very interesting.