A paper that is somewhat related to the question of how well this works is "Do Large Language Models Know What They Don’t Know? Evaluating Epistemic Calibration via Prediction Markets"

There a model gets a prediction market scenario (that in reality has already closed, but not from the model's POV), and it is tasked to predict the outcome AND give its confidence in the prediction.

Conclusion turns out to be "systematic overconfidence across all models". Probably worth keeping an eye on such research, might enable you to make the product better over time as new research comes out etc.