If you retry until it gives the answer you want then it only serves to amplify your bias. If you retry and see how often it agrees with itself then it serves to show there is no confidence in an answer all around.

It's a bit of a crutch for LLMs lacking the ability to just say "I'm not sure" because doing so is against how they are rewarded in training.

You’re still likely to just amplify your own bias if you don’t do some basic experimental controls like having some preselected criteria on how many retries you’re going to do or how many agreeing trials are statistically significant.