> The true probability of face 1 is 1/6, but Jev always chose face 1 and the probability it returned was about 83%

I think this is a misunderstanding of what that 83% probability means. You think you're asking Jev to assign a probability to a dice roll, and if so then obviously it should pick each answer 1/6 of the time.

However, what Jev is doing is assigning a probability to the answer it should choose. And it's choosing 1 with 83% probability, as the correct answer to give, when there's no additional knowledge available.

1 is a perfectly valid answer here. In the absence of other data there's no more (nor less) valid answer than that. and I assume that just like LLMs, the token prediction must assign higher probability to one outcome.

If you keep feeding the previous answers back in (if you can do that with Jev) I assume it would start giving other numbers because it's no longer operating from zero info. Maybe that would even reach ~1/6 after enough rounds, although maybe not.

TypeSafe defines the probabilities Jev returns as "calibrated probabilities". "Probability" here means the probability of the answer being correct. If the probability is 10%, the choice should be correct about one time in ten. So, when Jev returns 83% probability it should be correct about 83 times out of 100, but the choices were only correct about 19 times out of 100, and the true probability is 1/6.

"Higher probability should correspond to a greater chance that the answer is correct."

https://docs.typesafe.ai/introduction/machine-learning-prime...

As someone who hasn’t used Jev, this seems reasonable in the sense that “probability that I’ve given the best answer” seems like a useful thing to provide.

Anyway, if it is “probability that I gave the best answer” why 83%? I mean, the question is a sort of a textbook-style probability question in the sense that everything is well defined. Shouldn’t it be basically 100%?