Technically speaking when you send the prefix “The capital of France is “ into an LLM it will also produce probabilities across its whole vocabulary.

The probability values don’t really represent confidence in modern LLMs though, especially after RLHF and RLVR.

System One says they use RLCD, Reinforcement Learning for Calibrated Decisions, which presumably has accurate probabilities as an explicit optimisation goal.

… which they could provide in their APIs but are vehemently opposed to because it makes distillation much easier, and faster.