Maybe confidence is a loaded term, but LLMs do have a sense of procedural uncertainty. Couldn’t you ask it to respond y/n in response to a question and look at the next token sampling probabilities?