That's just not how they work, really. They don't know what they don't know and their process requires an output.

I think they're getting better at it, but it's likely just the number of parameters getting bigger and bigger in the SOTA models more than anything.

▲

adastra22 36 minutes ago | parent [-]

They do know what they don't know. There's a probability distribution for outputs that they are sampling from. That just isn't being used for that purpose.

▲

raddan 8 minutes ago | parent | next [-]

I’m not clear what you mean by “know.” If you mean “the information is in the model” then I mostly agree, distributional information is represented somewhere. But if you mean that a model can actually access this information in a meaningful and accurate way—say, to state its confidence level—I don’t think that’s true. There is a stochastic process sampling from those distributions, but can the process introspect? That would be a very surprising capability.

▲

Isamu 20 minutes ago | parent | prev [-]

Oh, you mean somewhere it is tracking the statistical likelihood of the output. Yeah I buy that, although I think it just tends towards the most likely output given the context that it is dragging along. I mean it wouldn’t deliberately choose something really statistically unlikely, that’s like a non sequitur.

	▲	tempest_ 9 minutes ago \| parent [-]
		From its point of view what does it mean "to know". Is it the token (or set of tokens) that are strictly > 50% probable or is it just the highest probability in a set of probabilities? While generating bullshit is not ideal for a lot of use cases you don't want your premier chat bot to say "I don't know" to the general public half the time. The investment in these things requires wide adoption so they are always going to favour the "guesses".