> When LLMs say something true, it’s a coincidence of the training data that the statement of fact is also a likely sequence of words;

Do you know what a "coincidence" actually is? The definition you're using is wrong.

It's not a coincidence that I train a model on healthcare regulations and it answers a question about healthcare regulations correctly.

None of that is coincidental.

If I trained it on healthcare regulations and asked it about recipes, it won't get anything right. How is that coincidental?

▲

jstrieb 11 hours ago | parent | next [-]

LLMs are trained on text, only some of which includes facts. It's a coincidence when the output includes new facts not explicitly present in the training data.

▲

anthonylevine 11 hours ago | parent [-]

> It's a coincidence when the output includes facts,

That's not what a coincidence is.

A coincidence is: "a remarkable concurrence of events or circumstances without apparent causal connection."

Are you saying that training it on a subset of specific data and it responding with that data "does not have a causal connection"> Do you know how statistical pattern matching works?

▲

Dilettante_ 10 hours ago | parent [-]

Can I offer a different phrasing?

It's not coincidence that the answer contains the facts you want. That is a direct consequence of the question you asked and the training corpus.

But the answer containing facts/Truth is incidental from the LLMs point of view, in that the machine really does not care, nor even have any concept of whether it gave you the facts you asked for or just nice-sounding gibberish. The machine only wants to generate tokens, everything else is incidental. (To the core mechanism, that is. OpenAI and co obviously care a lot about quality and content of the output)

	▲	anthonylevine 10 hours ago \| parent [-]
		Totally agree with that. But the problem is the phrase "coincidence" makes it into something it absolutely isn't. And it's used to try and detract from what these tools can actually do. They are useful. It's not a coin flip as to whether Bolt will produce a new design of a medical intake form for me if I ask it to. It does. It doesn't randomly give me a design for a social media app, for instance.

▲

delusional 11 hours ago | parent | prev [-]

> It's not a coincidence that I train a model on healthcare regulations and it answers a question about healthcare regulations

If you train a model on only healthcare regulations it wont answer questions about healthcare regulation, it will produce text that looks like healthcare regulations.

▲

mexicocitinluez 11 hours ago | parent [-]

And that's not a coincidence. That's not what the word "coincidence" means. It's a complete misunderstanding of how these tools works.

▲

delusional 11 hours ago | parent [-]

I don't think you're the right person to make any claim of "complete misunderstanding" when you claim that training an LLM on regulations would produce a system capable of answering questions about that regulation.

	▲	anthonylevine 11 hours ago \| parent [-]
		> you claim that training an LLM on regulations would produce a system capable of answering questions about that regulation. Huh? But it does do that? What do you think training an LLM entails? Are you of the belief that an LLM trained on non-medical data would have the same statical chance of answering a medical question correctly? we're at the "Redefining what words means in order to not have to admit I was wrong" stage of this argument