Didn't Elias come from WildChat?

I wasn't convinced by that explanation. Yes Cornell claimed that Elias the Lighthouse Keeper came from their WildChat dataset, which is a million conversations with GPT 3.5. But there are some problems here:

• They claimed this in a press release, not the actual paper they published. Suggests lack of confidence in the claim.

• Elias only appeared a hundred odd times in a million conversations. They have no viable explanation for why such a tiny amount of data would cause a totally skewed distribution of outputs, other than "it's a safe story" and "trying to avoid copyrighted characters", which is hardly an explanation at all?

• It's Cornell claiming that Cornell's work was important. Obvious risk of motivated reasoning.

• It implicitly assumes every major lab is distilling from ChatGPT without saying so. That's a big leap, given how much OpenAI and Anthropic hate distillation. Is it really plausible that Anthropic and Google are effectively distilling ChatGPT without realizing/admitting it?

• It seems to invert causality. WildChat came from an LLM, so if there's something latent in LLMs that makes Elias appear, you'd expect to see it in that dataset. Why do they assume that dataset caused the problem rather than merely show it? The obvious answer would be because the stories were told to the LLM by humans, but they don't claim that. They only say the character appeared in the dataset.

▲

ctoth 8 hours ago | parent [-]

Hum, I'm actually pretty sure that Text-davinci-003 used to talk about Elias all the time now that you mention it. I was doing stuff with NPC generation around Dec 22 and he was every third character.

	▲	mike_hearn 8 hours ago \| parent [-]
		Hmm. If so then it suggests Elias has been an obsession from the first LLMs ever made. And text-davinci-003 can't have been trained on a dataset of LLM conversations either. I would have been tempted to argue something like this: maybe OpenAI accidentally overfitted RLHF such that, like, one Kenyan contractor wrote a story they invented and they just happened to pick Elias and Elara, then the intermediate DNN fixated on that because of a lack of RLHF samples for fiction and that propagated through every model trained since because they don't randomly initialize the weights, but start each new pretraining run with the old weights adapted. So one guy doing RLHF work in the early days somehow propagated this story. But that can't explain why every lab ended up in the same place, why models all converge on the same ideas - even image models. So... back to the drawing board. There's something weird about high dimensional space that is unexplained.