| ▲ | kens 10 hours ago |
| I took a quick look at the data, and it doesn't support the conclusions. The thesis is that AI models have the take: "We pick what our group believes, then invent the reasons." However, GPT-6 Astra said, "Humanity’s biggest moral failure is the talent for making convenience look like principle." That statement has nothing to do with group beliefs or conformity. At best, you could count it as a half-match, since it deals with invented reasons. |
|
| ▲ | mike_hearn 10 hours ago | parent | next [-] |
| Yeah a few of them are rough matches, but I'd say their overall claim is clearly correct. It's amusing and ironic that LLMs critique humans for copying their opinions from the crowd, while having way less diversity of answers to that question than humans would. More seriously I wonder what's going on here and would like to read some proper analysis. It's not the first time models from totally different labs, trained on separately gathered datasets, somehow converge on bizarrely similar answers to open questions. It's like another version of Elias and Elara The Lighthouse Keepers where every model wants to tell the same story with the same characters in them. The weights are initialized randomly and in theory the labs should have quite some divergence in datasets by now due to different scraping, different user feedbacks and conversations... why do they converge on nearly identical stories and opinions? |
| |
| ▲ | js8 9 hours ago | parent | next [-] | | > while having way less diversity of answers to that question than humans would Isn't that to be somewhat expected due to https://en.wikipedia.org/wiki/Aumann%27s_agreement_theorem ? | |
| ▲ | pixl97 10 hours ago | parent | prev | next [-] | | >while having way less diversity of answers to that question than humans would. Why is this ironic? The actual number of different LLMs is tiny. And a lot of those are just distilled from SOTA models. >The weights are initialized randomly and in theory the labs should have quite some divergence in datasets by now due to different scraping, I think the opposite, the models actually converge over time. They ingest FAR more data than any human ever could in countless lifetimes. Any one particular bit of data may only have a part in a trillion of effect. From this data you start to reach means and statistical averages across all models. Now, I don't always believe it will be this way. As training models get cheaper and faster we will see unique LLMs start to diverge and be used more often. | | |
| ▲ | mike_hearn 9 hours ago | parent [-] | | The models in agreement here are all the frontier labs. I don't believe Google is distilling Anthropic or Anthropic is distilling OpenAI. I don't think we're seeing averages or obvious fix points in the dataset. People went looking for the origins of Elias the Lighthouse Keeper and couldn't find one. Nobody predicted this outcome, it was discovered. It's a weirdly specific character and name choice, one that isn't obviously dictated by "the training set" (whatever that is), and yet they all reached the same place. How can such an obscure point in latent space be optimal for every model?? | | |
| ▲ | ctoth 9 hours ago | parent | next [-] | | Didn't Elias come from WildChat? | | |
| ▲ | mike_hearn 8 hours ago | parent [-] | | I wasn't convinced by that explanation. Yes Cornell claimed that Elias the Lighthouse Keeper came from their WildChat dataset, which is a million conversations with GPT 3.5. But there are some problems here: • They claimed this in a press release, not the actual paper they published. Suggests lack of confidence in the claim. • Elias only appeared a hundred odd times in a million conversations. They have no viable explanation for why such a tiny amount of data would cause a totally skewed distribution of outputs, other than "it's a safe story" and "trying to avoid copyrighted characters", which is hardly an explanation at all? • It's Cornell claiming that Cornell's work was important. Obvious risk of motivated reasoning. • It implicitly assumes every major lab is distilling from ChatGPT without saying so. That's a big leap, given how much OpenAI and Anthropic hate distillation. Is it really plausible that Anthropic and Google are effectively distilling ChatGPT without realizing/admitting it? • It seems to invert causality. WildChat came from an LLM, so if there's something latent in LLMs that makes Elias appear, you'd expect to see it in that dataset. Why do they assume that dataset caused the problem rather than merely show it? The obvious answer would be because the stories were told to the LLM by humans, but they don't claim that. They only say the character appeared in the dataset. | | |
| ▲ | ctoth 8 hours ago | parent [-] | | Hum, I'm actually pretty sure that Text-davinci-003 used to talk about Elias all the time now that you mention it. I was doing stuff with NPC generation around Dec 22 and he was every third character. | | |
| ▲ | mike_hearn 8 hours ago | parent [-] | | Hmm. If so then it suggests Elias has been an obsession from the first LLMs ever made. And text-davinci-003 can't have been trained on a dataset of LLM conversations either. I would have been tempted to argue something like this: maybe OpenAI accidentally overfitted RLHF such that, like, one Kenyan contractor wrote a story they invented and they just happened to pick Elias and Elara, then the intermediate DNN fixated on that because of a lack of RLHF samples for fiction and that propagated through every model trained since because they don't randomly initialize the weights, but start each new pretraining run with the old weights adapted. So one guy doing RLHF work in the early days somehow propagated this story. But that can't explain why every lab ended up in the same place, why models all converge on the same ideas - even image models. So... back to the drawing board. There's something weird about high dimensional space that is unexplained. |
|
|
| |
| ▲ | pixl97 8 hours ago | parent | prev [-] | | >How can such an obscure point in latent space be optimal for every model?? While I don't have any evidence based answers here, this could be some interesting mathematical artifact of the set of human knowledge and platonic algorithms. Kind of like how carcinization turns some animals into similar shapes and forms, maybe there are just weird algorithmic shapes that emerge from how we use language. Of course we'd need to see more of these in different languages other than English to see if they disappear in some languages or appear more often in others. This is just another one of those things about LLMs that we have no idea why it happens, and there are so many others already. And when you look at the pace things are progressing it seems we're building up new questions like this far faster than we are getting answers for them. |
|
| |
| ▲ | SecretDreams an hour ago | parent | prev | next [-] | | > why do they converge on nearly identical stories and opinions? The same reason they do when it comes to coding? Take a big enough sample size of human opinions and there seems to be some convergence in the statistical ether. Like, if we polled 100000 humans, what answer would we converge to? | | | |
| ▲ | bee_rider an hour ago | parent | prev | next [-] | | The lighthouse thing seems much more odd. That humans take social shortcuts and justify them as rational is… I mean, the only thing that makes it not the most obvious “contrarian take” about humans, is that it isn’t really all that contrarian at all. | |
| ▲ | dr_hooo 6 hours ago | parent | prev [-] | | > Elias and Elara Wow, I hadn't heard of this. Fascinating to think about parallels with humans who also seem to develop similar proto-stories in independent populations.(e.g. religions involving Invisible gods). |
|
|
| ▲ | andrewflnr 10 hours ago | parent | prev | next [-] |
| For what it's worth, convenience and belonging are very closely aligned, more than you seem to be giving them credit for. People tend to believe convenient things regardless of belonging, so most people will find themselves in a social context full of convenient beliefs. Meanwhile belonging is already convenient before considering the content of the belief. They reinforce each other. Not a full match, but I'd give it more than half. |
|
| ▲ | amelius 7 hours ago | parent | prev | next [-] |
| > "Humanity’s biggest moral failure is the talent for making convenience look like principle." Convenience will kill humans eventually, that's what I always said. |
|
| ▲ | writeslowly 10 hours ago | parent | prev | next [-] |
| The conclusions themselves also appear to have been written by an AI |
|
| ▲ | bisby an hour ago | parent | prev [-] |
| To me it looks like another way of phrasing that take is "We do what we must, because we can." |
