Remix.run Logo
▲ swatcoder an hour ago

In this example, it's not an alignment thing.

It's more that its just reguritating the lazy, shallow garbage what would fare well in SEO or would be the most common reply in most online forums before. As digital signals to training and among inexpert RLHF reviewers, it has all the marks of being a good answer to the question as asked.

So of course models happily repeat it, even though it's largely naive and wrong-headed and useless.

It's the kind of question, with vague-but-typical phrasing, that most people don't have good insight into but that many people feel like can speak to (bullshit to) anyway by transposing some trendy generalized answer.

In the real world, you ask 10 people, get 6 of these shitty answers, and then 1 person who actually knows what they're talking about chimes in and you go "oh, yeah, that's what I was looking for, thanks!"

Because of how they work, LLM's, even with RLHF, have a very strong bias towards giving one of those 6 shitty answers when presented with this kind of question.

The seeming fix right now is to train them to be more inquisitive, to prompt users to think about and provide more refining context after that first prompt reply. You see this espcially in Google's AI Search responses already, where they try to present 2-3 refining questions in case the first bullshit answer didn't hit the mark.