Remix.run Logo
▲ windyVector an hour ago

This is first time i've really felt like commenting here, but super interesting article.

Having tested both restricted and unrestricted models, certain queries which I know to have a solution can feel 'gated' by the restricted models. Rather than simply saying "I can't answer that", they choose to answer a similar, but slightly different question, very much along the lines of what the article discusses. To me, this is a fault of RL avoiding these "I can't do that" type responses while still remaining aligned.

When you do push them on the solution, the models will often start to get more defensive, but reveal their awareness of it (along the lines of "Yes I know, but I can't tell you/do it"). It is very interesting to imagine the amount of solutions a model might have to a given query, that end up (intentionally or unintentionally) gated because of this process.

I'd be interested to see if this behaviour of "almost-answering" the question is just emergent from these alignment policies.

try an unrestricted model afterwards, id also be interested to see how that changes creativity.

▲swatcoder an hour ago | parent [-]

In this example, it's not an alignment thing.

It's more that its just reguritating the lazy, shallow garbage what would fare well in SEO or would be the most common reply in most online forums before. As digital signals to training and among inexpert RLHF reviewers, it has all the marks of being a good answer to the question as asked.

So of course models happily repeat it, even though it's largely naive and wrong-headed and useless.

It's the kind of question, with vague-but-typical phrasing, that most people don't have good insight into but that many people feel like can speak to (bullshit to) anyway by transposing some trendy generalized answer.

In the real world, you ask 10 people, get 6 of these shitty answers, and then 1 person who actually knows what they're talking about chimes in and you go "oh, yeah, that's what I was looking for, thanks!"

Because of how they work, LLM's, even with RLHF, have a very strong bias towards giving one of those 6 shitty answers when presented with this kind of question.

The seeming fix right now is to train them to be more inquisitive, to prompt users to think about and provide more refining context after that first prompt reply. You see this espcially in Google's AI Search responses already, where they try to present 2-3 refining questions in case the first bullshit answer didn't hit the mark.