| ▲ | Aurornis 2 hours ago | |
Two problems with modifying models like these, which you should be aware of. First, the training sets of these models are usually shaped around the refusal, too. They might not have enough of the knowledge to answer correctly even if you stop it from going down the refusal path. If the model was trained on data that gives a refusal to that topic, the real information might not be encoded in the model at all. You’re trying to force it to go down a path that produces an answer, which asking for hallucinations. Second, the quality can drop on unrelated questions. Depending on the question this may or may not happen. I know they post KL divergence charts but those tell you very little for a focused topic like this. So if you expect a model that will start correctly telling you info that its local government didn’t want included, this changes nothing. The best argument for these models is if you are trying to do a general purpose task but the model triggers a refusal based on vague reasons, like not wanting to reverse engineer something. | ||
| ▲ | orangeboats 2 hours ago | parent | next [-] | |
>So if you expect a model that will start correctly telling you info that its local government didn’t want included, this changes nothing. From experience, the models often do have the knowledge of those topics (strictly talking about the political ones). IMO the refusal is likely to be a product of post-training, as evidenced by various people gaming the prompts just enough to get a proper response out of the vanilla models. Probably only when you get to things like illicit drugs or NSFL topics, that things will go haywire with the refusals removed. | ||
| ▲ | radial_symmetry 2 hours ago | parent | prev [-] | |
"They might not have enough of the knowledge to answer correctly" Depends on the model. GPT-OSS is the main standout here, it was trained on a highly curated dataset so information that they didn't want in isn't in the pretraining at all. Most other models know the answer and were just taught refusal in post-training. | ||