| ▲ | visiondude 2 hours ago | |
so the coding tasks illicit refusal by anthropic classifiers and instead of updating tasks to do similar things that don’t trigger classifiers their choice is to count those as fails? feels wrong given their stated task list. | ||
| ▲ | Youden 2 hours ago | parent [-] | |
I think that's reasonable. The goal of the benchmark is to determine how the model performs on real-world tasks. If real-world tasks trigger Anthropic's classifiers, that's a failure on Anthropic's part. I feel that choosing tasks that don't trip the classifier would also be a form of bias towards Anthropic. In case it's not clear, the coding tasks are really benign things; there's nothing security-, health- or biology- related in there. The classifier being tripped is definitely unreasonable. | ||