| ▲ | mrbungie 3 hours ago | |
If you are not getting feedback from those p90 then your bad evals are the least of your problems. Also, unless those p90 customers are bringing a big portion of your income (again, if that happens you'd KNOW that your product is broken for them) I'd not pay for the most expensive LLM to "avoid doing evals". | ||
| ▲ | aabdi 2 hours ago | parent [-] | |
>If you are not getting feedback from those p90 then your bad evals are the least of your problems. the feedback is part of your online evals... if you have bad feedback you have bad evals > Also, unless those p90 customers are bringing a big portion of your income (again, if that happens you'd KNOW that your product is broken for them) I'd not pay for the most expensive LLM to "avoid doing evals". they are but that's besides the point. Two ways: - logically ROI of accuracy degradation as compared to cost has to be measured, sure. - emotionally, morally sub 10% groups in our customer distribution include such things as disabled people, minorities, and the elderly. In my view you should love all your customers. | ||