Remix.run Logo
▲ abhgh 4 hours ago

Maybe I should write up a post about this, but about evals ... I have feelings. Yes you're right that writing evals is hard. And it comes down to the simple fact that the thing that you're testing - a LLM-driven feature that presumably suffers from variance and inadequate accuracy - and your measuring device - an LLM judge - suffer from the fundamental problems. But the lure of natural language is such that instead of seeing this pipeline of LLM-driven things as a path of cascading errors, it is easy to view it as one set of problems correcting for another set.

In this world, the LLM-driven feature is a probabilistic entity, and the judge layered on top of that is another probabilistic entity. The real work is to look at the judge's response and infer the performance of the feature. Good evals make the inference step "pass-through", i.e., the judge outcomes have a 1:1 mapping with feature outcomes. But this kind of reliability often doesn't naturally appear. I think of reliability as being composed of three attributes: (a) consistency: do we get the same result for an eval run on the same llm with the same input (b) inter-llm agreement: do outcomes agree over multiple trials of running an eval over multiple llms with the same input (c) accuracy: does the eval agree with human judgement. You need extra work to guarantee these properties. And even when you can, you won't get 100% reliability, but say high 90s, and you still to make a judgement call as to call a feature "functional".

▲aabdi 3 hours ago | parent [-]

Two things, my problem isn't with the probabilistic nature, its:

1. with the quality of the data that customers use to eval.

2. with the lack of rigor/ops/testing around accuracy for deployment and regression.

I'll give you examples for failures IRL:

Offline:

- sample distribution for test data didn't have p99 customers in it, failed in production when it launched for those customers.

- test data sets were all synthetic, launch to real customers fails because data set was not representative of real.

Online:

- yeah we used LLM as a judge in prod/online eval, but the judge sampling failed to measured against our japanese customers, our bilingual customers, customers with media devices etc.

- the LLM as a judge was used, but there was no measure of human accept rate measures/alarms for various use cases (lightbulbs turning off, music playing, etc) so the lightbulbs started failing.

- there was no consistent human sampling of online data, so for example samsung TVs stopped working, but no one caught it until 3 months later when some VP was watching his TV and it broke and was accelerated despite it affecting 2-3% of customers.