Remix.run Logo
▲ dieselgate 4 hours ago

Can you elaborate on what you mean by Eval? All I can think of is eval() in js...

I hope you agree people should use the "right tool for the job" meaning a calculator for arithmetic rather than an LLM for everything.

▲aabdi 3 hours ago | parent | next [-]

Evals are tests of model accuracy. For example here are 100 people's sample houses, go test if the agent can turn off the lightbulbs correctly.

Right job sure. But, If you're replacing the LLM with a jev like as in the article, or switching with a smaller classifier, then you better have good data/evals otherwise you're reducing accuracy without you even knowing.

▲dominotw 3 hours ago | parent | prev [-]

its basically assertEquals(x,y) where function is calling llm to check equals