We needed a framework to write our evals in and easily update them, run A/B tests, set all kinds of constraints (ex: timeouts, number of turns), and configure the execution environment (ex: sandboxes, dependencies, how to handle AskQuestion). We also have an agent judge.

It’s all in a YAML file now.

We seem to be moving toward a world where companies rely more and more on evals to decide what to ship, so this should be super helpful for normalizing evals across teams.

finally a framework that allows me to monitor my coding agents!

Within 10 secs of reading this I was convinced I needed this in our pipeline.

For me the next step is plugging this into the company CI/CD to gate SKILL updates and catch tool call regressions before merging