Remix.run Logo
▲ evalstate 2 hours ago

Just use a harness that has been properly integrated and tested with each model, with published traces and comparisons (author of) https://fast-agent.ai/benchmarks/

GitHub and Copilot CLI have started to do the same btw (not sure if they publish the traces/reward hack results). But we should normalise this as consumers shouldn't be forced in to playing this game.