Remix.run Logo
▲ podoman 2 hours ago

> Mo is a custom agent harness for catching bugs that builds on the past 3 years of Momentic and millions of test traces from complex, real-world applications. Mo is built for recall: it finds the most real bugs, though Codex is a bit more precise at 86% vs 84%. The tradeoff is cost: Mo costs $2.67 per bug vs $0.80 for Codex, less than Claude Code and Devin.

is this good? so Mo costs more money per bug? I'm confused.

▲momentichenry an hour ago | parent | next [-]

Hey! I'm Henry, one of the engineers working on Mo. You're looking at the right numbers, but the takeaway is slightly different from ours.

The main thing we're trying to show is that Mo finds over 3x as many correct bugs in the same amount of time as the next-best QA agent. This is the main thing we optimize for since the worst thing you can do in QA is miss a real bug and ship broken code to prod. Currently, the tradeoff you make for that better recall is higher cost. I would say the precision is more or less comparable, but marginally lower.

Obviously we'd love to be beating every other agent on all of these numbers, and we're actively working on it, but for now this is an honest and transparent look at how the tool is doing.

Feel free to take a look at the full writeup for more detail!

▲mlcruz an hour ago | parent [-]

I feel like your writeup could be a little stronger if you made it clear how important is hitting those recall targets.

I also do believe search quality is recall dominated, but the link between the metrics and results are not as clear as they could have been. I think you have a stronger sell here than it looks at first glance.

Congratulations to the team tho! Those are very nice results

▲momentichenry 43 minutes ago | parent [-]

Thanks! That's fair feedback. We'll update the writeup to make this clearer. We're also hoping to release a v2 benchmark soon with more fixtures and hopefully an even more optimized Mo.

▲Smoofer 2 hours ago | parent | prev | next [-]

Right? Is the takeaway from that not just that Codex is better for cheaper?

▲ 2 hours ago | parent | prev | next [-]
[deleted]
▲luckydata 2 hours ago | parent | prev [-]

it's right there in the writeup, it costs more but finds more bugs.