Remix.run Logo
▲ dom96 2 hours ago

Depends highly on the task. It does pretty well on my benchmark[1] in terms of cost (and success rate).

1 - https://bench.killswitch-lang.org/

▲rico735 35 minutes ago | parent [-]

Out of interest, do you run the same tests more than once to generate that?

What I often observe is that models being asked the exact same prompt can alternate between getting something right and getting it wrong, thanks to the wonders of a non-deterministic output.