| ▲ | dom96 2 hours ago | |
Depends highly on the task. It does pretty well on my benchmark[1] in terms of cost (and success rate). | ||
| ▲ | rico735 35 minutes ago | parent [-] | |
Out of interest, do you run the same tests more than once to generate that? What I often observe is that models being asked the exact same prompt can alternate between getting something right and getting it wrong, thanks to the wonders of a non-deterministic output. | ||