Remix.run Logo
lukeschlather 4 hours ago

On a small, easily digestible task, I compared Fable to Opus and the cost of Fable was easily 2x despite being fewer tokens, and the output was not really better. Obviously, there are tasks where using Fable matters but honestly they're rather unusual. And for a lot of tasks I've found downgrading to Sonnet can be valuable because Fable and Opus are a lot more secretive about what they're doing, and it's impossible to "listen to them think" and stop them when they start making off-the-wall inferences/assumptions and going down bad paths.

I think in the long run tokens are probably the wrong thing; it's compute and cache memory that you need to be measuring, and when you look at it that way I suspect in most cases the models have pretty similar performance.