Remix.run Logo
▲ ritz_labringue 8 hours ago

We've moved past per token cost a long time ago. Smaller models use more tokens. What's interesting is the cost per task, on that front deepseek isn't particularly good.

https://artificialanalysis.ai/#intelligence-efficiency-tabs

▲bscphil 3 hours ago | parent | next [-]

The AA benchmarks seem to be mostly "complete this small intelligence-proving task from scratch". That consistently puts models with high intelligence to output tokens on the frontier, especially the OpenAI models, on the frontier. But a lot of software engineering work is many-turn work on established code bases. There's a lot of reading and a relatively large context unless it's carefully controlled. Writing only a few output tokens doesn't really cut costs because most of the spend is read tokens that get recycled through dozens of turns. That's basically the ideal case for DeepSeek because of how cheap cached tokens are.

▲joshheitzman 8 hours ago | parent | prev | next [-]

It highly depends on the task. So far, on the tasks I personally benchmark, DS v4.1 Flash @ deepinfra.com is always cheaper because their cached input price for it is so very low and their cache hit rate is so very high.

▲gregwebs 5 hours ago | parent | prev [-]

That analysis is unfortunately running DeepSeek on Max. DeepSeek recommends against that since it guarantees higher cost and rarely improves the outcome- it should be ran on "high".

▲bscphil 3 hours ago | parent | next [-]

They have "high" on there now too, just search for it.

▲jambutters 3 hours ago | parent | prev [-]

where does deepseek say they recommend against high?