| ▲ | wisty 18 hours ago | |
The article notes that this is mainly because the cheap models tend to be distillations of the frontier closed source models, and the developers of the original closed source frontier models have an edge in training their cheaper versions (since they have the source). I'd add that they also have funding, and want to blitzscale. OpenAI / Google can offer a cheap model to get more training data (plan dependent), or have a lot of capacity due to training requirements. It is also evidence of commodicisation / brutal competition. I wonder if they cut costs because there is stiff competition from open weight models (and arguably we are in the blitzscaling phase where they try to corner the market, not the enshitification stage). | ||
| ▲ | rbehrends 17 hours ago | parent [-] | |
While the distillation argument is brought up again and again, I find it very dubious. This is not to say that it doesn't happen, but I have my doubts that it is a limitation. DeepSeek V4.1 Flash, for example, can reason itself through some pretty complex mathematical problems. Definitely not millenium-prize level problems and frontier models are superior for research-level mathematics, but it is still surprisingly close even for some of that [1, 2]. Where it differs from the larger models is that it has less knowledge; respectably as it performs for its size, 552B parameters are still a lot less than the multiple trillions that frontier models have. Sol/Astra/Opus/Fable may know the solution to the problem directly or to a very adjacent problem, while DeepSeek has to work its way through it from first principles and then sometimes fails (because LLM reasoning is really a heuristic-guided search). But this is also a failure mode that I have seen with Luna. [1] As evidenced e.g. by ErdosBench. https://www.ulam.ai/erdosbench [2] Using this as an example over software engineering tasks here, because it tests reasoning capability more than knowledge retrieval. | ||