People who claim that the Chinese open weight models have some type of manifest advantage don't realize that the close weight models have a huge advantage as well: the researchers from OpenAI, Anthropic, Google, xAI, Meta are not dumb, they can read the white papers written by DeepSeek, Moonshot, etc, and they can inspect all those architectures and they can pick and choose the best tricks there are out there, and of course, they have access to their own in-house secret sauces.

Sure, any model that is not at the frontier can use the frontier model to generate synthetic high quality training data, so this can reduce significantly the training costs.

But at the scale of OpenAI, Anthropic and Google, it is quite likely that the (raw) training cost is very high anymore. Here's a few heuristics:

1. All the hyperscalers see a huge demand for inference. They can't deploy datacenters quickly enough to satiate all the demand they see. But, it's is impossible for the inference demand to be constant throughout a day or a week. If you use the times when the demand is lower than the peak demand (which is almost all the time) to dedicate the spare compute capacity to training, then your the cost of training compute is zero.

2. It is likely that increasingly a higher cost of the "training" is actually setting the guardrails, which is essentially post-training. As we've seen, without proper guardrails, the US Government won't allow you to serve inference. Anthropic was hit directly, but OpenAI delayed their 5.6 release as well to make sure the US Government is ok. This part of the training cost can't be reduced easily by using synthetic data generated by other models.

3. The frontier labs are also investing more and more in building an ecosystem around their models.

I am not a frontier lab insider, but take a look at the jobs posted on the Anthropic career page [1]. There are 74 jobs in "AI Research and Engineering" and by my count at most 15-20 are related to pure model training (of pre-training or RL type), and the rest are post-training, safety and security, alignment, interpretability, productivity and lots and lots of other things.

[1] https://www.anthropic.com/careers/jobs

▲

OrangeDelonge 12 hours ago | parent | next [-]

People who claim that Postgres has some type of manifest advantage don't realize that Oracle has a huge advantage as well…etc

▲

killingtime74 10 hours ago | parent | prev | next [-]

If the Google and meta engineers are not dumb how come they consistently trail behind the frontier labs and even the Chinese labs with a fraction of the funding.

Probably bad leadership

▲

hnfong 9 hours ago | parent | next [-]

I always suspect they have the most to lose if legal decisions on copyright issues don't go their way.

Imagine a scenario (theoretically possible but increasingly unlikely) where a US court decides that using "pirated" copyright data to train models is illegal. Now the AI developer has invested hundreds of billions of capital into a thing that is declared illegal and has to be scrapped.

This risk affects existing megacorps more than "startups" like OpenAI and Anthropic (and Chinese companies), because the megacorps have much more to lose. They actually have the cash to pay damages if the flood of copyright claims arrive at the door. This will not only bomb their AI development, but also the rest of their established businesses as well.

And thus I strongly suspect legal issues are holding them back a bit. Megacorps want to win the AI race, but not to the extent they stake the rest of their established business, while the newer companies' only product is AI, so they have to go all in.

Notice for example how Meta's Llama performed much more poorly after they got smacked by a bunch of lawsuits claiming that they torrented a bunch of copyright data.

(Disclaimer: I'm an outsider and everything I base my speculations on is public knowledge.)

▲

BoredomIsFun 7 hours ago | parent | prev | next [-]

they, rightfully so, have no faith in LLMs.

▲

kubb 10 hours ago | parent | prev [-]

That plus they don’t distill so they have worse RL examples.

▲

shunia_huang 8 hours ago | parent [-]

But they both spent tons of money on data collecting/labeling/generation, how is it bad compared to distillation? I thought their data are much better if they spent that much, and it seems they are stupid because with that much of resources putting in there with merely no output compared to the frontier models.

	▲	kubb 8 hours ago \| parent [-]
		Creating a RL example by hand is hundreds of times more expensive than generating one using an LLM. Of course the Chinese companies have incredibly talented researchers, and smaller, better organized org structures which account for the rest of the difference.

▲

protocolture 11 hours ago | parent | prev | next [-]

>they have access to their own in-house secret sauces.

I remember some feature lauded by Gemini was reverse engineered by the open weights guys in < 30 days.

If they dont publish some technical information its hard to protect in the US, but conversely, once it is published smart people from outside the copyrightosphere can start working to reverse engineer it.

>3. The frontier labs are also investing more and more in building an ecosystem around their models.

Theres nothing there that isnt immediately replaceable.

	▲	credit_guy 2 hours ago \| parent [-]
		> There's nothing there that isn't immediately replaceable. Indeed. But that was not my point. My point is that we still have this old impression that training cost is dominated by compute and it is hugely expensive, and the Chinese labs can short circuit that by distilling the American frontier models. I don't think the training compute cost is a big factor anymore for the American frontier models, because of the reasons I gave. If the Chinese models can get the training compute cost down by a factor of 100, that's not going to make them 100 times cheaper, and not even cheaper by a factor of 2. Maybe 10% cheaper or so.

▲

eeiei 12 hours ago | parent | prev [-]

It’s giving desperate!