> That is the danger of open source models. They destroy monopolies and shrink margins.

It's unclear that the leading open-source models could exist without having distilled the frontier models. If that's true, the "danger" is that these upstream models don't get built any more (because of the tremendous cost), and we end up just optimizing what we have. Also unclear whether this would be a net win or loss for humanity.

▲

Schlagbohrer 2 hours ago | parent | next [-]

The tech is at the point where synthetic data is almost high enough quality to train further models on, AND there are so many human users of cloud AI, that their usage can be gathered and trained upon (it's human generated and used in real world situations, so, it is of a good enough quality). The cat is fully out of the bag.

▲

petesergeant 2 hours ago | parent [-]

This deals with half of one of the three scaling factors only. Presumably data production (choosing your training data, etc), model size, and training time/power still contribute huge costs. Who's investing that cash if open-source models will distill out enough to compete without the investment costs?

	▲	sandworm101 2 hours ago \| parent [-]
		There is a community of people who compile LoRAs of their favorite subjects/styles in order to fine tune models. There is currently a bit of a gap between LOrA creation an integration of that data into larger models but I can see serious hobbyists stepping up to that plate soon.

▲

_aavaa_ 2 hours ago | parent | prev | next [-]

The upstream models weren't built by god, they were built by distilling the world's knowledge and cultural products.

If OpenAI/Anthropic stop doing it there's no reason to believe someone else won't. Or that we won't come up with a different approach.

	▲	petesergeant 2 hours ago \| parent [-]
		> they were built by distilling the world's knowledge and cultural products ... and a bunch of RLHF and proprietary training data etc etc, but regardless, this is still an exceptionally expensive process, that someone somewhere is paying for.

▲

adventured an hour ago | parent | prev [-]

Anthropic is zooming to $100 billion in annualized revenue.

The tremendous cost is matched by the tremendous business output.

In the span of less than five years, starting from their first $1 of revenue, Anthropic + OpenAI will have produced businesses doing at least $150 billion in combined sales. The demand isn't reducing at present, it's climbing, they can't come close to servicing the total demand for their frontier models.

SAP is under $40 billion in annualized revenue. They're 54 years old.