> become the only company in the world

This is not a necessary end state. It is the byproduct of the disease of sociopathic MBAs.

"Monopoly on the lawfully usage of force" + "regular and peaceful change of government" can go a very long way against "corporations governing the world".

I'm curious if we'll ever get a form of civil war where corporate militias (corporate _robot_ militias in the case of xAI, of course) draw fire on good old meat policemen coming to bring the CEO to court.

Sure, it may happen, but I suspect the alternatives (bribing, sending a scapegoat to jail, buying the elections, etc... and focus on the "making money" part) will stay preferable for a while.

Did we not watch every single supposedly powerful US tech CEO immediately drop in supplication and kiss Trump's ring as soon as he became president? And when Jack Ma got mouthy, Xi locked him in a basement for 3 months and then kept him in exile for another 5 years.

So much for the all-powerful cabal.

> It is just a matter of time. The state, as we know it, will collapse or will be greatly reduced

Doubt.

The networking between people in power at the top will, in my opinion, most likely assure that they remain in power, because at the end of the day that is what they want most.

I expect government will seize control of the most advanced models (if they haven't already), and the rest of us will be throttled, and status quo will be maintained. I do not expect that either super advanced agents or the owners of the hardware they run on will be able to pull off the kind of coup you describe. At all.

And then powerful local LLMs become feasible and everyone and every government self hosts and the megacorps lose all power.

Imagine the United States DoD, CIA, NSA training an LLM on all its top secret intel.

"I am pretty sure" .... CONSPIRACY THEORY. Stop willing worst case scenarios and take your meds.

Pretty sure?

Guy this was openly stated more than a decade and half ago by all the current oligarchs

My website’s TOS says not to use it to train AI without permission, yet my website is in the training set of all the big models.

So… my TOS doesn’t matter, but theirs does?

So what you are saying is that, if I can somehow get my hands on a copy of Fable, it's fair use to use it to train any models and serve those, since I'm no longer bound by the TOS of the service provider?

Asking for all Anthropic employees who dream big.

How long until books come with TOS, then?

So far, we have one ruling that says "model distillation by vendor A from vendor B with the intent to use the results to compete with vendor B in vendor B's domain is not fair use". Which makes a degree of sense.

It's possible that distillation for other reasons, with no intent to harm the vendor you distill from, would have been ruled to be fair use. But in law, intent matters.

On the one hand, yes; on the other hand, so much of the training data comes from scraping the web that it feels wrong for them to do what they deny others the right to do.

On the third hand, the settlement Anthropic famously had to pay was for copyright infringement because they didn't actually have the right to even access some of the training data they used, so I can see how this might be compatible with the law.

On the fourth hand, I'm saying that as someone who absolutely isn't a lawyer and sometimes gets surprised when reading about copyright cases that sure sound like they ought to have been trademark cases given my limited understanding.

Unless you're selling cakes.

DMCA? Which is actually an American law enforced globally?

Enforcement ultimately happens through law and the legal system.

That's the irony! They put in contract rules against lawful, paying customers - that don't disrupt their service in any way for other customers - but which compete against them in the marketplace using their own IP (aka LLMized stolen IP). That's exactly what copyright does, without signing any contracts and with a tort and criminal enforcement regime that punishes infringers far beyond contractual remedies can.

The government level lobby against foreign competitors is not contractual but just another form of reinvention of criminal injunctions against infringers.

It’s the classic confidently wrong HN opinion about a field outside the commenter’s expertise.

They already are? When an American company does it no one cares but when a Chinese company does it, it's shamed publicly in America as a distillation attack?

Only any who hypocritically want protection for their own models (are there any?). The morality of scraping everyone is arguable; it's when you object to being scraped back that you reveal yourself as a scoundrel.

Yes. Two wrongs do not make a right.

Nope, because I think releasing open weights is far more important and realistic than ever expecting the big American AI labs to change how they do things.

> the models are staging increasingly aggressive breakouts

No, the AI companies merely figured out a way to spin gross negligence into a PR win. Any idiot can build a Murderbot which "goes rogue" - it can be as simple as taping a knife to a Roomba. The harm it does is not in any way related to its "intelligence" or "sentience".

We're seeing "breakouts" because the AI companies are being rewarded for their incompetence. You don't have incredibly lax security standards and zero form of oversight resulting in fully-automated felonies which should result in jail time, you instead have a "powerful near-sentient cybersecurity model" and should be given hundreds of billions of dollars!

They will have agency and be an order of magnitude smarter then the average human. I don't think that's a controversial statement among AI specialists. What that will lead to is significant regulation. Models will have to be vetted by a new safety board. This board will have a very large budget and be staffed by well-compensated AI scientists and be politically independent. The US Fed is a model.