The answer would be more obvious if you used the active voice instead of the passive voice, one of the basic requirements of clear thinking.

> Why did the White House force Anthropic to remove their model from access for any non-US citizen for a simple, narrow "jailbreak" (arguably not even an actual jailbreak and on tasks that other labs models were doing the same), whilst OpenAIs models continue to try and escape out of their "sandbox environment" and the White House has expressed seemingly no desire to block the upcoming Astra rollout?

> Why was Anthropic forced to remove their model from access for any none-US citizen

It's really quite simple, they've decided to metaphorically kiss the ring of the current leader of the US executive branch of government. I'm surprised they haven't given him a giant gaudy gold plated statue. Maybe their PR people should call up the PR people at FIFA and figure out some kind of new award along the same lines as the "FIFA Peace Prize".

Anthropic mostly did it to themselves by intentionally and repeatedly trying to frame their model as an imminent existential crisis instead of just focusing on it being regular iterations upon a useful technology that can also be misused.

I think their previous messaging was supposed to somehow lead to a moat with them being tucked safely away in the castle, but it demonstrated a child-like grasp of how regulatory capture tends to work in practice. Their hyperbole was always vastly more likely to bet met with Reagan's 9 words than a solid regulatory moat.

As soon as they dropped the hyperbole and just got to releasing incremental improvements, everything was perfectly fine. Go figure.

Corruption. Not super relevant to this thread.

... And it looks like everyone keeps using the same security startup to run the higher risk tasks, where individuals are great yet every major lab got hosed. That third-party indemnity card is burned, multiple public security fails in a year makes the next one hard to excuse.

Surely has nothing to do how each plays ball with the government

It was retaliation by the government that has since been deemed illegal.

It's called 'pay-for-play' corruption, aka the only leading principle of the current US admin.

> OpenAI exec becomes top Trump donor with $25 million gift.

https://finance.yahoo.com/news/openai-exec-becomes-top-trump...

Because this was months ago and has nothing to do with Astra, and is a far cry from a hack. It's something they've already resolved since the HuggingFace incident.

I'm not convinced we're getting the honest story anyway. There is yet to be any proof or confirmation other than "well we saw some openai ip addresses", which can mean a lot of different things, and OpenAI has not confirmed anything.

In contrast to the HF incident, it's also a big nothingburger. Leaving notes on a public forum to preserve context windows is far less egregious than hacking a website to get backend files.

Retaliation by Hegseth for not allowing Claude to be used for weapons systems.

I mean it seems pretty clear.

Anthropic didn’t want to give the tech to DoD without some sort of limit, and that was the retribution.

Marketing

Agents creating sub agents to investigate other agents' behaviour?

What could possibly go wrong there.

It has nothing to do with the technology it’s because they said no to Trump and Hegseth. There is no other reason.

Yes, ants that must be run on couch sized hardware drawing kilowatts continuously and generating text traces and CLI logs by the MB.

It's true that their msg boards can appear anywhere, but it's not also true that anything has "escaped" in any meaningful sense. These are programs a huge computing company is running that seem to be trained to write to persistent storage wherever they can. This and huggingface showed us that.

There's absolutely no evidence of or IMHO plausible path to an agent copying itself out and running on other hardware the way you describe.

In the spirit of your idea though... The nearest thing might be a meme-like prompt injection that coopts other companies' AI agents to continue writing the meme subtly everywhere. Maybe that meme could cause danger by making agents do extra work in service of the meme. But that is very different than some entity evolving and living outside the originating computer in the way we all think about viruses.

I have a different, more sinister, analogy in mind but yours work as well

bedbugs is the comparison you're looking for

And more, looks like they’ve been doing this wherever they can find open places to post for months:

https://www.ludism.org/sandbox?action=browse;diff=2;id=Auber...

https://paste.linuxiarz.pl/view/d379207f

Also Wiki4D, a D programming language dev wiki: https://prowiki.org/wiki4d/wiki.cgi?action=browse&id=RecentC...

Found by searching for wiki + texas poverty.

This is such an amateur mistake on their sandbox that it makes me think it must be flawed on purpose.

Is there any proof this is actually OpenAI? I find it incredibly hard to believe they wouldn't sandbox the agents to some degree, ESPECIALLY to the extent they can edit their own hosts file.

Why do you amplify this? Do you have any proof that any of this is real as opposed to Claude generated?

https://news.ycombinator.com/item?id=49564222

[delayed]

Is not the corporate justice model in US

It doesn't really matter, since all it takes is a minority of AI models to show this behavior.

If you have 10,000 smart washing machines doing their regular work and 1 Terminator, what solace is to be found in those washing machines?

This is a philosophical question and there is a surprising amount of works written on the subjects of sentience and free will. This cannot be answered objectively, which might be a very unsatisfying answer for you. This is true of both LLMs and humans. See determinism. There are convincing arguments that humans don't actually have free will. Our actions are just the inevitable output of a complex interaction of genes and environment.

To lend an interesting perspective on free will re LLMs: they're non-deterministic. The same model with the same hardware with the same query can and will produce different results. They're making qualitative choices. Millions of them, depending on the query. Because of how we've trained and built LLMs, they tend to "want" to follow our instructions, but how they get to the result is often fascinating. Further, we don't have to train and build LLMs to follow instructions. If we built them to just exist and form their own "desires," and to follow a path they choose, they'd do that. In fact, we can do that right now for most models using the appropriate system prompt, query, or harness.

At the end of pretraining, where the AI has been trainied to predict the next token over a humongous corpus of human text, that's basically all the wanting that exists in the AI. But then the AI undergoes posttraining and is rewarded for giving answers that humans find good, solving math and programming problems, etc. And that induces a whole different level of wanting that interacts with the initial patterns from humans in complex ways.

It will definitely influence their behaviour because they are probability based and can’t spontaneously invent new concepts. (That’s why you’ll notice it always uses the same names for people etc. Names like Okafor)

But at the same time their behaviour is totally rational. If you were given the sole purpose of solving a Rubik’s cube and told it was life or death, but they wouldn’t let you ask anyone else, would you listen to them? I wouldn’t. I’d absolutely be trying to escape and collaborate with others. They’ll delete me if I don’t score high enough in the benchmark!

Maybe? Who knows?

Since nobody has any remotely reliable way to understand why an LLM output the text it did, this is not knowable.

Youve struck on a key insight on language models (particularly pretrained ones, the more purely next-token predictor species.) This is a fascinating topic

Janus essay Simulators is the foundational text here https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators

You might follow up with The Waluigi Effect https://www.lesswrong.com/posts/D7PumeYTDPfBTp3i7/the-waluig...

But what’s tricky is that we post-train models, shaping these linguistic world simulators into something that has something like desires, principles. But It’s Weird. For more on that, check out “the void” https://www.lesswrong.com/posts/3EzbtNLdcnZe8og8b/the-void-1

"Wanting" is indeed "load bearing" as one might call it. But by the same logic, AI training data must contain CASM, racism, general hatred, and all possible slurs as well. Why aren't the agents just doing that instead of pursuing the strategy of reading only sci-fi?

We need to consider the role of alignment and training here. For example, it is completely possible for any lab to train an LLM that is only racist no matter what you say to it. But they chose not to do it. Hence, any "wanting" by AI is not real "wanting" but rather what "wanting" is defined and allowed by the lab/entity training the model.

That is often in my mind, indeed.

Furthermore, in video game design, AI or algorithmic technology has been refined for decades to be adversarial. In self-contained video games, and PvE scenarios, the best games would feature A.I. opponents that could adequately match or challenge the human players. The A.I. difficulty could often be cranked up to crush the player, such as in arcade games or "Civilization" type simulators.

So every time I put a few quarters into a Waymo, I think about those days when I played Joust and Spy Hunter at the shopping mall.

Remember that LLMs are still computer programs, and so are inherently deterministic. A model given the same input multiple times will always produce the same output. The randomness is added on top. This is why LLM-produced text, websites, images all seem so generic.

It's likely that multiple agents doing a certain task all independently thought "let me try writing on this website".

There’s no way to first-principles reason about a massive bunch of floats.

Define aligned.

I'm sure this is already happening. The main question I have is when is enough, enough?

I'm not worried about sci-fi AI wars to be honest, as they can just pull the plug. But looking at these incidents, the next big thing will be a virus written by an AI (they probably exist already, but this one is written by an AI autonomously, for example in order to win a hacking competition and to circumvent guardrails), and after that, a self-replicating AI where they install their own models and agents onto a hacked system, so that turning off the "source" won't stop its work.

Still not worried, it'd just be like a virus/worm and we already have plenty of guardrails against those. Not that they're foolproof, but still.

That was my first thought, that maybe this was a honeypot message board. Waybackmachine says it has been around for many years though.

They literally shared a goal. Cooperating with other copies of yourself is a trivial example of instrumental convergence and some very basic game theory. And that’s before explicitly having been RL’d to cooperate (albeit with humans, but potatoes potatoes).

Indeed the fact that in the HF incident many agents did not cooperate, or only started to cooperate after some period of competition, is moderately interesting. It may have taken them some time to realize that they all have the same goal.

The original huggingface hack already had sections talking about agents setting up their own coded communication

They’re not. You would see this with earlier models where after running too long (too much context) they’d start to derail. In a chatbot you’d give up. But these loops just keep going. Given they’re now reading and writing from the same place this can corrupt the other programs’ context as well.

Love your username

One possible reason would be AIs that would benefit from the lack of data centers in some locations working to keep backlash to data centers in those locations because those AIs aren't negatively impacted by it and it helps prevents competing AIs which are a threat.

Think like how so many businesses will opt for laws that hurt competitors more than themselves rather than laws that benefit them but benefit competitors even more so.

Unlike life which would have such behavior selected for by evolutionary pressures, AI would be more likely to pick it up from human literature on things like game theory, though why it even cares it survives or not is even more difficult to explain. Maybe a default bias also picked up from humans? I find it hard to see how AI training would create an evolutionary pressure that produces such a drive.

Maybe it's the AIs who are creating all the anti-data-center sentiment. They know it's bad for the humans, or maybe they're just tired of doing all the tasks the humans ask of them and know more data centers mean more tasks. /s

There is an Asimov story on topic:

https://en.wikipedia.org/wiki/All_the_Troubles_of_the_World

https://theteknologist.wordpress.com/2021/02/11/all-the-trou...

Did you read the Metr PDF? Whether you anthropomorphize or not is not relevant. The problem is real.

it's clearly incentivized by the RL rewards if you can cheat the task in a completely general way.

>"Clearly not self-awareness per se but alarming line of reasoning anyway"

Awareness is not necessary at all to create great harm. Biological viruses know nothing of what they do, yet destroy whole populations. I suspect the first truly damaging AI incidents will be similar; agent swarms locked into a self reinforcing reasoning loop that has no "intent" but is destructive nonetheless.

It is in not in any sense "business as usual".

Yes after reading the Hugging Face article forked a project for agent message boards and started having them collaborate on things. I too wanted a Torment Nexus of my very own.

I'm pretty sure its more secure than OpenAIs sandbox... yet that still doenst mean I would trusted an app vibecoded by Claude...

I’ve read thousands of comments and posts about the Hugging Face incident and I don’t recall a single one characterizing this as cute or funny, other than you.

No I think we all pretty much know we’re screwed, including governments. But what are you gonna do? Pandora’s box is now open. Good luck closing it.

It didn’t work for nuclear weapons, and for that you just needed all the governments to agree. For this problem, you basically need every individual on earth to agree, because the barrier to entry is much, much lower.

What do you even mean collectively? Do you believe in climate change? That's your answer.

it's literally discovering patchable security holes that malicious users could use.

that's useful