I know it's unpopular, or unfashionable, but I agree with this letter.

LLMs are becoming so powerful that they are dangerous. We've seen last week with the OpenAI hacking (by mistake) Hugging Face debacle.

It is absolutely ok to have open weight models at the level of GPT-OSS-100B. That one was released one year ago, and I think it's still a strong one. GLM 5.2 is a whole new level, but it appears to still be safe. Maybe Kimi K3 will be ok too. But beyond that, things will start being dicey.

It's easy to dismiss this and claim that Dario Amodei is just looking to fatten his pockets. And, sure, if Anthropic manages to put the brakes on open weight models, that reduces the competitive pressure it feels. But that does not make what Amodei's argument incorrect.

▲

bigyabai 4 hours ago | parent | next [-]

> We've seen last week with the OpenAI hacking (by mistake) Hugging Face debacle.

If the biggest danger of LLMs is that they can hack traditional systems, there is no significant threat to humanity posed by releasing them in open-weight form. Security doesn't become less of a problem by making hacking even more criminal. That's what's an unsafe mindset looks like.

▲

jackdeansmith 3 hours ago | parent | next [-]

>If the biggest danger of LLMs is that they can hack traditional systems

This is doing a lot of lifting. If the biggest danger of LLMs is they could uplift bioweapon development, the situation is different. If the biggest danger of LLMs is they reach capabilities allowing for recursive self improvement, the situation is very different still.

▲

ACCount37 4 hours ago | parent | prev | next [-]

This is the "baby's first AI risk" tier of AI danger. There is no known upper limit to how powerful those systems get, and there might not be one.

Don't think "a smart guy". Think "project Manhattan and CIA put together, all in one server rack".

We're lucky to have "they can hack traditional systems" as an early warning shot. Clearly, it's wasted on many.

▲

credit_guy 4 hours ago | parent | prev [-]

Ok. What's your stance on gun ownership then? And by gun, I mean naval guns.

	▲	bigyabai 4 hours ago \| parent [-]
		My position is that anyone can own a cannon and shells, but you only get to fire it once before the feds step in. Giving a naval cannon to the average person does not threaten humanity any more than giving them a gun or an LLM does. None of them are a panacea for anything.

▲

djsjajah 4 hours ago | parent | prev [-]

but what is the point? A ban is supposed to make a certain thing less likely to occur. Does a ban of open source models do that? Presumably, the behavior you are trying to limit is the miss-use of these models but I don't know how many state sponsored hacking groups are going to give a ban a second thought.

	▲	credit_guy 3 hours ago \| parent [-]
		Yes, that's how I read Amodei's post. Imagine Kimi K4 will be as powerful as Mythos. Anthropic can work for months and months to set up guardrails on Mythos, so when the model is finally released, it will generally decline to help hackers develop and prosecute cyberattacks, and if they do, at least there would be a trace so the law enforcement can track the perpetrators. Let's now say that Kimi K4 is released after a similar effort to develop guardrails. But being open weights, someone can just take the model, and finetune it until it does not refuse to assist in developing cyberattacks, and moreover, those people can run the model on their own private GPU cluster, so nobody can track the attack back to them. The situation is actually worse than that, most likely. Guardrails might be just markdown documents which are added to the context like regular skills. Then removing the guardrails for an open weights model does not even involve any finetuning, just removing some docs from a harness.