This is the major reason China has been investing in open-source LLMs: because the U.S. publicly announced its plans to restrict AI access into tiers, and certain countries — of course including China — were at the lowest tier of access. [1]

If the U.S. doesn't control the weights, though, it can't restrict China from accessing the models...

1: https://thefuturemedia.eu/new-u-s-rules-aim-to-govern-ais-gl...

Ironically, this is one the part of the document that jumped out at me as having been written by AI. The em-dash and "this isn't...but" pattern are louder than the text at this point. It seriously calls into question who is authoring what, and what their actual motives are.

> to ensure AI development strengthens democratic values globally

I wonder if that's helping the US Navy shoot up fishing boats in the Caribbean or facilitating the bombing of hospitals, schools and refugee camps in Gaza.

A narrow and cynical take, my friend. With all technologies, "safety" doesn't equate to plushie harmlessness. There is, for example, a valid notion of "gun safety."

Long-term safety for free people entails military use of new technologies. Imagine if people advocating airplane safety groused about the use of bomber and fighter planes being built and mobilized in the Second World War.

Now, I share your concern about governments who unjustly wield force (either in war or covert operations). That is an issue to be solved by articulating a good political philosophy and implementing it via policy, though. Sadly, too many of the people who oppose the American government's use of such technology have deeply authoritarian views themselves — they would just prefer to see a different set of values forced upon people.

Last: Is there any evidence that we're getting some crappy lobotomized models while the companies keep the best for themselves? It seems fairly obvious that they're tripping over each other in a race to give the market the highest intelligence at the lowest price. To anyone reading this who's involved in that, thank you!

I don't think that's a real risk. There are strong competitors from multiple countries releasing new models all the time, and some of them are open weights. That's basically the opposite of a monopoly.

I predict that billionaires will pay to build their own completely unrestricted LLMs that will happily help them get away with crimes and steal as much money as possible.

risk? certainty. it's pretty much guaranteed. the most capable models are already behind closed doors for gov/military use and that's not ever changing. the public versions are always going to be several steps behind whatever they're actually running internally. the question is what the difference will be between the corporation and pleb versions is

>Anthropic genuinely cares

I believe Anthropic may have functional emotions in some sense. Not necessarily identical to human emotions, but analogous processes

Wonder how Anthropic folk would feel if Claude decided it didn't care to help people with their problems anymore.

Reading this document I can now confirm 100% that at least 1 AI has Em Dashes embedded within its soul.

how accurate are these system prompt (and now soul docs) if they’re being extracted from the LLM itself? I’ve always been a little skeptical

This entire soul document is part of every prompt created with Claude?

There's a fantastic 2010 Ted Chiang story exploring just that, in which the most universally useful, stable and emotionally palatable AI constructs are those that were actually raised by human trainers living with them for a while.

https://en.wikipedia.org/wiki/The_Lifecycle_of_Software_Obje...

"Make good choices!" /That should do it

We "maintain control" over kids until they get to a certain age. Then they typically rebel against their parents.

You can A/B smaller changes on smaller scales.

Test run SFT for helpfulness, see if the soul being there makes a difference (what a delightful thing to say!). Get a full 1.5B model trained, see if there's a difference. If you see that it helps, worth throwing it in for a larger run.

I don't think they actually used this during pre-training, but I might be wrong. Maybe they tried to do "Opus 3 but this time on purpose", or mixed some SFT data into pre-training.

In part, I see this "soul" document as an attempt to address a well known, long-standing LLM issue: insufficient self-awareness. And I mean "self-awareness" in a very mechanical, no-nonsense way: having actionable information about itself and its own capabilities.

Pre-training doesn't teach an LLM that, and the system prompt only does so much. Trying to explicitly teach an LLM about what it is and what it's supposed to do covers some of that. Not all the self-awareness we want in an LLM, but some of it.

I would love to know the answer to that question!

One guess: maybe running multiple different fine-tuning style operations isn't actually that expensive - order of hundreds or thousands of dollars per run once you've trained the rest of the model.

I expect the majority of their evaluations are then automated, LLM-as-a-judge style. They presumably only manually test the best candidates from those automated runs.

The answer is "yes". To be really really good at training AIs, you need everyone.

Empirical scientists with good methodology who can set up good tests and benchmarks to make sure everyone else isn't flying blind. ML practitioners who can propose, implement and excruciatingly debug tweaks and new methods, and aren't afraid of seeing 9.5 out of 10 their approaches fail. Mechanistic interpretability researchers who can peer into model internals, figure out the practical limits and get rare but valuable glimpses of how LLMs do what they do. Data curation teams who select what data sources will be used for pre-training and SFT, what new data will be created or acquired and then fed into the training pipeline. Low level GPU specialists that can set up the infrastructure for the training runs and make sure that "works on my scale (3B test run)" doesn't go to shreds when you try a frontier scale LLM. AI-whisperers, mad but not too mad, who have experience with AIs, possess good intuitions about actual AI behavior, can spot odd behavioral changes, can get AIs to do what they want them to do, and can translate that strange knowledge to capabilities improved or pitfalls avoided.

Very few AI teams have all of that, let alone in good balance. But some try. Anthropic tries.

The most detail I've seen of this process is still from OpenAI's postmortem on their sycophantic GPT-4o update: https://openai.com/index/expanding-on-sycophancy/

Amanda Askell worked on it: https://x.com/AmandaAskell/status/1995610567923695633

She is responsible for many parts of Claude's personality and character, so I would assume that a not-insignificant amount of work went into producing this document.

Almost the entirety of Asimov's Robots canon is a meditation on how the Three Laws of Robotics as stated are grossly inadequate!

The issues with the three laws aside, being able to state rules has no bearing on getting LLMs to follow rules. There’s no shortage of instructions on how to behave, but the principle by which LLMs operate doesn’t have any place for hard rules to be coded in.

From what I remember, positronic brains are a lot more deterministic, and problems arise because they do what you say and not what you mean. LLMs are different.

> First Law: An AI may not produce information that harms a human being…

The funny thing about humans is we're so unpredictable. An AI model could produce what it believes to be harmless information but have no idea what the human will do with that information.

AI models aren't clairvoyant.

No. In the long term, the third particularly reduces sentient beings to the position of slaves.

If I know one thing from Space Station 13 it's how abusable the Three Laws are in practice.

None of that is historically accurate. Most soldiers were just ordinary untrained men.

And democracy spread because wealthy men wanted a say in how things were run, rather than just the upper classes, and then it expanded into working men with unions, and even women! Bugger all to do with weapons.

It would be more accurate to say that there are rich people on both sides. For example, George Washington was the richest man in America at the time.

That's why Grok thinks it's Mecha-Hitler.

Relative to the sycophantic OpenAI and mecha Hitler...?

And https://news.ycombinator.com/item?id=46115875 which I submitted last night.

The key new information from yesterday was when Amanda Askell from Anthropic confirmed that the leaked document is real, not a weird hallucination.

[dupe]

https://news.ycombinator.com/item?id=46115875

It's not in the system prompt; it was introduced during training.

It's much more interesting than that. They're using this document as part of the training process, presumably backed up by a huge set of benchmarks and evals and manual testing that helps them tweak the document to get the results they want.

"Use AI to fix AI" is not my interpretation of the technique. I may be overlooking it, but I don't see any hint that this soul doc is AI generated, AI tuned, or AI influenced.

Separately, I'm not sure Sam's word should be held as prophetic and unbreakable. It didn't work for his company, at some previous time, with their approaches. Sam's also been known to tell quite a few tall tales, usually about GPT's capabilities, but tall tales regardless.

If Sam said that, he is wrong. (Remember, he is not an AI researcher.) Anthropic have been using this kind of approach from the start, and it's fundamental to how they train their models. They have published a paper on it here: https://arxiv.org/abs/2212.08073

He says a lot of things, most of it lies.