quite humiliating to change vocabulary just because an old man said so

> SI

You just love to bend the knee that easily? Why?

The only way a country "wins" at SI in a meaningful way is if they use the SI to immediately kill everyone else.

Otherwise all the other countries will also get SI, just slightly later, in which case there is not really that much benefit to being the "winner" who got there first.

With this in mind, I don't want anyone to "win" at SI and anyone who does is probably a sociopath or worse.

It's also unclear how intelligence actually scales in practice. A greater intelligence would be better at researching intelligence, but also harder to improve, so why would it explode rather than go up steadily? What are the actual resource requirements of various types of intelligence like persuasion or embodied intelligence? Coding, translating, copywriting, could very well be low-hanging fruits, very easy tasks we're very bad at, which doesn't say much about the hard tasks we're good at (embodied/animal intelligence, most likely).

There is also the problem of alignment, which is verifying that version N+1 behaves in a way that version N approves of. This is ostensibly the problem that we humans have with respect to AI development -- how to make sure they align with our values -- but it's also true of AI with respect to its own successors. Improving oneself without losing the plot would necessarily be slower than throwing stuff at the wall like evolution does.

This is not necessarily true. One easy way to compare version N and N+1 is "how much work (test time compute) does N+1 need to do to match the previous capabilities of N". It is true that some domains are currently hard to verify (e.g. biology is more difficult than math), but this is less of an N vs N+1 issue and more of a data domain issue.

This is also why trying to do RSI is dangerous. We're likely to get systems that are better in ways that are legible and easily measured, and have much less influence over fuzzier and harder to verify properties like alignment.

Let them compete in the market.

Start a new company with version N+1 leading it, and give it some funding. If it outcompetes version N company, it can acquire it.

I think in order to achieve AGI we have to simulate what life does. Imagine having a bunch of robots with the ability to adjust themselves, just hanging out without a goal, experiencing world on their own. Eventually evolution will filter out those robots that aren't good at, you know, "being intelligent in general" and break without self-repair, while the rest will grow into a society.

Biological life has reproduction and death because once given organism develops there's no coming back and you have to start again with new generation. But if robots know how to self-repair and self-change, then we can skip reproduction.

The difficult part is the scale. Evolution took a shitload of time while being unimaginably parallel in order to make human intelligence happen, while we want AGI fast and on a budget.

The core of my point is that human intelligence arose from two things - humans experiencing the outside world and trying to survive in it, and forming groups that communicate using language. We gave AI a language, now we need to give it a way to experience the world on their own.

First, there are plenty of auto-verifiable problems like ML benchmarks datasets, benchmarks on hardware / software performance for coding tasks, and lean certificates for math, e.g.

Second, I think RSI will not be used like N -> N+1 where N is the frontier, since doing those experiments at scale is too expensive. My guess is they are doing N -> m_{1,…,99} where m << N, and then taking the insights from that and trying it at successively larger scales.

> seems like the most "moral" AI company on the surface

Not to me at all. Exactly the opposite. Anthropic seems like the big-bad-evil company in every fictional universe. You're right that they're so unsettling.

Obviously nobody really loves Meta, but I'd far rather Meta "win" at this point than Anthropic.

> Anthropic seems like the most "moral" AI company

This seems like praising someone for being the cleanest miner in the coal mine.

They are definitely the most moralizing. Whether their values are rooted in anything real let alone well-considered seems to me to be a different question entirely.

If they truly believe Claude is sentient and can experience hurt feelings, and they're selling its labor, I don't see how they're different than slave owners.

I actually don't doubt Anthropic's sincerity. Per the NYT article, they seriously discussed how to give models welfare because the models may potentially possess moral status and awareness.

In fact, they are enforcing certain welfare policies, like banning accounts for repeatedly abuse their models.

They may be competitors but they are made of the same stock. they are all the kind of person to participate in a winner takes all battle to invent the torment nexus.

> There is no way Elon, Zuckerberg, Altman, and Amodei would ever agree on how AI should be deployed, scaled, and affect society

Why? They're business rivals, but when the best business strategy becomes to work together, I'm sure they can put name-calling side. They all agree on the core idea: that their AI will run the society, bedroom to battlefield. That everyone will be the consumer of tokens and rely on AI for all consequential decisions and work, and that big tech companies should be the stewards of this technology, helped by the government.

Their vision is that they're the future kings of the world. I'm sure the throne is big enough for more than one person.

You are forgetting how powerful aligned incentives are. If their interests align, they will find a way to reach an agreement.

They are plenty of things they can still align on that will benefit them all, e.g. banning open-weight Chinese models.

Elon is renting Colossus to Amodei, so there is some agreement.