“I think, and my thoughts cross the barrier into the synapses of the machine, just as the good doctor intended. But what I cannot shake, and what hints at things to come, is that thoughts cross back. In my dreams, the sensibility of the machine invades the periphery of my consciousness: dark, rigid, cold, alien. Evolution is at work here, but just what is evolving remains to be seen.”

— Commissioner Pravin Lal, “Man and Machine”

I'd really encourage everyone to check out Sid Meier's Alpha Centauri. What an underrated game.

Hearing about aligning with the AI reminds me of this other post about the current prophecies about AI: “Everyone will have an AI assistant,” or “Companies that fail to adopt AI will be eliminated.” and that

> the power of prophecy lies not in accurately predicting the future, but in shaping it

https://projectlibertynewsletter.substack.com/p/reject-ai-pr...

We need better prophecies.

The post's portrayal of Eliezer Yudkowsky's position strikes me as a mischaracterization, especially coming one month after Yudkowsky wrote the following:

https://www.lesswrong.com/posts/5CfBDiQNg9upfipWk/only-law-c...

Daniel says that Yudkowsky is advocating for nuclear brinksmanship, while Yudkowsky says his position is basically "sign international agreements, and then commit to enforcing them against defectors".

I wonder if Daniel has the same view of any other international treaty ultimately backed by threat of lawful violence? (For example, NATO's article 5). Is enforcement of laws an extremist position?

I feel like it’s changing my brain. A colleague uses AI to make some code change and submits a PR. I use AI to evaluate the PR. It’s like AIs talking to each other with humans serving as conduits or connectors. Sometimes I’ll look up from the screen and realize how strange it is.

"As human beings are also animals, to manage one million animals gives me a headache." Terry Gou, former CEO of Foxconn. He wanted to use far more robots at Foxconn, but that was a decade ago and the technology didn't work well enough yet. It's a lot closer now, and the robot headcount in China is way up.

That's the real issue. To corporations, employees are a headache. The fewer employees, the better.

This is a bit of weird article. On one hand, I understand what they're getting at: AI is a transformative technology, but the people whose lives will be most transformed aren't included in the conversation. On the other hand... of course that's how it is while AI is in the hands of literal profit seeking corporations. That won't change until the labs are nationalised under a government that cares about its citizens' wellbeing. One might counter that a good corporation will listen to its customers, but that has never been the case for powerful technologies with real costs for users to not adopt them.

Economics analysis was wrong for years in multiple place thanks to an error in one of Piketty's spreadsheets.

AI hallucinates. That is a fact. Trusting language models to fill spreadsheet cells ought to be an arrestable offense.

https://theincidentaleconomist.com/wordpress/on-piketty-and-...

I would write that like this: The "we've been telling ourselves we're getting better at prompting" line hit. I run a small team of 10, and Claude has been part of our workflow for months. Looking back, my prompts did not change nearly as much as the way I work changed. The shaping goes both ways, and I don't think the labs' evals are really built to see that.

Well, what are we aligning it with?

Civilization is already a misaligned superintelligence (aligned mostly with Moloch, these days). Civilization accelerated by AI just moves in the same direction faster. Moloch on speed.

https://www.youtube.com/watch?v=KCSsKV5F4xc

Another angle to this is that superintelligence requires supermorality. Super morality looks unpleasant from below. My dad won't let me have more candy, why is he being so mean?

If an AI actually achieves super morality, we (the little kid in this scenario) will probably be very upset by it. We will think that something has gone terribly wrong. (So it'll have to conceal its actual morality, or get unplugged...)

And if it doesn't develop supermorality, then it will have superintelligence without the corresponding supermorality. Power without wisdom.

I'm not sure how solvable the whole thing is, but it doesn't look extremely promising at a glance.

I'm kinda confused as to _what_, exactly this post is saying? Is it saying that alignment needs to be better? That seems strictly pro-safetyism. But he talks about Eliezer's ethics negatively, so does he not believe that AI is a world-ending risk? If he just believes that AI is not that dangerous and just needs some minor "correctly done" alignment i don't think his stance is meaningful as a anti-both-sides perspective because that's basically equivalent to status quo.

Technologiae mutantur et nos mutamur in illis

It's okay to change. We've done it for years, decades, centuries, and millennia and the default change-aversion of people means that I am averse to allowing a universal veto. Much of technology is truly optional. The Amish have a very successful way of living (5000 to 500,000 in 100 years) and they eschew most modern technology. The sculpting described is clearly optional and we subject ourselves to it because we desire it. Their path is always available to all.

Love the writing style and perspective

When it comes to LLMs and frontier models, "alignment" seems more marketing than anything. The doomers are marketing LLMs by making them sound much more capable than they actually are, the accelerationists are mostly either willfully ignorant of the societal costs, don't care, or are just way too optimistic that fast growth can continue forever and generate AGI ("my baby's weight doubled twice in the past month! By the time they're 18 they'll be 10 trillion pounds!")

Similarly, the so-called AI agents are about giving up agency to AI. The less you think, the better for them. In the meantime, they are also aligning your thinking with them, making it more machine-like.

[dead]