SOMA was a cognito-hazard for me and my roommate in college; we played it in the dark together while on some sort of mild hallucinogen and when it came time for Simon to find that high-pressure dive suit, we lost our minds (no pun intended). Watching the WAU twist its way through PATHOS II in whatever way worked first is a particularly jarring analogy for what has happened to our own profession. I can't help but think it would be nice for Frictional Games to revisit this topic again soon.

Sidenote: It breaks my heart that all the great underwater-settings in media are hotbeds of horror scenarios. I think Subnautica broke the mold for this, here's to hoping the next generation of aquanauts take to the depths from that series.

I like to get into heated debates with friends that have played SOMA about whether or not the events in the ending that is presented to you were all necessary and effective, or perhaps undermined the overall message in some way.

Specifically (and no spoilers, but I will be talking structure), you see parts A -> B -> C.

I believe that part C makes the sequence of A -> B much less effective, by essentially removing a lot of the tension caused by seeing A, believing what it shows, and then immediately cutting to the reality of B.

C only really takes away some of that tension, and I feel like it was added because of concerns about how a simple A -> B -> fade to black, would leave players feeling. Arguably it's the truest representation of part of the game's message, but to me feels like a bit like it's shying away from really making you face the specific truth highlighted well by B.

Alternatively, keeping all the elements but playing them as A -> C -> B, would keep the message intended by seeing A -> B, and make it gentler for the player to receive, but ultimately remove the powerful effect of the buildup from A leading immediately to the reveal of B.

Dropping C entirely would lose the confirmation of 'Seeing both sides', however I believe A -> B is a more powerful vision, and players can come to question whether C even exists by themselves.

SOMA is existentialism 2.0 (wrote about it here https://p.migdal.pl/blog/2019/06/games-in-which-you-walk-and...).

To concepts you menton, I would add grey goo and x-risk.

One of my favorite games, and I recommend it to anyone who loves a good existential sci-fi horror. If you are not comfortable with stealth games, it also has a "Safe Mode" where enemies aren't a threat to you if you just want to experience the story.

Spoiler warning for those that havent played--

I forget the details exactly, but one scene stuck with me. It was a screen in one of the labs, where an experiment was running over and over. It was an uploaded consciousness of one of the test subjects, stuck in an interview room. He kept realizing he was trapped in a simulation and would start panicking. The computer would reboot him, trying another sequence to get him to not realize he was an AI. I think you as the player are given the option to turn him off forever, iirc.

The same theme was present in season 2 of Raised By Wolves. (RIP my favorite show in some years)

I just love this game. So many good things about it. Not only the one about AI you mentioned, also the issue of consciousness and the self. I simply love how the game tackles this in a way that can only be done with an interactive first person PoV.

Also, I just love this phrase:

> "I woke up in my bed today... a hundred years ago."

The thing about fiction is, it's fiction. The author can write whatever outcome they want. It doesn't necessarily tell us anything about the real world.

There's a story about some Kenyans outrunning a Cheetah in 6km. It had been killing their livestock, so they decided to go after it.

Cheetahs are very fast, but humans have way more endurance.

Yep. That said, unlike cheetahs, there’s plenty of evidence of leopards attacking humans. And these days, it’s the leopards, the closed-AI types and misanthropes -- telling everyone, “AI will take your job and there’s nothing you can do about it.”

I can destroy my refrigerator pretty easily. If I care about food inside, I can take it out into a new one. So, seems I control it by that definition. Your idea "but food will spoil"/"I will lose a small amount of wealth" seems irrelevant to the strict definition.

Conversely I think it's a bad definition, it's a show of what is the frame of the mind of the person who states that: "I want to show my control by destroying my things, look how powerful I am" which sounds like a toddler. That's how you portray psychopathic/narcissistic disorders in movies.

Could you elaborate on that last part?

Isn't this a misinterpretation of what everyone in the AI safety space is worried about, though? I think the idea is that having an AI that interprets everything in a super-literal way would probably be catastrophic, but we can't even build that. It would be a nice world-ending problem to have.

You forgot the Luddites. No AI-is-a-tool fallacy is complete without the Luddites! Alternatively, one can use the Antichrist.

Lem’s Cyberiad was my favorite as a kid and still is!

I predicted on this site in 2016 the massive social and economic impacts AGI would have and specifically when RL data loops are not available to anyone but major players:

https://news.ycombinator.com/item?id=12168228

I even wrote up a whole article that specifically called RL loop based development as the future:

https://medium.com/@andrewkemendo/the-ai-revolution-will-be-...

> Reinforcement Learning tasks rely on ridiculous amounts of data. Whereas with traditional software architecture, where you accomplish tasks through explicit task instruction, RL trains for tasks based on millions of tests through a reward system. Most importantly once you have trained it to some minimum level, if you deploy it correctly, then it should continue improving — so long as you bake feedback into the UX. Imagine that instead of telling excel what to do, you and every other user will have a conversation with excel, improving the system incrementally.

For all its talk of inoculation, this is a terribly written essay. They do not make a point, nor even arguments, instead, opting to ramble in hopes that you forget whatever it was you were thinking.

The issue is simple. Just like us (who are arguably complex, look at what we're building over here, this AI computer stuff!), entities have simple core needs (like food, water, power, etc.).

An infinitely smart AGI has the potential, nay, likely cause, to require infinite resources. We're already seeing the effect in the computing sector on e.g. chips, there's no reason to think this trend won't continue...

Lets circle back to the hydrogen argument, will we blow ourselves up. Real concern, abated by hard numbers. Different atmosphere, different concentrations, different pressure, different possible outcomes.

Today, we don't have those numbers. We don't have those calculations. I don't disagree with the point at the end "about how people can exploit other people, or through carelessness introduce immoral behavior into automated systems". These are issues, too. But saying there are other issues, don't worry about this big issue over here, is the absolute worse argument possible.

That's hand waving.

If anything the hard takeoff theory is too conservative. It turns out you don't need self-improvement to get to superintelligence. You just need a ridiculous amount of money. Where can you get a ridiculous amount of money? The market will give it to you because FOMO.

AI is easier than people 10 years ago thought it would be. It's also easier to align than people feared it would be. It's the humans using the AI that are hard to control.

1. Like climate change, right?

2. At what cost? Much like the climate change above, you'll have people on the AI side even when it's out in the field extincting us.

4. Adding, over time synthetic data and its generating algorithms can become unaligned with human needs/behaviors (an example would be our current stock market, numbers must go up!).

8. Going back to climate change, it was predicted a long time ago, and while the explosion of automobiles has greatly improved human lives the risks of climate change could erase a lot of that. Might have been better if we dealt with the problem before we have to give the thermometer worried looks.

That's just Pascal's wager with a secular/cyberpunk reskin, and has all the same trappings. Why aren't you a monk?

Ah yes, Pascal's mugging.

"Intelligent agents" we have around run on a metabolic budget of 25W and a hardware platform the size of a melon.

Human intelligence doesn't scale upwards well. Individual humans only get this smart, and there are gains from getting multiple humans to work together - but the more of them you add, the larger is your communication and coordination overhead. In no small part because humans are self-interested agents that simply aren't designed to compose their capabilities seamlessly. You can't get a vastly superhuman intelligence simply by piling together more humans.

Human intelligence doesn't scale sideways well either. Unskilled labor is cheap and plentiful, but if you have a human with a very specific skill, the process of getting more of that capability is very long and very involved. Often, it's easier to redesign an entire process to run on worse humans than it is to train more humans for better performance.

Institutions are more capable than individuals, but far less capable than the sum of individuals within them. At many corporations, the majority of individual productivity is absorbed by management overhead and corporate rot.

AI isn't bounded by those limitations.

AI can scale intensively and extensively. AI can be scaled up by upping the compute budgets. AI can be replicated and copied indefinitely. AI doesn't have the innate human "I don't live to work, I work to live" overhead. AI can outclass human intelligence by a long shot.

The "moat" that's there is already being eroded by modern day LLMs. Betting that future AI systems can't cross it is folly.

I have the same feeling. I'm not worried about superintelligent AI because we are only training them on human level intelligence. By what mechanism does our current AI technology take the leap to technologies that humans have never conceived of?

Our current AI is more like a fancy Google search than some kind of machine God.

I'm old enough to remember when grey goo and nanotechnology was the apocalyptic scenario du jour for a short time after some guy at MIT wrote a book, and because he was at MIT people took it seriously even though it was ridiculous. If someone at the University of Kentucky or Kansas had written such a book, it would have been ignored. When prestige manages to align with bad ideas, it's pretty awful, and it can derail the entire civilization for a while.

I was like... nanotechnology and grey goo already exist. It's called biology. The scenarios I was reading were silly. They violated conservation laws and laws of physics. But people were believing it and calling for limits on nanotechnology research.

I remember arguing with smart people on this, and that was when I started to realize that there's two kinds of dumb. I had the same realization later when I argued with an incredibly intelligent guy who was absolutely convinced the moon landings didn't happen. See, there's dumb-dumb and smart-dumb, and the people who thought grey goo would eat Earth or that the Apollo landings were a hoax were the latter. Smart-dumb is high-IQ rationalization of ultimately irrational and absurd ideas, and the smarter you are the more effectively you can do this.

I've met some really shockingly brilliant fools over the years who believe in all kinds of outlandish conspiracy theories, absolute literalist religious fundamentalism, idiotic political doctrines that directly contradict basic logic and all of lived human history, and so on. All of them can engage in sophisticated airtight rationalizations.

I sometimes wonder if this is one of the evolutionary forces constraining intelligence. In my experience, smarter people are somewhat more likely to believe highly sophisticated and complex stupid things, and they are much better at convincing others of these things. That's probably more dangerous to them, their family and friends, and the species than dumb people believing simple silly things that are easily debunked.

On AI...

Is AI potentially dangerous? Very. It's already dangerous in a number of ways. The biggest right now is probably mass production of personalized propaganda, mass surveillance, and mass manipulation. There's also the potential that bad actors could use it to accelerate their ability to make things like garage WMDs (biotech, chemical weapons, etc.). None of this requires hard take-off superintelligence. It's just inherent risks to a powerful technology.

These are not entirely new risks. They were already present in the Internet and computing. AI just raises them to a higher level.

The extreme hard take-off stuff is silly, and it actually distracts us from talking about the much more realistic dangers and coming up with reasonable solutions that don't also throw away the huge benefits of these technologies.

Nobody cares what we (people who have been working on AGI a long time) think.

We were dismissed as cranks before and now we’re just ignored by whomever is promising the most money to investors.

So, par for the course. Everyone in AI has lived through all the cycles so far so this is just the biggest one yet.

The fact that AI researchers and heads of labs aren't being assassinated tells me that the people who claim they are concerned about the end of the world aren't actually that serious.

Even in the LLM dept: LLMs are the most general AI systems to date, and the performance only ever goes up.

Whether adding +5% per model release is enough to get a broadly superhuman system remains to be seen. But my take is that there's no such thing as "not working on AGI" in the frontier labs. Everything that's being put into modern frontier systems is AGI groundwork, one way or the other.

It’s not advances on the underlying operation of matrix multiplication that have driven ai progress to date. It’s the layers above that; trying different neural architectures (transformers w/attention mechanisms), and also different data and training regimes (different ways of doing reinforcement learning) that are the main drivers of improved performance. Perpetual motion is a physical impossibility. Whereas Ai is already being used to improve the workflow of ai researchers, thus speeding up improvements in said research. It’s not hard to see that AI could well be spun up to continue to try new arrangements of the aforementioned levers that drive ai progress on its own.

Presumably there's more efficient hardware foundations to perform these efficiently, and potential at the various abstraction layers for more efficiency. Obviously this is not unbounded - simple things would seem to have a physical limit to the potential improvement.

But if you think of the optimization space: different physical representations, different approaches (photo, quantum, etc), more parallelism - there's undoubtedly a lot of headroom even on the matrix multiplication side. I would imagine there's a lot left on the table when it comes to the abstractions we've built. Infinite? No, but lots of potential.

And what does a machine with a few orders of magnitude more power come up with? I'm not readily able to predict what something like that could create (maybe it's tapped out, but I doubt it).

It seems to come down to an article of faith (as referenced in the article) that there's a lot more potential to be extracted in our current exploitation paths. Which I think is probably reasonable.

Heck, even if a theoretical machine tops out at 3-5 orders of magnitude faster/more complex, I'm sure that could do some amazing things that look like magic to us.

I actually agree. At some point, a RSI system has to interact with real-world, and that imposes serialization constraints. It is harder to know how much that slow-down would be and how much speed-up we will get before that. But a RSI cannot simply be a exponential growth forever.