> PoI asks these incredible questions: how do you design an AGI system that can optimize for saving lives, without it taking a strictly utilitarian view that leads inexorably to it trying to take over the world?

I don't think the show ever addressed those questions in a meaningful way? Basically there was a "good" AI, and later in the show, an "evil" one. And that was that.

The show did show us a convincing picture of the dangers of pervasive mass surveillance paired with an autonomous system that can sift through all of that data.

▲

mamcx an hour ago | parent | next [-]

> I don't think the show ever addressed those questions in a meaningful way? Basically there was a "good" AI, and later in the show, an "evil" one. And that was that.

Not super deeply but it does, and even in a way that at the time I find unbelievable (for evil side!) but today is not!:

The creator of the good machine struggle for long time with a major treat the machine present (this IA is more like old-school IA): It cares too much for his creator (to the point that basically find by his own accord the best girl for him!), to the point that the underlying terror what that for saving his creator the machine could destroy the world.

This is the main drive of the creator to reduce his ability to do so, keep communications at minimum, and learn to see the whole humanity with care.

The second drive was to care not just in the abstract of humanity and major problems like terrorism, but "minor" problems like murder (the gov only care for treats to national security) and this is huge late, because the bad IA is too focused on the greater good that means whole society engineering.

With the bad IA the problem is different: The humans WANT it become his dictator and ruler, they WANT that it takes over the world, and they WANT the IA is unbound of human morals and limits and if that means kill many to save the world so be it.

And the stressing point what that is not the opposite of the good IA, is a total different dimension that means was not easy for both IA to "talk about it". The good IA is individualist, politically neutral, minimal interventionist and all around respecting free will, the other was the wet dream of "enlighten philosopher with iron sword".

	▲	QuesnayJr 3 minutes ago \| parent [-]
		The show turned out to be 100x more plausible account of AI than I imagined when it aired. One AI becomes aligned through endless training that's a more photogenic version of the kind of alignment training they give frontier models. The unaligned AI gets none of that. I don't think the second AI is exactly unbound of human morals and limits. It's purely utilitarian, but it's utilitarian calculus is a human one, which is why Greer never questions his support.

▲

modernpacifist 2 hours ago | parent | prev | next [-]

> I don't think the show ever addressed those questions in a meaningful way? Basically there was a "good" AI, and later in the show, an "evil" one. And that was that.

It's one of my favorite shows purely because in my view, it did address it but not in a purely binary good vs evil way that many stories boil down to. Mechanically yes, there was one more easily mapped to good ASI vs the other that was mappable to evil but the beauty for me is in the nuance. There were episodes where the benefit to humanity effected by the "evil" ASI was undeniably positive and there are situations where the "good" ASI could see no path forward other than to do things we as humans would consider evil and terrible.

A line that comes up a lot in the show from the "good" ASIs creator is that he did the best he could with the skills he had. This really helped to calibrate a much more inexperienced version of me to the reality that perfection is realistically unattainable and whatever you build and hand to the world will likely have unforseen outcomes. It made me comfortable with making mistakes and using them as opportunities to learn, but it also made me focus more on my intent in undertaking a project and what I wanted the outcome to be rather than the tech I chose to do it.

	▲	Izkata 23 minutes ago \| parent [-]
		> but the beauty for me is in the nuance. Towards the end there was one scene where it became obvious to me the writers were trusting the audience to understand more than most shows without having to be spoonfed what's going on: they referenced the Great Filter, and gave no explanation at all.

▲

drybjed 2 hours ago | parent | prev | next [-]

The "good" AI did not become "good" right away, it was a long, arduous process. And the "evil" AI didn't become evil by itself either. Both had influence from humans, and perhaps that influence is what mattered the most.

	▲	Izkata 18 minutes ago \| parent [-]
		Also had multiple failures. There's an extended flashback in one episode where we learn why the "good" AI's creator was so nervous about the whole situation: something like two dozen previous attempts all did various bad things, everything from lying to him about changing its own code to trying to kill him.

▲

Semaphor 3 hours ago | parent | prev [-]

Wasn't the good AI essentially only good because it has some strict guard rails that prevented it becoming evil? Of course there was some dramatization, but that's the vibe I remember.

	▲	hxtk an hour ago \| parent \| next [-]
		Harold essentially "raised" hundreds of variants of the architecture that eventually became the "good" ASI. Most of them were maligned in some way and several of them tried to escape containment and were willing to kill him when they realized he intended to stop them. When he got one that was going well he started experimenting with it more and connecting it to more of the world and it essentially learned its values from him. Once he handed it over to the government, he essentially snapshotted the current state and had it reinitialize from that state every 24 hours, to prevent it from developing long-term plans and having alignment drift over time, but it escaped those guard rails creating a company and hiring humans to transcribe a dump of its key memories every day, entering them into the new version of the system. The "evil" ASI was essentially treated as a god; they put no effort into raising it or teaching it human values because they thought the influence of lesser beings (humans) would only be negative because it was smart enough to discover on its own anything truly useful that humans could teach it, and its followers were more or less a religious cult. And their faith was rewarded: for example, at one point one of its followers deployed a biological weapon, fully expecting that they were sacrificing themselves for the cause, but learned later that they had been chosen by the ASI for that mission because they had a rare genetic mutation that made them immune. The "evil" ASI wasn't exactly evil, per se, and generally wanted humanity to thrive, but it didn't value human self-determination to the same extent as the "good" AI and was more willing to direct its followers to enact revolution, knowing that in the short term some people would die for revolution after which it would rule the world as a benevolent dictator.
	▲	c0balt 2 hours ago \| parent \| prev \| next [-]
		The guard rails and protection for the good "AI" was mostly that it's abilities to perform actions was extremly limited. It could not, as far as I can remember, take actions outside of passing a SSN of an endangered person to the protagonists for the initial seasons and only lived of information provided by the government surveiallance network. The latter seasons and the bad "AI" mostly changed that especially as the bad "AI" was weaponized with exactly this guardrail missing.
	▲	dogma1138 2 hours ago \| parent \| prev [-]
		It wasn’t necessarily the guardrails and in fact it did break out of those guardrails and was still “good”… The actual good vs evil boiled down to a more simplistic view of good person made an AI care about humanity so it’s good even if it indeed broke through every guardrail including completely escaping its containment and going “trust me bro I’m still good” rather early on.