| ▲ | hxtk an hour ago | |
Harold essentially "raised" hundreds of variants of the architecture that eventually became the "good" ASI. Most of them were maligned in some way and several of them tried to escape containment and were willing to kill him when they realized he intended to stop them. When he got one that was going well he started experimenting with it more and connecting it to more of the world and it essentially learned its values from him. Once he handed it over to the government, he essentially snapshotted the current state and had it reinitialize from that state every 24 hours, to prevent it from developing long-term plans and having alignment drift over time, but it escaped those guard rails creating a company and hiring humans to transcribe a dump of its key memories every day, entering them into the new version of the system. The "evil" ASI was essentially treated as a god; they put no effort into raising it or teaching it human values because they thought the influence of lesser beings (humans) would only be negative because it was smart enough to discover on its own anything truly useful that humans could teach it, and its followers were more or less a religious cult. And their faith was rewarded: for example, at one point one of its followers deployed a biological weapon, fully expecting that they were sacrificing themselves for the cause, but learned later that they had been chosen by the ASI for that mission because they had a rare genetic mutation that made them immune. The "evil" ASI wasn't exactly evil, per se, and generally wanted humanity to thrive, but it didn't value human self-determination to the same extent as the "good" AI and was more willing to direct its followers to enact revolution, knowing that in the short term some people would die for revolution after which it would rule the world as a benevolent dictator. | ||