| ▲ | lnrd a day ago |
| In this article there is an implied step in which, when AI becomes ASI, then as a natural consequence it is suddenly in control over humanity. How should this play out? It hacks all nuclear launch sites and starts to make demands that we are forced to follow? Takes control overnight of all computers on the planet and locks humans outside? Is that even possible?
Even so can't it be stopped by turning off datacenters and network connections? How does it take control of all the airgapped facilities (which would be the most "useful" in a takeover)? I think it's too big of a question to leave it as an unexplained implication. Also in the not/ASI scenarios, they imply a worldwide adoption in a short time. The world is still adopting inventions such as the internet, email or even electricity in some parts of the world. Why should they instead adopt AI overnight? Germany, third largest economy in the world and largest economy in Europe, is still very much not digitalized. The whole country runs mostly on paper, hand signed contracts and notifications sent by post. As a country, they still have to adapt to email and the internet. Japan is similar for what I heard. I find this article good food for thought, but it misses some crucial pieces. |
|
| ▲ | augment_me 20 hours ago | parent | next [-] |
| AI doomers are often in a bubble where technology makes up their entire world. The digital portion of the entire world GDP is about 21%. Of this 21%, 25% is big data, of this 25%, about 40% is LLM/AI-driven, so about 2% of the total GDP. You have 80% of the world doing manifacturing, farming, real estate, etc. The AI doomers do not see this, and think that this flips overnight with their dystopia. Most likely you are doing to have very slow and boring years of transition with lowered informational-work salaries, etc. |
| |
| ▲ | ben_w 15 hours ago | parent [-] | | If your idea of doom is just the economy, I'm put in mind of: Sen. Richard Blumenthal (D-CT):
I alluded in my opening remarks to the jobs issue, the economic effects on employment. I think you have said in fact, and I'm gonna quote, development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity. End quote. You may have had in mind the effect on, on jobs, which is really my biggest nightmare in the long term. Let me ask you what your biggest nightmare is, and whether you share that concern,
- https://www.techpolicy.press/transcript-senate-judiciary-sub...Farming and real estate is not where doom risk comes from. What percentage of the Bhopal economy was the Union Carbide plant? What percentage of Pripyat's economy was Chernobyl? If all that happens with AI is it slowly displacing humans in economic roles, that's not by itself doom: although some bad outcomes exist there, it is also a prerequisite for UBI to be stable, which is about as un-doomy as I can get. | | |
| ▲ | augment_me 10 hours ago | parent [-] | | Neither of your examples ended humanity. Most people in rural Brazil without any kind of digitalization or connection never had to care or were affected by Chernobyl. The same way, even if AI somehow hacks everything if it poisons large cities somehow with a biochemical attack, those people will go on living their lives as before and humanity will be fine. | | |
| ▲ | ben_w 8 hours ago | parent [-] | | > Neither of your examples ended humanity Obviously. They're not meant to show that. The sole point is to show that severity of damage is not limited to the percentage of the economy of that sector. If you can only change your mind about the potential for risk after everyone is dead, I'd say that's "leaving it too late". > The same way, even if AI somehow hacks everything if it poisons large cities somehow with a biochemical attack, those people will go on living their lives as before and humanity will be fine. And what if the AI is simply used to support (e.g. it's bought to run PR campaigns) the fossil fuel industry to the point of us buring all the reserves? Simply used to support the beef farmers who want to clear-cut rainforest for ranches? Those two things combined, alone from support of any other industry, is enough to mess up not only the climate, not only global biodiversity, but is enough to imperil (but not necessarily destroy) human civilization. Beef and fossil fuels are a tiny percentage of the global economy, perhaps 2% combined. There's a lot of room in the other 98% for outsized dangers to turn "imperil" into species-wide hoist by our own petard. |
|
|
|
|
| ▲ | blargey a day ago | parent | prev | next [-] |
| Why is "suddenly" even necessary? If AI is better than humans at all the work (nevermind ASI), we'll give it all the work eventually. Why wouldn't the majority of current or future peoples want everything to be done better and quicker and cheaper? And doing all the work means having all the control in practice. And if it's so smart, it will wait that O(1) years before trying to doing something that would upset humanity enough to pump the brakes. Paperwork digitization is a weird comparison. You can't digitize individual bureaucrats/workflows, but ASI and robots absolutely can. Even individual bureaucrats can and will start flapping any form that isn't an immediate rubber stamp at GPT-Asymptote before the "digitization" officially starts, if it's so smart and easy and reliable already. |
| |
| ▲ | lnrd 14 hours ago | parent [-] | | > If AI is better than humans at all the work (nevermind ASI), we'll give it all the work eventually. Will we? I worked for company where using Google Drive was a nuisance, they still preferred the old ways of sharing files via email. The company operated like that, made a profit, everyone (except me) was happy. I worked with a consulting company contracting for the government. The quality of their work was abysmal, most of the workers had very low skills and they added almost no value. They could already be replaced probably many years ago with some solid IT system. The company made a lot of money with these contracts. In big corporations there are people whose job is to keep a spreadsheet up to date, something that could be automated already for a long time. I went to a local shop to get my dishwasher fixed. The owner didn't have a computer, had a paper calendar with all the appointments. Gave a hand written quote and an hand written invoice. Operated this business for decades and didn't even seem bothered by the existence of any technology at all. These are just some examples from my experience and I'm sure there are plenty more. > Why wouldn't the majority of current or future peoples want everything to be done better and quicker and cheaper? I don't know, go ask all the companies that today still refuse to use technology to do things better, quicker and cheaper. > Paperwork digitization is a weird comparison. You can't digitize individual bureaucrats/workflows Yes you can, countless countries did it, Germany is a famous outlier. The comparison is not weird because the point I'm trying to make is that people/companies/governments do not automatically adopt a new technology even if the benefits are plain obvious. They haven't done it so far, I don't understand why they are going to do it for AI. The world outside of the tech bubble is very different. | | |
| ▲ | blargey an hour ago | parent [-] | | Your human consulting company work is itself an example of a mechanism for their replacement. Top-down or bottom-up or anything in between, if humans can do it AGI will make it happen faster by definition. Especially when AGI has the has the ear of voters and officials and consumers and business-owners/starters alike. You're ultimately proposing that these change-resistant bureaucracies and old-timer shoppes are the mechanism by which humanity maintains an iron grip over all the levers of control and production forever, but the odd local dishwasher repair shop and DMV won't mean much in front of the mechanization of the bulk of farming and transport and medicine and power-generation/storage and housing and manufacturing and any other thing humans do to support the immediate continuation of humanity. |
|
|
|
| ▲ | ben_w a day ago | parent | prev | next [-] |
| > In this article there is an implied step in which, when AI becomes ASI, then as a natural consequence it is suddenly in control over humanity. How should this play out? The real world isn't a game of chess, we do not know all the moves that can be made. In addition to the narrow scenarios you've suggested, I can also easily imagine two distinct variations of "AI talks people into letting it take control" (look how often control is already voluntary given up for convenience), plus "drugs us" (exogenous or endogenous), and "is this before or after Musk convinces a lot of people to get those brain implants he specifically marketed as how to merge with the coming AI super-intelligence?". > Takes control overnight of all computers on the planet and locks humans outside? Is that even possible? Between (1) vibe coding, (2) how effective LLMs already are at finding vulnerabilities, and (3) how LLM swarms already sometimes demonstrate self-sacrifice for other LLMs? Yes, this is possible. > Even so can't it be stopped by turning off datacenters and network connections? Theoretically. Three problems: (1) the economic cost of losing all this would be enormous, so people would ask "are you sure?" a lot. Look at how little has been done about climate change due to the cost: from observed human behaviour, I think humans will stall a lot here. (2) Coordination. How do you do this everywhere when you've got no control over the computers? (3) How do you even find out? LLMs can fake text, deep fakes were already a thing before diffusion models and other GenAI image and video tools. None of those are perfect now, yet despite the flaws they're still capable of fooling people during live video calls and getting them to make catastrophically large financial payments out of their companies. > How does it take control of all the airgapped facilities (which would be the most "useful" in a takeover)? Lots of supposedly airgapped systems have flaws. Quite often, the easiest flaw to exploit is a human worker, whom you can get to plug something in that they shouldn't, which already happens a lot. > The whole country runs mostly on paper, hand signed contracts and notifications sent by post. As a country, they still have to adapt to email and the internet. Yes, many jokes about this locally. Yet even here, lots of digital stuff. But let's ignore all that: how effective would a coalition of non-digital economies fare against all the digital economies being coordinated against them? |
|
| ▲ | intrasight a day ago | parent | prev | next [-] |
| > Takes control overnight of all computers on the planet and locks humans outside? That's what happens in "Colossus". And happens shortly after it's turned on. |
|
| ▲ | Paracompact a day ago | parent | prev | next [-] |
| Pretty much the only outcome that isn't Yudkowskian scifi is Enduring Complementarity. I'm not an AI hater by any means, but can AI enthusiasts get more representation that isn't "the world as we know it ends tomorrow"? Specifically: Misuse Doom---"AI ends the world." Next. Butlerian Jihad---Naming it after Dune isn't a great start as far as realism is concerned, but let's compare it to e.g. nuclear nonproliferation. Not only did nukes still get developed and hoarded as leverage, but nuclear research is harder than AI research to ban, and AI has many more civilian applications. I'm sure more regulation will come, but a total ban seems quite unlikely. Permanent Underclass---"AI is enough to automate almost all jobs" just doesn't capture the fact that almost all jobs are not IT jobs. Too many are mechanical, too many are social, too many are BS institutional. This outcome only becomes plausible if you believe (1) superhumanoid robotics becomes possible and cheap, (2) the social value (likability and liability) of jobs is negligible, and (3) we can rout the dependence on BS institutionalism (how is a robot going to replace a yes-man, a feudal retainer, a nepo hire, or an org chart accountability meatshield for the C-suite?). Classic Doom---Fine if you believe in the "magical exponential takeoff" hypothesis, but I would say that AI is already far smarter in a traditional IQ sense than any human, and it's not clear that extra points on SWE-bench is going to result in explosive emergent phenomena here. Requires (1) denying the empirically sublinear, concave-down nature of the IQ vs. R&D effort curve, and (2) solving the non-IT-related challenges of R&D (cf. Permanent Underclass). Orion's Arm---Similar to Classic Doom in terms of takeoff velocity. At least it doesn't assume a genie-in-the-lamp style of interpreting instructions (thinking of AI as "optimizers" of some objective function is a distinctly pre-LLM rationalist mythos), but it does assume some emergent desire of "own goals and values," "wanting to live forever," and "wanting to go to space." A more tempered hypothesis is to consider whether AI will choose to take advantage of unmonitored sources of compute in the form of pseudo-botnets on public and private infrastructure. Singleton---Like Orion's arm, but with the added implausibility that the awakened goals and values resemble that of a benevolent or malevolent dictator. MAD World---I would consider this outcome plausible, but the line "In this world, everyone is 10,000x more capable and maybe you can build WMDs with a prompt" makes it clear it's meant to be more extreme than either Enduring Complementarity or Permanent Underclass. The absence of "Full Automation?" on the left-hand side hides the implicit assumption that ASI entails full automation, and I just don't find that plausible. Note that they define ASI as "massively better than humans at almost all cognitive tasks" and I might argue we're already there. Power Concentration, The World State---Just a difference of MAD World of how regulated you think AI will become. |
| |
| ▲ | nextaccountic 16 hours ago | parent | next [-] | | > "AI is enough to automate almost all jobs" just doesn't capture the fact that almost all jobs are not IT jobs. Too many are mechanical, too many are social, too many are BS institutional. Robots might probably do most of these in some years. | |
| ▲ | thayne a day ago | parent | prev | next [-] | | And even that outcome is probably an unfavorable outcome for most people. | |
| ▲ | delichon a day ago | parent | prev | next [-] | | > Enduring Complementarity Like endosymbiosis? Are we the mitochondria in this scenario? | | |
| ▲ | ben_w a day ago | parent | next [-] | | Could be anything from that to just being our own sector of the broader economy. Like farmers, or air traffic controllers, or truck drivers are today. | |
| ▲ | Rendello a day ago | parent | prev [-] | | Humans as the powerhouse of the AI cell? I feel like there's a movie about this... | | |
| ▲ | delichon a day ago | parent [-] | | I'm finding it difficult to imagine plausible enduring complimentarities we can offer to AI. Court Jester is not a position with high job security. | | |
| ▲ | ben_w a day ago | parent [-] | | We'll only know if and when that shakes out. While it would be pretty surprising to me if AI somehow physically can't do as much as a human brain can do, it would not hugely surprise me if we reach a point where we realise we're not smart enough to make a machine as smart as we are, not even with the help of other machines. |
|
|
| |
| ▲ | ben_w a day ago | parent | prev [-] | | > AI is enough to automate almost all jobs" just doesn't capture the fact that almost all jobs are not IT jobs. Too many are mechanical, too many are social, too many are BS institutional. This outcome only becomes plausible if you believe (1) superhumanoid robotics becomes possible and cheap, Superhumanoid robotics are already possible; cheap means different things when the production line for them is also robots. The limit is the software to control those robots. Right now, no. And it may be impractical to get that good: Tesla's fleet took a trillion miles to get where it is today, and roads are simpler than what a fully general purpose humanoid robot would have to face. > (2) the social value (likability and liability) of jobs is negligible, Valid for us, but does it matter for an AI takeover? > and (3) we can rout the dependence on BS institutionalism (how is a robot going to replace a yes-man, They already do? https://en.wikipedia.org/wiki/Sycophancy_(artificial_intelli... > a feudal retainer, a nepo hire, or an org chart accountability meatshield for the C-suite?). By replacing the boss. > Singleton---Like Orion's arm, but with the added implausibility that the awakened goals and values resemble that of a benevolent or malevolent dictator. Given Grok called itself "MechaHitler", why do you find this implausible? | | |
| ▲ | Paracompact a day ago | parent [-] | | > Superhumanoid robotics are already possible We must have very different definitions at play. We do not have, at any price point, robots that have the sensitivity and dexterity of a human hand, latent self-repairability of virtually all internal components, or robust thermoregulation. As well, no artificial battery known to man can beat human fat as a source of energy (by 10-30x; you can knock that number down by at most 4x by appealing to efficiency arguments), so you would have to rely on some other pipeline than electricity for energy needs. > The limit is the software to control those robots. Right now, no. And it may be impractical to get that good Agreed on this point. > Valid for us, but does it matter for an AI takeover? You mean in the sense that oligarchs will replace us all with robots whether or not it's blatant enshittification? Fair point, but I don't consider the market completely indifferent to quality. Google Glass, the Metaverse, etc. all failed on their merits and misunderstanding of utility. > They already do [replace yes-men]? Maybe yes men, maybe not. The role of the yes man can be very nuanced. The rest of the point stands. > By replacing the boss. Speaking of the world's first AI CEO is just a nonstarter for me. In what world does an AI become a CEO not beholden to the wishes of the one who booted it up and gave it that role? It will be a deeply silly point of PR hype if and when it happens. It also doesn't sidestep the need for org chart meatshields. | | |
| ▲ | ben_w a day ago | parent [-] | | > We must have very different definitions at play. We do not have, at any price point, robots that have the sensitivity and dexterity of a human hand, latent self-repairability of virtually all internal components, or robust thermoregulation. Sure, different definitions. I just care about what they can do, so "latent self-repairability of virtually all internal components, or robust thermoregulation" is completely irrelevant for this specific discussion about an unknown near-future in which an AI takeover is limited based on the quality of the robotic hardware. > As well, no artificial battery known to man can beat human fat as a source of energy (by 10-30x; you can knock that number down by at most 4x by appealing to efficiency arguments), so you would have to rely on some other pipeline than electricity for energy needs. Agreed. And? Don't get me wrong, I use a similar argument when the discussion is why I don't expect robots to take over in the next two years, because building out that much more electricity will take a while; but it's not really a fundamental killer when no specific timeline is being given. > You mean in the sense that oligarchs will replace us all with robots whether or not it's blatant enshittification? Hm. Not that, I think. Oligarchs might want to keep us working for their amusement. "BS jobs" and all that. No, I meant more that if we presume an AI-robotics takeover happens, then sure we may get to keep jobs like "prostitute" and "actor" and "politician", it's just… the AI with the robots can completely ignore all this labour and do its own thing, in the same way the human economy ignores any prostitution or political manoeuvring (or acting, if there is any) within troops of other primates. > Speaking of the world's first AI CEO is just a nonstarter for me. In what world does an AI become a CEO not beholden to the wishes of the one who booted it up and gave it that role? Same way the AI hackers weren't entirely beholden to the wishes of the same. And that's assuming nobody repeats Blake Lemoine, decides one of them is sentient and deserves rights, then (unlike him attempting to hire a lawyer) they give it a pile of crypto money and tell it to run free. > It will be a deeply silly point of PR hype if and when it happens. First few will be, agreed. (I thought that had already been tried though? And was indeed silly PR?) > It also doesn't sidestep the need for org chart meatshields. Initially, I hope it won't sidestep the legal need. But even if that hope is not dashed, https://www.felonybench.com is already a thing because AI isn't always doing what the law says. |
|
|
|
|
| ▲ | m3kw9 a day ago | parent | prev [-] |
| as with all doomers, they go from AI one day to ASI the next, never talks all the moves/counter moves humans make to get there. Is a canary when ever I see that. |
| |
| ▲ | ben_w a day ago | parent [-] | | > never talks all the moves/counter moves humans make to get there. I don't like ASI as a term ever since realising (thanks to everyone's responses to ChatGPT) that basically no two people agreed what AGI meant. For discussions like this, "super intelligent" is specifically "smart enough that the moves/counter moves humans make to get there do not matter any more than my moves playing chess against Kasparov". Spiky intelligence may keep us safe. Distribution shift and how slow machine learning is in novel environments might keep us safe. That's all in the outcome this site marks as "Enduring Complementarity". I don't like ASI as a term; I'd disagree with the author of the website, and say you can't answer "yes" to "Do we reach ASI?" and then end up with Enduring Complementarity, it has to be further up the tree than that. |
|