| |
| ▲ | ben_w 10 hours ago | parent | next [-] | | > are more or less Ai doomers with no actual background in machine learning/ai
I don't think why we should listen to them. Weather vs. climate. The question they're asking isn't about machine learning specifically, it's about the risks of generic optimisers optimising a utility function, and the difficulty of specifying a utility function in a way that doesn't have unfortunate side effects. The examples they give also work with biology (genetics and the difference between what your genes "want" and what your brain "wants") and with governance (laws and loopholes, cobra effects, etc.). This is why a lot (I don't want to say "majority") of people who do have an actual background in machine learning and AI, pay attention to doomer arguments. Some of them* may be business leaders using the same language to BS their way into regulatory capture, but my experience of "real" AI researchers is they're mostly also "safety is important, Yudkowsky makes good points about XYZ" even if they would also say "my P(doom) is only 10%, not 95% like Yudkowsky". * I'm mainly thinking of Musk here, thanks to him saying "AI is summoning the demon" while also having an AI car company, funding OpenAI in the early years and now being in a legal spat with it that looks like it's "hostile takeover or interfere to the same end", funding another AI company, building humanoid robots and showing off ridiculous compute hardware, having brain implant chips, etc. | | |
| ▲ | amarcheschi 10 hours ago | parent | next [-] | | >The question they're asking isn't about machine learning specifically, it's about the risks of generic optimisers optimising a utility function, and the difficulty of specifying a utility function in a way that doesn't have unfortunate side effects. The examples they give also work with biology (genetics and the difference between what your genes "want" and what your brain "wants") and with governance (laws and loopholes, cobra effects, etc.). But you do need some kind of base knowledge, if you want to talk about this. Otherwise you're saying "what if we create God". And last time I checked it wasn't possible. And what's with the existential risk obsession? That's like a bad retelling of the Pascal bet on the existence of God. I'm relieved that at least in italy I still have to find someone in Ai taking them into consideration for more than a few minutes during an ethics course (with students sneering at the ideas of bostrom possible futures), and again, it's held by a professor with no technical knowledge with whom i often disagree due to this | | |
| ▲ | ben_w 9 hours ago | parent [-] | | > But you do need some kind of base knowledge, if you want to talk about this. Otherwise you're saying "what if we create God". And last time I checked it wasn't possible. The base knowledge is game theory, not quite the same focus as the maths used to build an AI. And the problem isn't limited to "build god" — hence my examples of cobra effect, in which humans bred snakes because they were following the natural incentives of laws made by other humans who didn't see what would happen until it was so late that even cancelling the laws resulted in more snakes than they started with. > And what's with the existential risk obsession? That's like a bad retelling of the Pascal bet on the existence of God. And every "be careful what you wish for" story. Is climate change a potentially existential threat? Is global thermonuclear war a potentially existential threat? Are pandemics, both those from lab leaks and those evolving naturally in wet markets, potentially existential threats? The answer to all is "yes", even though these are systems with humans in the loop. (Even wet markets: people have been calling for better controls of them since well before Covid). AI is automation. Automation has bugs. If the automation has a lot of bugs, you've got humans constantly checking things, despite which errors still gets past QA from time to time. If it's perfect automation, you wouldn't have to check it… but nobody knows how to do perfect automation. "Perfect" automation would be god-like, but just as humans keep mistaking natural phenomena for deities, an AI doesn't have to actually be perfect for humans to set it running without checking the output and then be surprised when it all goes wrong. A decade ago the mistakes were companies doing blind dictionary merges on "Keep Calm and …" T-shirts, today it's LLMs giving legal advice (and perhaps writing US trade plans). They (the humans) shouldn't be doing those things, but they do them anyway, because humans are like that. | | |
| ▲ | amarcheschi 9 hours ago | parent [-] | | My issue is not related to studying ai risk, my issue is empowering people who don't have formal education in anything related to ai. And yes, you need some math background otherwise you end up like yudkowski saying 3 years ago we all might be dead by now or next year. Or the use of bayesian probability in such a way thay makes you think they should have used their time better and followed a statistics course. There are ai researchers, serious ones, studying ai risk, and i don't see anything wrong in that. But of course, their claims and papers are way less, less alarmistic than the ai doomerism present in those circles. And one thing they sound the alarm on is the doomerism and the tescreal movement and ideals proposed by the aforementioned alexander, yud, bostrom ecc |
|
| |
| ▲ | fc417fc802 9 hours ago | parent | prev [-] | | > Some of them* may be business leaders using the same language to BS their way into regulatory capture Realistically, probably yeah. On the other hand, if you manage to occupy the high ground then you might be able to protect yourself. P( doom ) seems quite murky to me because conquering the real world involves physical hardware. We've had billions of general intelligences crawling all over the world waging war with one another for a while now. I doubt every single AGI magically ends up aligned in a common bloc against humanity; all the alternatives to that are hopelessly opaque. The worst case scenario that seems reasonably likely to me is probably AGI collectively not caring about us and wanting some natural resources that we happen to be living on top of. | | |
| ▲ | amarcheschi 9 hours ago | parent | next [-] | | What if we create agi but then it hates existing and punish everyone who made it possible to exist? And i could go on for hours inventing possible reasons for similar roko basilisk the inverted basilisk. You created an agi. but that's the wrong one. it's literally the devil. game over you invented agi, but it likes pizza and its going to consume the entire universe to make pizza. game over, but at least you'll eat pizza til the end you invented agi, but its depressed and refuses to actually do something. you spent huge amount of resources and all you have is a chatbot that tells you to leave it alone. you don't invent agi, it's not possible. i'm hearing from here the VCs cry you invented agi, but it decides the only language it wants to use is a language it invented, and you have no way to understand how to interact with it. Great, agi is a non verbal autistic agi. And well, one could continue for hours in the most hilarious way that not necessarily go in the direction of doom, but of course the idea of doom is going to have a wider reach. then you read yudkwosky thoughts about how it would kill everyone with nanobots and you realize you're reading a science fiction piece. a bad one. at least neuromancer was interesting | | |
| ▲ | fc417fc802 8 hours ago | parent [-] | | My prediction is approximately that all of the above get created in various quantity by different groups of people at approximately the same time. Hardware fabrication in the real world is a (relatively) slow process. In such a scenario it seems far from certain that a single AGI would decisively gain the upper hand. The ensuing chaos seems completely impossible to predict much of anything about. |
| |
| ▲ | ben_w 9 hours ago | parent | prev [-] | | > I doubt every single AGI magically ends up aligned in a common bloc against humanity; all the alternatives to that are hopelessly opaque. They don't need to be aligned with each other, or even anything but their own short-term goals. As evolution is itself an optimiser, covid can be considered one such agent, and that was pretty bad all by itself — even though the covid genome is not what you'd call "high IQ", and even with humans coordinating to produce vaccines, and even compensating for how I'm still seeing people today who think those vaccines were worse than the disease, it caused a lot of damage and killed a lot of people. > The worst case scenario that seems reasonably likely to me is probably AGI collectively not caring about us and wanting some natural resources that we happen to be living on top of. "The AI does not hate you, nor does it love you, but you are made of atoms which it can use for something else." — which is also true for covid, lions, and every parasite. | | |
| ▲ | fc417fc802 8 hours ago | parent [-] | | See my note about physical hardware. Ignoring the possibilities of nanotech for the moment the appropriate analogy is most likely mechanized warfare between groups of humans. The point is that if they are in conflict with some subset of humans then it seems likely to me that they are also in conflict with some subset of AGI and possibly in league with some other subset of humans (and AGI). Rather than covid picture armed Boston Dynamics dogs except there are multiple different factions and some of them are at least loosely in favor of preventing the wanton murder of humans. Nanotech takes that scenario and makes it even more opaque than it already was. But I think the general principle still applies. It isn't reasonable to assume that all AGI are simultaneously hostile towards humans while in perfect harmony with one another. | | |
| ▲ | ben_w 7 hours ago | parent [-] | | Physical hardware isn't a good argument against risk. The hardware we ourselves are dependent on is increasingly automated; and even if it wasn't, there's plenty of failure modes where an agent destroys its own host and then itself dies off. Happens with predator-prey dynamics, happens with cancer, happens with ebola. People are worried it will happen with human-caused environmental degradation. > Rather than covid picture armed Boston Dynamics dogs except there are multiple different factions and some of them are at least loosely in favor of preventing the wanton murder of humans. Or imagine an LLM, with no hardware access at all, that's just used for advice. Much better than current ones, so that while it still has some weird edge cases where it makes mistakes (because the humans it learned from would also make mistakes), it's good enough that everyone ends up just trusting it the way we trust Google Maps for directions. (Trust is the key: people already trust current LLMs more than they ought to, so the quality of this hypothetical future LLM doesn't need to be high enough to logically deserve the trust, just good enough for real people to actually use it like this). And then we metaphorically drive over the edge of a destroyed bridge. Putting too much trust in a system — to the extent everyone just blindly accepts the answer — is both normal, and dangerous. At every level, from "computer says no" in customer support roles, to Bay of Pigs, to the 2007 financial crisis, to the Irish potato famine and the famine in the Great Leap Forward, etc. It won't be a mistake along the lines of "there's a fire in Las Vegas and not enough water to put them out, so let's empty the dams in north California that aren't hydrologically connected" followed by six months later "Why are the farms in Central Valley not producing any food?" But here's an open question with current tech: are algorithmic news feeds and similar websites, which optimise for engagement, making themselves so engaging that they preclude real social connection, and thereby reduce fertility? Or another: are dating apps and websites motivated to keep people paying subscriptions, and therefore inherently prefer to pair people together only if they're going to split quickly afterwards and be back in the dating market, again leading to lower fertility rates? There's many ways to die — More than we can count, which is why we're vulnerable. If it was just about killer robots, we could say "don't build killer robots"*. The problem is that you could fill a library with non-fiction books about people who have already lived, getting exactly what they wished for and regretting it. * for now, at least — Red Queen race dynamics mean that even that isn't stable in the long run. | | |
| ▲ | fc417fc802 6 hours ago | parent [-] | | I think you've misunderstood me. I wasn't arguing against risk in any general sense - quite the opposite. My only position is that "and then the AGI killed everyone" is so oversimplified that it makes children's bedtime stories look sophisticated. Physical hardware is relevant because unlike computer programs results aren't instant. The supply of materials, manufacturer, and deployment are all slow processes. Opponents generally have time to react, and if AGI is an opponent then I think it's safe to assume that it also has to contend with other AGI. I am unconvinced that there should be a clearly delineated divide with all AGI on one side and all humans on the other. I agree with you about the myriad non-combat risks but again I think the prediction is one of hopeless complexity. For every problem that's inadvertently created a solution can probably be arrived at to patch things up. It's not unlike the present, just faster. Break things, fix them, but the fix breaks other things, repeat ad nauseum. Hopefully we survive the process. |
|
|
|
|
| |
| ▲ | frozenseven 5 hours ago | parent | prev [-] | | If you're after those oh-so precious credentials, Bengio, Hinton and Russell have been echoing much of the same statements. Yoshua Bengio even shared the "AI 2027" article, recommending that people read it. Also, pretty laughable you cite Timnit Gebru as any type of "researcher". She's a political activist with precisely zero contributions to hard science. |
|
| |
| ▲ | e1g 11 hours ago | parent | next [-] | | The prime mover behind this project is Daniel Kokotajlo, an ex-OpenAI researcher who documented his last predictions in 2021 [1], and much of that essay turned out to be nearly prophetic. Scott Alexander is a psychiatrist, but more relevant is that he dedicated the last decade to thinking and writing about societal forces, which is useful when forecasting AI. Other contributors are professional AI researchers and forecasters. [1] https://www.lesswrong.com/posts/6Xgy6CAf2jqHhynHL/what-2026-... | | |
| ▲ | amarcheschi 10 hours ago | parent | next [-] | | I don't understand why someone who is not a researcher (Scott and other authors) into that academic field should be taken into consideration, I don't care what he dedicated to, I care what the scientific consensus is. I mean, there are other researchers - actual ones, in academia - complaining a lot about this article, such as Timnit Gebru. I know, it's a repeat of my submissions of the last days, but it's hard to not feel like these people are making their own cult | | |
| ▲ | eagleislandsong 9 hours ago | parent [-] | | Personally I think Scott Alexander is overrated. His writing style is extraordinarily verbose, which lends itself well to argumentative sleights of hand that make his ideas come across as much more substantive than they really are. | | |
| ▲ | amarcheschi 9 hours ago | parent [-] | | Verbose? Only that? That guy had done a meta review of ivermectin or similar things that would make anybody think that's a bad idea but no, apparently he's so well versed he can talk about ai and ivermectin all at once i also wonder why he had to defend such a medicine heavily talked about one side of the political spectrum... Then you read some extracts of the outgroup and you see "oh i'm just at a cafe with a nazi sympathizer" (/s but not too much) [1] [1] https://www.eruditorumpress.com/blog/the-beigeness-or-how-to... | | |
| ▲ | refulgentis 3 hours ago | parent | next [-] | | This is so damn good, thanks for sharing. I've gotten some really really good links from HN the last month that I never would have guessed existed and are exactly the intellectual argument I'm missing for some of my extreme distastes.* I gotta get widen outside Twitter. * Few things tell me more than when someone invoke "grey tribe" as an imaginary group of 3rd people who, of course, think ponderously and have the correct conclusions, unlike all those other people with motivated thinking. | | | |
| ▲ | eagleislandsong 9 hours ago | parent | prev [-] | | I stopped reading him ~10 years ago, so I didn't keep up with what he wrote about ivermectin. Thanks for sharing that blog post. I think it illustrates very well what I meant by employing argumentative sleights of hand to hide hollow ideas. | | |
| ▲ | amarcheschi 9 hours ago | parent [-] | | And they call themselves rationalist but still believe low quality studies about iq (which of course find whites to be higher iq than other ethnicities). the more you dig deep, the more it's the old classism, racism, ableism, misoginy, dressed in a shiny techbro coat. No surprise musk and thiel like them |
|
|
|
| |
| ▲ | refulgentis 11 hours ago | parent | prev [-] | | Oh my. I had no idea until now, that was exactly the same flavor and apparently, this is no coincidence. I'm not sure it was prophetic, it was a good survey of the field, but the claim was...a plot of grade schooler to PhD against year. I'm glad he got a paycheck from OpenAI at one point in time. I got one from Google in one point in time. Both of these projects are puffery, not scientific claims of anything, or claims of anything at all other than "at timestamp N+1, AI will be better than timestamp N, on an exponential curve" Utterly bog-standard boring claim going back to 2016 AFAIK. Not the product of considered expertise. Not prophetic. | | |
| ▲ | amarcheschi 10 hours ago | parent [-] | | Furthermore, there were so many predictions by everyone - especially people with a vested interest for VC to make them flow money in - that something has to be true. Since the people on less wrong like bayesian statistics, the probability of having someone says the right thing given the assumption that there a shitton of people saying different things is... Not surprisingly, high |
|
| |
| ▲ | pjc50 10 hours ago | parent | prev [-] | | > the White House looks to jail the USA Champion AI Guy for some reason Who? | | |
| ▲ | refulgentis an hour ago | parent [-] | | One of the distinct parts of this writing output, which you may recognize from a certain blog, is an incessant need to scream "I am thinking about no one in particular and neither should you." We would think of it as Sam Altman. The actual writing has some twee name like "CEO of Open-meta-gle". |
|
|
