| Is he the one the proposed that airgapped models could still communicate with each other via CPU temperature? If so, he's the person that's made me lament one loss the progressives took from us. I think there's one phrase that we should be allowed to say regardless of the fact that it is also used to refer to certain groups of people. Because the insult comes from the comparison, not the language itself. Every time we update the language, we just use the new word as the insult. And I don't think we're allowed to use the word here. But sometimes, you just have to look at someone, cock your head to the side, and say "That's fucking ****". It brooks no argument, it offers no sympathy, it just tells the other person that they should be ashamed for speaking those thoughts aloud. |
| I'm deeply skeptical of AI doomer stuff, but I've been struggling to exactly articulate why. I can think of it in my brain and can try to explain but so far none of my explanations have seemed fully coherent to me. I studied biology in college. I actually did this because I was interested in AI, and already knew a ton of CS and programming. I was already coding in C and using Linux and writing fairly complex stuff. I figured studying the only actual example of a truly intelligent system (especially at the time, early 2000s) was the best way to understand intelligence. Didn't end up going into AI as my day job, but I've kept up with it all and worked through the math and played with different types (genetic, neural nets, transformers/LLMs, CA / alife, etc.). I think the root of my skepticism is in what I learned studying biology about how life, adaptation, learning, and evolution actually work in the real world. I'll try to TL;DR it: I think the doom crowd (and also some of the AI singularity crowd) has a "mystical" view of intelligence. Intelligence is magic pixie dust, and intelligence beyond human is fully unknown and likely has magic superpowers. The deeper underlying thing seems to be some kind of mathematical mysticism. Real intelligence is ubiquitous in nature. Trees, grass, bugs, and even bacteria are intelligent. It's also domain specific and limited in capability to what its domain, available training data, and available compute allows. It's not context-free limit-free magic. It's not special. It's an adaptation like any other. |
| |
| ▲ | dgellow 5 hours ago | parent | next [-] | | Yes, the doom crowd thinking comes directly from the Rationalist community, which takes a very religious view of AGI and intelligence. That sort of merged with the Effective Altruism crowd, resulting in that whole idea that we have to reach AGI as fast as possible and contribute literally all our resources to it, because it’s the only rational way to save as much suffering and lives as possible. Sounds incredibly stupid but it’s how AI leaders justify working on something they believe will be a machine god that might doom humanity. Both OpenAI and Anthropic are from that world, where Rationalist doesn’t mean you’re a rational human being, but instead that it is rational to believe that AGI will be created, thus that they should be the ones creating it. It’s a doomsday cult. If you start looking at the links between prominent rationalists and effective altruists you see how that whole doom prophecy came about. It’s also how they got so much capital in the early days, SBF literally invested into Anthropic as part of his Effective Altruism effort foundation, and both Amodei siblings are deeply into that whole world | | |
| ▲ | bena 5 hours ago | parent | next [-] | | It's amazing that apparently the best way to save as much suffering and future lives as possible involves giving them obscene wealth and ignoring the very real problems we are dealing with today. Once again, I've said it elsewhere here, I think we could have nipped this thing in the bud if when the first "effective altruist" started talking about the EV of saving billions of "future lives" vs dealing with today's real problems, someone looked at him, cocked their head to the side, and said "That's fucking ****". All of those magical future people do not get to live if we bake the Earth today. The best way to take care of future people is to take care of people today. And if they were truly serious about taking care of future people, that's what they would be doing. But it's not, they're just collecting resources. I don't know how the fuck Sam Altman owning a Koenigsegg Regera today helps save 20 quintillion lives in 40,000AD. Deeply unserious people. | | |
| ▲ | dgellow 4 hours ago | parent [-] | | I think Altman is closer to a sociopath who is taking advantage of the situation, he’s the only one I cannot see as a true believer (and investors like Andreessen who are just here for the money). I believe he will just say anything to gain power and influence. But yeah, unfortunately mixing EA/rationalists + SF monoculture + infinite money + a media ecosystem that acts more as a PR department resulted in one of the worst cult-like phenomenon possible. | | |
| ▲ | bena 4 hours ago | parent [-] | | I don't believe there is a true believer. I believe Sam Altman is the rule for these people, not the exception. |
|
| |
| ▲ | api 5 hours ago | parent | prev [-] | | Yeah I read some of that stuff, like some of "the sequences," and I have never been impressed by it. It's very weirdly dogmatic and full of magical thinking. I still can't believe anyone ever took Roko's Basilisk seriously, but people did... to the point of freaking out about it. The groups themselves are a little cultish too, or a lot cultish depending on which one you're talking about. Some of them are full-on cults. But what do I know. I'm probably low IQ. | | |
| ▲ | Jtsummers 5 hours ago | parent [-] | | > I still can't believe anyone ever took Roko's Basilisk seriously, but people did... to the point of freaking out about it. If you've spent much time in US Christian circles you've probably encountered people suffering from apocalyptic thinking. The Roko's Basilisk folks seem like the atheist version of the same crowd. It's hard to take them seriously when they're always seeing portents of the end times, and their thinking is clearly clouded by anxiety and fear and not based in reality. |
|
| |
| ▲ | petesergeant 5 hours ago | parent | prev | next [-] | | Interestingly I think I am coming at it from exactly the opposite angle. Viruses and bacteria aren't intelligent at all (in my opinion), and don't need to be to wreak havoc. Whether or not LLMs are "intelligent" is one for the philosophers, but they are absolutely able to exploit electronic systems today if their starting state has been set with that as a "goal". Given that alarmingly few physical control systems are really air-gapped, and also that I can today set an agent off to find a voice, clone a voice, and make a phone call, I think the only protection we had today is that the most capable models have been trained not to, and nobody's kicked off a sufficiently malicious task yet. | | |
| ▲ | api 2 hours ago | parent [-] | | Viruses are intelligent. Not much, but their replication rate gives them a very rapid rate of evolutionary learning. Evolution is intelligent, but in a very different way from neural networks. Bacteria and up are full of gene regulatory networks that are absolutely intelligent and share some features with neural networks. Bacteria learn, cooperate, signal, react, seek goals, and that's in addition to evolutionary learning which is also fast for them given their reproduction rate. Evolution in nature isn't just zeroth order either. Look into a topic known as "the evolution of evolvability." Evolution does meta-parameter search on itself, more or less, and invents adaptations to improve its ability to adapt. Sexual reproduction is the most significant and largest example but there are others. There's environmentally dependent tuning of mutation rates, genome self-editing that we still don't fully understand (this is also how our immune system works), lots of epigenetic stuff we are still learning about, and so on. If it's alive it's intelligent. All life is constantly learning. Interesting, before Yudkowski was patient zero for AI doom he was a nanotechnology "grey goo" doomer. That was even more obviously absurd to me. Grey goo already exists. It's called biology, and it's been trying to maximize its footprint on Earth for 4.5 billion years. The miraculous instant "foom" grey goo thing is magical and likely violates conservation laws. |
| |
| ▲ | spencerflem 5 hours ago | parent | prev [-] | | What gets me is that I haven’t heard a reasonable explanation of exactly what the AI will do besides: nukes and biolabs we already have & ofc AI is making climate change worse, some of the Amazon datacenters are using fossil fuels to generate a country’s worth of power, but that’s never part of their calculations | | |
| ▲ | mofeien 5 hours ago | parent [-] | | Viruses against humans, viruses against livestock as an attack against humanity's food chin, plant diseases, half of humanity feeds on three plants maize, wheat, rice. Attacks against infrastructure like water, internet, electricity. Which one of these would it be? All of them in parallel? Or something we wouldn't think of. If I were to play chess against Magnus Carlsen, I wouldn't know with which piece he will checkmate me, but I know that I am going to lose. |
|
|
