| ▲ | BoppreH a day ago |
| This should be the wake-up call to pause development of models any bigger, enforced globally just like nuclear proliferation. Even without considering rogue AIs, our society relies on individuals having limited destructive power. A dictator can ask the armed forces to bomb an entire country, or a lone terrorist can bomb a small number of targets, but no lone terrorist can bomb an entire country. Here current AI has enabled one person to solve what is arguably the most famous problem in mathematics. What happens if a jackass uses the next AI to design a supervirus, or break AES, or develop an unpatchable hardware attack (a la rowhammer) that bricks computers remotely? The last month of non-stop vulnerabilities is just the tip of the iceberg as models improve and people have more time to work with them. It's not too late, but we must stop now. |
|
| ▲ | netsec_burn a day ago | parent | next [-] |
| Speaking to "break AES" - this is the wrong way to look at the problem. If AI finds AES is broken, then it was always broken, and we did not understand yet how. This is why you can "discover" zerodays - they were there all along. I prefer to know about the risks and design around them, than avoid the problem entirely (leaving them open to exploitation by bad actors). |
| |
| ▲ | BoppreH a day ago | parent | next [-] | | I myself work in the cryptography department of a large company, so I'm not arguing the semantics of cryptanalysis, or crypto agility, or patching. I simply meant "imagine if suddenly anyone could decrypt AES ciphertexts", and that society is not ready for such sudden disruption. It's like everyone suddenly developing telepathy. It's not dangerous per se, and we could adapt if done slowly enough, but all at once? Catastrophic. | | |
| ▲ | scotty79 17 hours ago | parent [-] | | Society is never ready for anything. It always progresses reactively only in response to technological advancement. | | |
| |
| ▲ | whimsicalism 14 hours ago | parent | prev [-] | | I'm not sure if I'm so comfortable with this dogma when it is applied to, say, our immune system. |
|
|
| ▲ | dcrazy a day ago | parent | prev | next [-] |
| The only reason nuclear nonproliferation works is because the inputs are relatively rare and traceable. (Yes U238 is abundant but you can’t mine enough of it or buy the centrifuges to turn it into U235 without getting noticed.) I have all the tools to develop my own ML training pipeline on my machine today. In order to choke off the inputs to “LLM proliferation,” we’d need to confiscate every computer, destroy all copies of the Internet, and throw out all books and other archived media. It’s not gonna happen. |
| |
| ▲ | uselessTA 3 hours ago | parent | next [-] | | RAND & others have published work about this, and they believe frontier model training would be much easier to track than nuclear weapons were You need a datacenter larger than a football field that draws the power of a small city to train a frontier model, and need chips that are produced by the most complicated supply chain in history (owned almost entirely by US allies like Taiwan & Europe). And that's before you consider on-chip mechanisms that seem entirely feasible. This is much easier to trace & much harder to reproduce than precision spinning containers. Though I agree seems much harder to detect people running models that were already trained. | |
| ▲ | BoppreH a day ago | parent | prev | next [-] | | > The only reason nuclear nonproliferation works is because the inputs are relatively rare and traceable. There are only three companies in the world capable of mass-producing the chips used in SOTA AI training, they all use machines from a single company (ASML), and it also requires enormous power-hungry datacenters. Meanwhile, uranium can be dug up from the ground. You say that buying centrifuges will get you noticed, but why can't we do the same for SOTA ML hardware? It honestly sounds easier to control. And differently from nuclear weapons, both sides are afraid of rogue AIs. This is a danger to our civilization, I'm sure we can figure something out. > I have all the tools to develop my own ML training pipeline on my machine today. Are you saying you can train the successor for Claude Opus 5.5 on your machine? Already today you can get on a list by buying too many smoke alarms due to the americium content, or drug precursors, or a thousand harmless but suspicious products. I would be ok with a world where you likewise need a justification to buy thousands of GPUs at once. | |
| ▲ | tomku a day ago | parent | prev [-] | | Maybe we just have very different imaginative worlds, but after seeing the kind of crazy shit that people will believe thanks to online radicalization, I'm a lot less sanguine about the possibility that people will take a shot at your list of horribles. There are a lot of people out there who seem to genuinely believe that what's at stake is the very survival of humanity and the continued existence of truth, art, beauty and all the other good things. After enough exposure to that kind of pressure-cooker radicalization, something like a repressive world government that uses violence to ensure that only the "right" people have access to dangerous things like literacy and computation could seem like a small price to pay for saving us all. Sure, it'd suck in a lot of ways, but Altman and Amodei will have left the brave revolutionaries no choice. I think it's very unlikely something like that would succeed, but I'm increasingly sure we're going to see serious attempts to do all of the things on your list, and they could do a lot of damage even in failure. |
|
|
| ▲ | doug_durham a day ago | parent | prev | next [-] |
| How is solving an abstract math problem related to having unlimited destructive powers? There is no relation. |
| |
| ▲ | BoppreH a day ago | parent [-] | | My post already gave three examples. If you have a problem with those and can't think of any more, I'll be happy to provide another dozen, it's not a hard exercise. Also, "unlimited" is your word, not mine. |
|
|
| ▲ | roywiggins 21 hours ago | parent | prev | next [-] |
| > pause development of models any bigger, enforced globally just like nuclear proliferation. How? American couldn't enforce its brief monopoly on nuclear weapons either. |
| |
| ▲ | reasonableklout 16 hours ago | parent [-] | | No, but we have successfully kept possession of nuclear weapons to 9 countries out of hundreds for decades. | | |
| ▲ | defrost 16 hours ago | parent [-] | | Possession as in ownership, nine. Possession as in have in their territory, fifteen. "Successfully kept possession (limited to)" .. seems a little over confident- some of these nations acquisitions and expansions occurred despite others not wanting that to occur. An additional twenty-eight countries have "virtual" nuclear weapons, having endorsed others to use actual nuclear weapons on their behalf. |
|
|
|
| ▲ | unknown-unknown a day ago | parent | prev | next [-] |
| John von Neumann helped Manhattan. https://www.youtube.com/watch?v=W06g7gIfwRE Similarly, now apply Fields medal level AI to R and D & hard deep tech problems. One month back, https://www.bloomberg.com/news/videos/2026-10-02/lockheed-ta... https://www.youtube.com/watch?v=JI7trhPHx3Y&list=PLXkles7kFx... "We all saw the headline that they were able to overcome the math problem for the that researchers had worked on for years and we have some math and physics challenges as we employ this next sensor capability on the F-35 and OpenAI is working next to us to uncover some of those math and physical challenges." |
| |
| ▲ | BoppreH a day ago | parent [-] | | Yeah, that's how my fears start. I don't want to find out what happens to the global stage if one country develops autonomous killer drone swarms or something. The militaries might be salivating at the opportunity, but I hope that the risk of rogue AIs brings us all to the same side on pausing development of bigger AI models. |
|
|
| ▲ | ZetaReverie a day ago | parent | prev | next [-] |
| We should try to stop it as hard as we can, but in full honestly, I don't think we ever could have. I think any intelligent species that doesn't go extinct is fated to reach this point from the moment it awakens... |
|
| ▲ | staplehorse6553 13 hours ago | parent | prev | next [-] |
| >What happens if a jackass uses the next AI to design a supervirus simple, just wear a mask, duh. or two. three tops, for maximum efficiency. |
|
| ▲ | dthakur a day ago | parent | prev | next [-] |
| Genie is out of the bottle. |
| |
| ▲ | BoppreH a day ago | parent [-] | | A genie is out of the bottle, but it's weak enough that the consequences so far have been limited to job loss and medium-sized IT problems. The creation of better models is not inevitable, it's done by human engineers using immense hardware, and therefore can be stopped before things get worse. |
|
|
| ▲ | postalrat a day ago | parent | prev | next [-] |
| Go ahead and stop. Others won't. |
|
| ▲ | potsandpans 21 hours ago | parent | prev | next [-] |
| Who gets the keys to the world ending demon? Dario amodei? Sam altman? Donald Trump? |
|
| ▲ | thejokeisonme a day ago | parent | prev [-] |
| This one person isn't a jackass. Sorry for the quibble. |
| |
| ▲ | BoppreH a day ago | parent [-] | | I didn't say he was a jackass. Also, to preempt another comment: yes, I know that he's a great mathematician. But we've had great mathematicians before, and even working together for 150 years they were not able to prove Riemann. And as models get better, the pedigree required for groundbreaking results (including dangerous ones) will naturally lower. |
|