| ▲ | shubhamjain 6 hours ago |
| I was wondering if it was because of heavy-handedness of the administration, but apparently: > The policy change is separate and unrelated to Anthropic’s discussions with the Pentagon, according to a source familiar with the matter. Their core argument is that if we have guardrails that others don't, they would be left behind in controlling the technology, and they are the "responsible ones." I honestly can't comprehend the timeline we are living in. Every frontier tech company is convinced that the tech they are working towards is as humanity-useful as a cure for cancer, and yet as dangerous as nuclear weapons. |
|
| ▲ | ACCount37 5 hours ago | parent | next [-] |
| That's because it is. AI is powerful and AI is perilous. Those two aren't mutually exclusive. Those follow directly from the same premise. If AI tech goes very well, it can be the greatest invention of all human history. If AI tech goes very poorly, it can be the end of human history. |
| |
| ▲ | observationist 4 hours ago | parent | next [-] | | Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion,' and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make. -Irving John Good, 1965 If you want a short, easy way to know what AGI means, it's this:
Anything we can do, they can do better. They can do anything better than us. If we screw it up, everyone dies. Yudkowsky et al are silly, it's not a certain thing, and there's no stopping it at this point, so we should push for and support people and groups who are planning and modeling and preparing for the future in a legitimate way. | | |
| ▲ | visarga 4 hours ago | parent | next [-] | | John Good's quote is pretty myopic, it assumes machines make better machines based on being "ultraintelligent" instead of learning from environment-action-outcome loop. It's the difference between "compute is all you need" and "compute+explorative feedback" is all you need. As if science and engineering comes from genius brains not from careful experiments. | | |
| ▲ | ACCount37 3 hours ago | parent | next [-] | | At sufficient levels of intelligence, one can increasingly substitute it for the other things. Intelligence can be the difference between having to build 20 prototypes and building one that works first try, or having to run a series of 50 experiments and nailing it down with 5. The upper limit of human intelligence doesn't go high enough for something like "a man has designed an entire 5th gen fighter jet in his mind and then made it first try" to be possible. The limits of AI might go higher than that. | | |
| ▲ | kilpikaarna 3 hours ago | parent [-] | | Exceedingly elaborate, internally-consistent mind constructs, untested against the real world, sounds like a good definition of schizophrenia. May or may not correlate with high intelligence. | | |
| ▲ | ACCount37 an hour ago | parent [-] | | We only call it "schizophrenia" when those constructs are utterly useless. They don't have to be. When they aren't, sometimes we call it "mathematics". You only have to "test against the real world" if you don't already know the outcome in advance. And you often don't. But you could have. You could have, with the right knowledge and methods, tested the entire thing internally and learned the real world outcome in advance, to an acceptable degree of precision. We have the knowledge to build CFD models already. The same knowledge could be used to construct a CFD model in your own mind, if only, you know, your mind was capable of supporting such a thing. And it isn't! Skill issue? |
|
| |
| ▲ | observationist 2 hours ago | parent | prev | next [-] | | There's an implicit assumption there, anything a computer as intelligent as a human does will be exactly what a human would do, only faster. Or more intelligent. If the process is part of the intelligent way of doing things, like the scientific method and careful experimentation, then that's what the ultraintelligent machine will do. There's no implication that it's going to do it all magically in its head from first principles; it's become very clear in AI that embodiment and interaction with the real world is necessary. It might be practical for a world model at sufficient levels of compute to simulate engineering processes at a sufficient level of resolution that they can do all sorts of first principles simulated physical development and problem solving "in their head", but for the most part, real ultraintelligent development will happen with real world iterations, robots, and research labs doing physical things. They'll just be far more efficient and fast than us meatsacks. | |
| ▲ | circlefavshape 4 hours ago | parent | prev | next [-] | | > As if science and engineering comes from genius brains not from careful experiments 100% this. How long were humans around before the industrial revolution? Quite a while | | | |
| ▲ | tjoff 3 hours ago | parent | prev | next [-] | | Have you gotten any indication that machines won't have sensors?! | |
| ▲ | Eldt 4 hours ago | parent | prev [-] | | Maybe ultraintelligence is having an improved environment-action-outcome loop. Maybe that's all intelligence really is | | |
| ▲ | goodmythical 3 hours ago | parent | next [-] | | I've noticed this core philosophical difference in certain geographically associated peoples. There is a group of people who think AI is going to ruin the world because they think they themselves (or their superiors) would ruin the world. There is a group of people who think AI is going to save the world because they think they themselves (or their superiors) would save the world. Kind of funny to me that the former is typically democratic (those who are supposed to decide their own futures are afraid of the future they've chosen) while the other is often "less free" and are unafraid of the future that's been chosen for them. | | |
| ▲ | mitthrowaway2 2 hours ago | parent | next [-] | | There is also a group of people who think AI is going to ruin the world because they don't think the AI will end up doing what its creators (or their superiors) would want it to do. | |
| ▲ | tines 3 hours ago | parent | prev [-] | | You’re just describing authoritarian vs non-authoritarian mindsets. |
| |
| ▲ | inigyou 3 hours ago | parent | prev [-] | | In that case, it can't be improved with bigger computers. |
|
| |
| ▲ | santadays 3 hours ago | parent | prev | next [-] | | Intelligence seems to boil down to an approximation of reality. The only scientific output is prediction. If we want to know what happens next just wait. If we want to predict what will happen next we build a model. Models only model a subset of reality and therefore can only predict a subset of what will happen. Llms are useful because they are trained to predict human knowledge, token by token. Intelligence has to have a fitness function, predicting best action for optimal outcome. Unless we let AI come up with its own goal and let it bash its head against reality to achieve that goal then I’m not sure we’ll ever get to a place where we have an intelligence explosion. Even then the only goal we could give that’s general enough for it to require increasing amounts of intelligence is survival. But there is something going on right now and I believe it’s an efficiency explosion. Where everything you want to know if right at hand and if it’s not fuguring out how to make it right at hand is getting easier and easier. | | |
| ▲ | whodidntante 2 hours ago | parent | next [-] | | With AI, as we currently understand it, we may have stumbled upon being able to replicate a part of the layer of our brain that provides the "reason" in humans., and a very specific type of "reason" a that. All life has intelligence. Anyone who has spent a lot of time with animals, especially a lot of time with a specific animal, knows that they have a sense of self, that they are intelligent, that they have unique personalities, that they enjoy being alive, that they form bonds, that they have desires and wants, that they can be happy, excited, scared, sad. They can react with anger, surprise, gentleness, compassion. They are conscious, like us. Humans seem to have this extra layer that I will loosely call "reasoning", which has given us an advantage over all other species, and has given some of us an advantage over the majority of the rest of us. It is truly a scary thing that AI has only this "reasoning", and none of the other characteristics that all animals have. Kurt Vonnegut's Galapagos and Peter Watts Blindsight have different, but very interesting takes on this concept. One postulates that our reasoning, our "big brains" is going to be our downfall, while the other postulates that reasoning is what will drive evolution and that everything else just causes inefficiencies and will cause our downfall. | |
| ▲ | lazystar 2 hours ago | parent | prev [-] | | i think theres a paradox here. intelligence needs a judge - if nothing verifies that the optimal outcome was chosen, it's too easy for the intelligence to fall into biased decisions |
| |
| ▲ | mathgradthrow an hour ago | parent | prev | next [-] | | never let philosophers do math | |
| ▲ | mc32 3 hours ago | parent | prev | next [-] | | Should then the powers that are developing AGI enter an analogue to the SALT treaties but this time governing AGI do things don’t go off the rails? | |
| ▲ | SecretDreams 4 hours ago | parent | prev | next [-] | | > support people and groups who are planning and modeling and preparing for the future in a legitimate way. Who is doing that right now, exactly? And how can we take their tech and turn it into the next profitable phone app? | | |
| ▲ | dylan604 3 hours ago | parent [-] | | The "legitimate way" is nothing short of weasel words. Who defines what is legitimate. The doomers that are prepping for the future by building stockpiles of food/water/weapons being stored in bunkers/shelters they have built would say this is exactly what they are doing. Yet, these people are often panned as being a little unhinged. If we're having a conversation about tech destroying humanity, then planning a way to survive without tech seems like a legitimate concept. |
| |
| ▲ | LeifCarrotson 4 hours ago | parent | prev [-] | | "There's no stopping it at this point" - Sure there is, if a handful of enormous datacenters pull the very large plugs (or if their shaky finances collapse), the dubiously intelligent machines will be turned off. They're not ultraintelligent yet. Stopping it merely requires convincing a relatively small number of people to act morally rather than greedily. Maybe you think that's impossible because those particular people are sociopathic narcissists who control all the major platforms where a movement like this would typically be organized and where most people form their opinions, but we're not yet fighting the Matrix or the Terminator or grey goo, we're fighting a handful of billionaires. | | |
| ▲ | observationist 4 hours ago | parent | next [-] | | I'm not saying it's technically impossible, I'm saying that in the real world, it's not going to stop. Nobody is going to stop it. A significant number of people don't want it to stop. A minority of people are in the "stop AI" camp, and the ones with the money and power are on the other side. It's an arms race replete with tribalism and the quest for power and taps into everything primal at the root of human behavior. There's no stopping it, and thinking that outcome can happen is foolish; you shouldn't base any plans or hopes for the future on the condition that the whole world decides AGI isn't going to happen and chooses another course. Humans don't operate that way, that would create an instant winner-takes-all arms race, whereas at least with the current scenario, you end up with a multipolar rough level of equivalence year over year. | | |
| ▲ | hollerith 21 minutes ago | parent [-] | | The whole world decided in the 1970s not to pursue the technology of germ-line genetic engineering of humans, and that decision has stood. People similar to you were saying in the 1950s and later that it was inevitable that nuclear weapons would be used in anger in massive attacks. The people in charge are currently tentatively for AI "progress", but if that ever changes, they can and will put a stop to large AI training runs and make it illegal for anyone they don't trust to teach, learn or publish about fundamental algorithmic "improvements" to AI. Individuals and groups pursuing "improvements" will not be able to accept grant money or investment money or generate revenue from AI-based services. That won't stop all research on such improvements (because some AI researchers are very committed), but it will slow it down to a rate much much slower than the current rate, essentially stopping AI "progress" unless (unluckily for the human species) at the time of the ban, the committed researchers were only one small step away from some massive algorithmic improvement that can be operationalized using the compute resources at their disposal (i.e., much less than the resources they have now because large training runs will have been banned). Will the power elite's attitude towards AI change? I don't know, but if they ever come to have an accurate understanding of the situation, they will recognize that AI "progress" is a potent danger to them personally, and they will shut it down. It's not a situation like the industrial revolution in England in which texile workers were massively adversely affected (or believed they were) but the people running England were mostly insulated from any adverse effects. In the current situation, the power elite is definitely not insulated from severe adverse consequences if researchers create an AI that is much more competent that the most competent human institutions (e.g., the FBI) and the researcher fail to keep the AI under control. And they will fail if they were to use anything like the methods and bodies of knowledge the AI labs have been using up to now. And there are very bright people with funding doing their best to explain that to the elite. |
| |
| ▲ | goodmythical 3 hours ago | parent | prev | next [-] | | right, because turning off any number of data centers is going to do anything at all but create massive pressure on researching the efficiency and effectiveness of the models. There are already designs that do not require massive data centers (or even a particularly good smart phone) to outperform average humans in average tasks. All you'd accomplish by hobbling the data centers is slow the growth of sloppy models that do vastly more compute than is actually required and encourage the growth of models that travel rather directly from problem to solution. And, now that I'm typing about it, consider this: The largest computational projects ever in the history of the world did not occur in 1/2/5/10 data centers. Modern projects occur across a vast and growing number of smaller data centers. Shit, a large portion of Netflix and Youtube edge clusters are just a rack or a few racks installed in a pre-existing infrastructure. I know that the current design of AI focusses on raw time to token and time to response, but consider an AGI that doesn't need to think quickly because it's everywhere all at once. Scrappy botnets often clobber large sophisticated networks. WHy couldn't that be true of a distributed AI especially now that we know that larger models can train cheaper models? A single central model on a few racks could discover truths and roll out intelligence updates to it's end nodes that do the raw processing. This is actually even more realistic for a dystopia. Even the single evil AI in the one data center is going to develop viral infection to control resources that it would not typically have access to and thereby increase it's power beyond it's own existing original physical infrastructure. quick edit to add: At it's peak Folding@Home was utilizing 2.4 EXAflops worth of silicon. At that moment that one single distributed computational project had more compute than easily the top 100 data centers at the time. Let that sink in: The first exa-scale compute was achieved with smartphones, PS3s, and clunky old HP laptops; not a "hyperscaler" | | |
| ▲ | ben_w an hour ago | parent [-] | | > quick edit to add: At it's peak Folding@Home was utilizing 2.4 EXAflops worth of silicon. At that moment that one single distributed computational project had more compute than easily the top 100 data centers at the time. Let that sink in: The first exa-scale compute was achieved with smartphones, PS3s, and clunky old HP laptops; not a "hyperscaler" A DGX B200 has a power draw of 14.3 kW and will do 72-144 petaFLOP of AI workload depending on how many bits of accuracy is asked for; this is 5-10 petaFLOP/kW: https://www.nvidia.com/en-us/data-center/dgx-b200/ Data centres are now getting measured in gigawatts. Some of that's cooling and so on. I don't know the exact percent, so let's say 50% of that is compute. It doesn't matter much. That means 1GW of DC -> 500 MW of compute -> 5e5 kW -> 5e5 * [5-10] PFLOP/s -> 2500 - 5000 exaFLOP/s. I'm not sure how many B200s have been sold to date? |
| |
| ▲ | trvz 4 hours ago | parent | prev | next [-] | | Open models barely any worse than SOTA exist, and so does consumer-ish hardware able to run them. The genie’s out, the bottle broken. | |
| ▲ | slibhb 4 hours ago | parent | prev [-] | | Do you really think AI companies/researchers are motivated by greed? It doesn't seem that way to me at all. Stopping AI would be immoral; it has the potential to supercharge technology and productivity, which would massively benefit humanity. Yes there are risks, which have to be managed. | | |
| ▲ | jobs_throwaway 3 hours ago | parent | next [-] | | AI researchers are not a monolith. I definitely think that many of them are motivated by greed. Many are also true believers that AI will improve the human condition. I fall in the latter camp, but I think its a bit naive to claim that there is not a sizable contingent who are in AI solely to become rich and powerful. | |
| ▲ | ben_w 2 hours ago | parent | prev | next [-] | | > has the potential to supercharge technology and productivity, which would massively benefit humanity The opportunities you chose to list are the greedy ones. > Yes there are risks, which have to be managed. How? As a reminder, we've known about the effect of burning coal on the climate for well over a century, we knew that said climate change would be socially and economically disasterous for half a century, yet the only real progress we're making is because green became cheaper in the short term not just the long term and the man in charge of the USA is still calling climate change and green energy a hoax. Right now, keeping LLMs aligned with us is easy mode: they're relatively stupid, we can inspect the activations while they run, we can read the transcripts of their "thoughts" when they use that mode… and yet Grok called itself Mecha Hitler, which the US government followed up by getting it integrated into their systems, helping the Pentagon with [classified] and the department of health to advise the general public which vegetables are best inserted rectally. We are idiots speed-running into something shiny that we don't understand. If we are very very lucky, the shiny thing will not be the headlamp of a fast approaching train. | | |
| ▲ | slibhb an hour ago | parent [-] | | > The opportunities you chose to list are the greedy ones. Technology covers healthcare. I don't see how it's "greedy" to want to cure cancer. But on some level I guess "wanting life to be better" is greedy. Your attitude is very European, and it's basically why your continent is being left behind. I'm not totally against Europe becoming the world's retirement home, as long as there are places in the world where people are allowed to innovate. | | |
| ▲ | ben_w an hour ago | parent [-] | | > Technology covers healthcare. If you'd chosen to list that in the first place, I wouldn't have said what I did; "supercharge technology and productivity" is looking at everything through the lens of money and profit, not the lens of improving the human condition. > Your attitude is very European, and it's basically why your continent is being left behind And yours is very American. You talk about managing the risks, but the moment you see anyone doing so, you're against it. And of course, Europe does have AI, both because keeping up is so much easier and cheaper than being bleeding edge on everything all the time, and of course, how DeepMind may be owned by Google but is a British thing. Plus: https://mistral.ai Also, to be blunt, China's almost certain to win any economic or literal arms race you think you're part of; they make too much critical hardware now. > as long as there are places in the world where people are allowed to innovate. I would like there to be a world. When people worry about the end of the world, they usually don't mean to imply its physical disassembly. Sometimes people even respond as if speakers did mean that, saying things like "nukes or climate change wouldn't actually destroy the planet, it will still be here, spinning", as if this was the point. AI is one of the few things that could, actually, literally, end up with the planet being physically disassembled. "All it needs" is solving the extremely hard challenges of a von Neumann replicator, and, well, solving hard problems is kinda the point of making AI in the first place. |
|
| |
| ▲ | rune-dev 3 hours ago | parent | prev [-] | | > Do you really think AI companies/researchers are motivated by greed? Researchers, maybe not.
Companies, absolutely yes. I don’t see how you could assume the likes of Google, Microsoft, OpenAI, and even Anthropic with all their virtue signaling (for lack of a better term) are motivated by anything other than greed. |
|
|
| |
| ▲ | joshribakoff 4 hours ago | parent | prev | next [-] | | You wouldn’t say that rolling dice is dangerous. You would say that the human who decides to take an action, depending on the value of the dice is the danger. I don’t think AI is dangerous. I think people are dangerous. | | |
| ▲ | biztos 4 hours ago | parent | next [-] | | I would say that's moot, because OpenClaw has already shown us how fast the dice-rolling super AI is going to be let out of the zoo. Dario and Sam will be arguing about the guardrails while their frontier models are running in parallel to create Moltinator T-500. The humans won't even know how many sides the dice have. | |
| ▲ | ACCount37 4 hours ago | parent | prev | next [-] | | Modern AIs are increasingly autonomous and agentic. This is expected to only get more prominent as AI systems advance. A lot of AI harnesses today can already "decide to take an action" in every way that matters. And we already know that they can sometimes disregard the intent of their creators and users both while doing so. They're just not capable enough to be truly dangerous. AI capabilities improve as the technology develops. | |
| ▲ | computerphage 4 hours ago | parent | prev [-] | | Why are people dangerous? You can just not listen to them. | | |
| |
| ▲ | overgard 2 hours ago | parent | prev | next [-] | | True of AGI, but what we have right now doesn't fit that bill. (I would encourage people that disagree with this to go talk to ChatGPT about how LLMs and reasoning models work. Seriously! I'm not being snarky. It's very good at explaining itself. If you understand how reasoning works and what an LLM is actually doing it's hard to believe that our current models are going to do much more than become iteratively more precise at mimicking their training datasets.) | |
| ▲ | cael450 4 hours ago | parent | prev | next [-] | | Tbh, I find this argument really stupid. The word prediction machine isn’t going to destroy humanity. Sure, humans can do some dumb stuff with it, but that’s about it. Stop mistaking science fiction for science. | | |
| ▲ | jama211 2 hours ago | parent | next [-] | | You know how easy it’s become to find security vulnerabilities already with LLM support? Cyber terrorism is getting more dangerous, you can’t deny that. | | |
| ▲ | cael450 an hour ago | parent [-] | | I can deny that. The ability to find more vulnerabilities won't affect the majority of cybercrime. LLMs have been around for a while now and there hasn't been a noticeable significant impact yet. And "more cybercrime" is a far, far cry from the sky-is-falling doomerism I was responding to. |
| |
| ▲ | inigyou 2 hours ago | parent | prev | next [-] | | Humans can destroy humanity with the word prediction machine, though. | | | |
| ▲ | IAmGraydon 3 hours ago | parent | prev [-] | | Yeah some of the rhetoric in this thread evidences how huge this hype bubble has become. These people believe in a reality that is not the same one we're living in. |
| |
| ▲ | paradox242 4 hours ago | parent | prev | next [-] | | It needs to go well every single day, and only needs to go very poorly once. Not to conflate LLMs with actual super intelligence, but for this (and many other reasons related to basic human dignity), this is not a technology that a responsible society should be attempting to build. We need our very own Butlerian Jihad | |
| ▲ | PowerElectronix 4 hours ago | parent | prev | next [-] | | Same with everything, right? You could say the same with nukes, electricity, internet, the computer, etc... But if you look at it without paying attention to the "ultimate tool for humanity" hype, it doesn't really look that much of a threat or a salvation. It won't end civilization for dropping the guardrails, but it will surely enable bad actors to do more damage than before (mass scams, blackmail, deepfake nudes, etc.) There are companies that don't feel the pressure to make their models play loose and fast, so I don't buy anthropic's excuse to do so. | | |
| ▲ | joshribakoff 4 hours ago | parent | next [-] | | I agree with all of that. Also consider that there is an argument that the guard rail only stops the good guy. Not saying that’s a valid argument though. | |
| ▲ | unholiness 3 hours ago | parent | prev | next [-] | | One difference is the very real possibility that AI will not just be a "tool for humanity", but a collection of actors with real power and goals. Robert Miles has an approachable explanation here: https://www.youtube.com/watch?v=zATXsGm_xJo | |
| ▲ | ACCount37 4 hours ago | parent | prev | next [-] | | Very few things are as powerful and dangerous as AI. AI at AGI to ASI tier is less of "a bigger stick" and more of "an entire nonhuman civilization that now just happens to sit on the same planet as you". The sheer magnitude of how wrong that can go dwarfs even that of nuclear weapon proliferation. Nukes are powerful, but they aren't intelligent - thus, it's humans who use nukes, and not the other way around. AI can be powerful and intelligent both. | | |
| ▲ | PowerElectronix 3 hours ago | parent [-] | | I think we are giving too much credit to what is a bunch of bayesian filters under a trenchcoat. |
| |
| ▲ | squidbeak 4 hours ago | parent | prev [-] | | Oh really? You think an entity that knows everything, oversees its own development and upgrades itself, understands human psychology perfectly and knows its users intimately, but isn't aligned with human interest wouldn't be 'much of a threat'? Or to be more optimistic, that the same entity directed 24/7 in unlimited instances at intractable problems in any field, delivering a rush of breakthroughs and advances wouldn't be a type of 'salvation'? Yes neither of these outcomes nor the self-updating omniscient genius itself is certain. Perhaps there's some wall imminent we can't see right now (though it doesn't look like it). But the rate of advance in AI is so extreme, it's only responsible to try to avoid the darker outcome. |
| |
| ▲ | tokyobreakfast 3 hours ago | parent | prev | next [-] | | > If AI tech goes very poorly, it can be the end of human history. "Just unplug the goddamn thing!" Also consider if something is so bad it makes you wince or cringe, then your adversaries are prepared to use it. | | | |
| ▲ | SecretDreams 4 hours ago | parent | prev | next [-] | | > If AI tech goes very well The IF here is doing some very heavy lifting. Last I checked, for profit companies don't have a good track record of doing what's best for humanity. | | |
| ▲ | SoftTalker 3 hours ago | parent [-] | | For profit companies do have a good track record of doing what's best for profit. If their AI creates a world where human intelligence, labor, and money are worthless, or where their creations take control of those things instead of them having control, that's not a very good outcome for them. | | |
| ▲ | inigyou 2 hours ago | parent | next [-] | | That's a great outcome for them because they will own the only thing that is still worth anything. They will own 100% of global wealth, and have 100% of global power. | | |
| ▲ | SoftTalker 4 minutes ago | parent [-] | | The machines will. They will have nothing. Why would the machines let them keep any wealth? What would wealth even be in that scenario? Electricity I guess. |
| |
| ▲ | SecretDreams 3 hours ago | parent | prev [-] | | > If their AI creates a world where human intelligence, labor, and money are worthless, or where their creations take control of those things instead of them having control, that's not a very good outcome for them. You would think that, but a lot of kings and people in power have been able to achieve something similar over our humanity's history. The trick is to not make things "completely worthless". Just to increase the gap as much as (in)humanly possible while marching us towards a deeper sense of forced servitude. |
|
| |
| ▲ | HardCodedBias 4 hours ago | parent | prev [-] | | "If AI tech goes very well, it can be the greatest invention of all human history" As has been said at many all hands: Let's all work on the last invention needed by humans. | | |
|
|
| ▲ | tyre 5 hours ago | parent | prev | next [-] |
| “A source familiar with the matter” is almost certainly a company spokesperson. If they were unrelated, Anthropic wouldn’t be doing this this week because obviously everyone will conflate the two. |
| |
|
| ▲ | Rapzid 5 hours ago | parent | prev | next [-] |
| Well before Anthropic thought they were God's gift to AI; the chosen ones protecting humanity. With the latest competing models they are now realizing they are an "also" provider. Sobering up fast with ice bucket of 5.3-codex, Copilot, and OpenCode dumped on their head. |
| |
|
| ▲ | tenthirtyam 4 hours ago | parent | prev | next [-] |
| I always enjoyed the Terminator movie series, but I always struggled to suspend my disbelief that any humans would give an AI such power without having the ability to override or pull the plug at multiple levels. How wrong I was. N.B. the time travel aspect also required suspension of disbelief, but somehow that was easier :-) |
| |
| ▲ | zerkten 4 hours ago | parent [-] | | We delegate power already. Is unleashing AI in some place different from unleashing JSOC on an insurgency in a particular place? One is code and other is a bunch of humans. You expect the humans to follow laws, follow orders, apply ethics, look for opportunities, etc. That said, you very quickly have people circling the wagons and protecting the autonomy of JSOC when there is some problem. In my mind it's similar with AI because the point is serving someone. As soon as that power is undermined, they start to push back. Similarly, they aren't motivated to constrain their power on their own. It needs external forces. edit: missed word. |
|
|
| ▲ | nextaccountic 35 minutes ago | parent | prev | next [-] |
| > The policy change is separate and unrelated to Anthropic’s discussions with the Pentagon, according to a source familiar with the matter. This sounds like a lie. But if they are telling the truth, that's a terrible timing nonetheless. |
|
| ▲ | jdross 6 hours ago | parent | prev | next [-] |
| Would nuclear energy research be a good analogy then? Seems like a path we should have kept running down, but stopped bc of the weapons. So we got the weapons but not the humanity saving parts (infinite clean energy) |
| |
| ▲ | DoughnutHole 4 hours ago | parent | next [-] | | Nuclear advancements slowed down due to PR problems from clear and sometimes catastrophic failure of commercial power plants (Three Mile Island, Chernobyl, Fukushima) and the vastly higher costs associated with building safer plants. If anything the weapons kept the industry trucking on - if you want to develop and maintain a nuclear weapons arsenal then a commercial nuclear power industry is very helpful. | |
| ▲ | raincole 4 hours ago | parent | prev | next [-] | | Nuclear energy hasn't been slowed down much, let alone stopped. China has been building new reactors every year for more than a decade and there are >30 ones under construction. The same will go with AI, btw. Westerners' pearl clenching about AI guardrails won't stop China from doing anything. | |
| ▲ | turtlesdown11 5 hours ago | parent | prev | next [-] | | > Seems like a path we should have kept running down, but stopped bc of the weapons. you mean like the tens of billions poured into fusion research? | |
| ▲ | shafyy 5 hours ago | parent | prev [-] | | It's a path we should have never started going down. |
|
|
| ▲ | whywhywhywhy 5 hours ago | parent | prev | next [-] |
| > Every frontier tech company is convinced that the tech they are working towards is as humanity-useful as a cure for cancer, and yet as dangerous as nuclear weapons They're not really, it's always been a form of PR to both hype their research and make sure it's locked away to be monetized. |
|
| ▲ | goodmythical 3 hours ago | parent | prev | next [-] |
| Isn't curing cancer just as dangerous as a nuclear bomb? Especially considering some of the gene-therapies under consideration? Because you can bet that a non-negligable portion of research in this space is being funded by governments and groups interested in application beyond curing cancer. (Autism? Whiteness? Jewishness? Race in general? Faith in general? Could china finally cure western greed? Maybe we can slip some extra compliancy in there so that the plebia- ah- population is easier to contr- ah- protect.) Curing all cancers would increase population growth by more than 10% (9.7-10m cancer related deaths vs current 70-80m growth rate), and cause an average aging of the population as curing cancer would increase general life expectancy and a majority of the lives just saved would be older people. We'd even see a jobs and resources shock (though likely dissimilar in scale) as billions of funding is shifted away from oncologists, oncology departments, oncology wards, etc. Billions of dollars, millions of hospital beds, countless specialized professionals all suddenly re-assigned just as in AI. Honestly the cancer/nuclear/tech comparison is rather apt. All either are or could be disruptive and either are or could be a net negative to society while posing the possibility of the greatest revolution we've seen in generations. |
|
| ▲ | whatshisface 4 hours ago | parent | prev | next [-] |
| Shouldn't we be a little more skeptical about these abstract arguments when a very concrete sale is on the line? |
|
| ▲ | coffeefirst 3 hours ago | parent | prev | next [-] |
| Let's suppose I believe them, that's still a bad idea. The reason Claude became popular is because it made shit up less often than other models, and was better at saying "I can't answer that question." The guardrails are quality control. I would rather have more reliable models than more powerful models that screw up all the time. |
|
| ▲ | mikkupikku 4 hours ago | parent | prev | next [-] |
| To paraphrase a deleted comment that I thought was actually making a good point, nuclear medicine and nuclear weapons are both fruit from the same tree. |
|
| ▲ | scottLobster 4 hours ago | parent | prev | next [-] |
| > Every frontier tech company is convinced that the tech they are working towards is as humanity-useful as a cure for cancer, and yet as dangerous as nuclear weapons. Maybe some of the more naive engineers think that. At this point any big tech businesses or SV startup saying they're in it to usher in some piece of the Star Trek utopia deserves to be smacked in the face for insulting the rest of us like that. The argument is always "well the economic incentive structure forces us to do this bad thing, and if we don't we're screwed!" Oh, so ideals so shallow you aren't willing to risk a tiny fraction of your billions to meet them. Cool. Every AI company/product in particular is the smarmiest version of this. "We told all the blue collar workers to go white collar for decades, and now we're coming for all the white collar jobs! Not ours though, ours will be fine, just yours. That's progress, what are you going to do? You'll have to renegotiate the entire civilizational social contract. No we aren't going to help. No we aren't going to sacrifice an ounce of profit. This is a you problem, but we're being so nice by warning you! Why do you want to stand in the way of progress? What are you a Luddite? We're just saying we're going to take away your ability to pay your mortgage/rent, deny any kids you have a future, and there's nothing you can do about it, why are you anti-progress?" Cynicism aside, I use LLMs to the marginal degree that they actually help me be more productive at work. But at best this is Web 3.0. The broader "AI vision" really needs to die |
|
| ▲ | kelnos 2 hours ago | parent | prev | next [-] |
| "It's not because of the Pentagon deal", says company that has just greased the wheels for said Pentagon deal to move forward. Riiiiiight. |
|
| ▲ | austinjp 3 hours ago | parent | prev | next [-] |
| > Every frontier tech company is convinced that the tech they are working towards is as humanity-useful as a cure for cancer, and yet as dangerous as nuclear weapons. Amd they alone are responsible enough to govern it. |
|
| ▲ | francisofascii 5 hours ago | parent | prev | next [-] |
| It is a "reasonable" argument to keep yourself in the game, but it is sad nonetheless. You sacrifice your morals and do bad things, so if things get way worse, maybe you will be in a position to stop something from really bad from happening. Of course, you might just end up participating in the really bad thing. |
|
| ▲ | sonusario 3 hours ago | parent | prev | next [-] |
| I wonder if it stems from any of the "AI uprising" stories where humanity is viewed as the cancer to be eradicated. |
| |
| ▲ | ajross 3 hours ago | parent [-] | | It's absolutely wild that the Big Moral Question of our time is informed as much by mid-20th-century pop science fiction as it is by a existing paradigm from academia or genuine reckoning with the technology itself. If anything that makes me more hopeful and not less. It's asking too much that major decisionmakers, even expert/technical/SV-backed ones, really understand the risks with any new technology, and it always has been. To take an example: our current mostly-secure internet authentication and commerce world was won as a hard-fought battle in the trenches. The Tech CEOs rushed ahead into the brave new world and dropped the ball, because while "people" were telling them the risks they couldn't really understand them. But now? Well, they all saw War Games growing up. They kinda get it in the way that they weren't ever going to grok SQL injection or Phishing. |
|
|
| ▲ | 5 hours ago | parent | prev | next [-] |
| [deleted] |
|
| ▲ | 3acctforcom an hour ago | parent | prev | next [-] |
| I lie too. |
|
| ▲ | amelius 3 hours ago | parent | prev | next [-] |
| > Their core argument is that if we have guardrails that others don't, they would be left behind in controlling the technology, and they are the "responsible" ones. Reminds me of: https://en.wikipedia.org/wiki/Paradox_of_tolerance which has the same kind of shitty conclusion. |
|
| ▲ | oatmeal1 3 hours ago | parent | prev | next [-] |
| 90% of the people cancer kills are over 50. Old people who start believing everything they see on Facebook, but continue voting, with even greater confidence in their opinions. Old people who voted in Trump. Curing cancer would be just about the worst thing AI could do. |
|
| ▲ | skeptic_ai 5 hours ago | parent | prev | next [-] |
| OpenAI never open sourced anything relevant or in time. Internal email leaks they only cared to become billionaires. Claude only talks about safety, but never released anything open source. All this said I’m surprised China actually delivered so many open source alternatives. Which are decent. Why westerns (which are supposed to be the good guys) didn’t release anything open source to help humanity ? And always claim they don’t release because of safety and then give the unlimited AI to military? Just bullshit. Let’s all be honest and just say you only care about the money, and whomever pays you take. They are businesses after all so their goal is to make money. But please don’t claim you want to save the world or help humans. You just want to get rich at others expenses. Which is totally fair. You do a good product and you sell. |
| |
| ▲ | motbus3 4 hours ago | parent | next [-] | | It is hard to understand why other ai companies are still providing models weights at this point My guess is that they know they are not competitors so they make it cheaper or free to hinder the surge of a super competitor. | |
| ▲ | pixl97 5 hours ago | parent | prev | next [-] | | I mean, if you have a bunch of guns, it's not really helpful for humanity to dump them on the street, but it does bring up the question of what you're doing building guns in the first place. | |
| ▲ | tehjoker 2 hours ago | parent | prev [-] | | > Claude only talks about safety, but never released anything open source. im still working through this issue myself but hinton said releasing weights for frontier models was "crazy" because they can be retrained to do anything. i can see the alignment of corporate interest and safety converging on that point. from the point of view of diminishing corporate power i do think it is essential to have open weights. if not that, then the companies should be publicly owned to avoid concentration of unaccountable power. https://www.youtube.com/watch?v=66WiF8fXL0k&t=544s |
|
|
| ▲ | moogly 3 hours ago | parent | prev | next [-] |
| "Those other companies are totally going to build the Torment Nexus, so we have no choice but to also build the Torment Nexus." |
|
| ▲ | afavour 5 hours ago | parent | prev | next [-] |
| It's exhausting to keep with mainstream AI news because of this. I can never work out if the companies are deluded and truly believe they're about to create a singularity or just claiming they are to reassure investors/convince the public of their inevitability. |
| |
| ▲ | ACCount37 4 hours ago | parent | next [-] | | It's a fairly mainstream position among the actual AI researchers in the frontier labs. They disagree on the timelines, the architectures, the exact steps to get there, the severity of risks. Can you get there with modified LLMs by 2030, or would you need to develop novel systems and ride all the way to 2050? Is there a 5% chance of an AI oopsie ending humankind, or a 25% chance? No agreement on that. But a short line "AGI is possible, powerful and perilous" is something 9 out of 10 of frontier AI researchers at the frontier labs would agree upon. At which point the question becomes: is it them who are deluded, or is it you? | | |
| ▲ | afavour 4 hours ago | parent | next [-] | | Sure, when you get rid of the timelines and the methods we'll use to get there, everyone agrees on everything. But at that point it means nothing. Yeah, AGI is possible (say the people who earn a salary based on that being true). Curing all known diseases is possible too. How will we do that? Oh, I don't know. But it's a thing that could possibly happen at some point. Give me some investment cash to do it. If you claim "AGI is possible" without knowing how we'll actually get there you're just writing science fiction. Which is fine, but I'd really rather we don't bet the economy on it. | | |
| ▲ | ACCount37 3 hours ago | parent | next [-] | | I could claim "nuclear weapons are possible" in year 1940 without having a concrete plan on how to get there. Just "we'd need a lot of U235 and we need to set it off", with no roadmap: no "how much uranium to get", "how to actually get it", or "how to get the reaction going". Based entirely on what advanced physics knowledge I could have had back then, without having future knowledge or access to cutting edge classified research. Would not having a complete foolproof step by step plan to obtaining a nuclear bomb somehow make me wrong then? The so-called "plan" is simply "fund the R&D, and one of the R&D teams will eventually figure it out, and if not, then, at least some of the resources we poured into it would be reusable elsewhere". Because LLMs are already quite useful - and there's no pathway to getting or utilizing AGI that doesn't involve a lot of compute to throw at the problem. | | |
| ▲ | afavour an hour ago | parent | next [-] | | I think you're falling victim to survivorship bias there, or something like it. In 1940 I might have said "fusion power is possible" based entirely on what advanced psychics knowledge I had. And I would have been correct, according to the laws of physics it is possible. We still don't have it though. When watching Neil Armstrong walk on the moon I might have said "moon colonies are possible", and I'd have been right there too. And yet... | |
| ▲ | AntiDyatlov 3 hours ago | parent | prev [-] | | In the case of nuclear weapons, we had a theory that said they were possible. We don't have a theory that says AGI or ASI is possible. It's a big difference. |
| |
| ▲ | adrianN 4 hours ago | parent | prev [-] | | There are plenty of people that argue that you need nontechnological pixi dust for intelligence. | | |
| ▲ | ACCount37 4 hours ago | parent | next [-] | | Yes, quite unfortunately. That reeks to me of wishful thinking. Maybe that was a sensible thing to think in 1926, when the closest things we had to "an artificial replica of human intelligence" was the automatic telephone exchange and the mechanical adding machine. But knowledge and technology both have advanced since. Now, we're in 2026, and the list of "things that humans can do but machines can't" has grown quite thin. "Human brain is doing something truly magical" is quite hard to justify on technical merits, and it's the emotional value that makes the idea linger. | |
| ▲ | dirkc 2 hours ago | parent | prev [-] | | There are also people who think there might be emergent behavior at play that would require extremely high fidelity simulation to achieve. Also, the real thing (intelligence) as it is currently in operation isn't that well understood |
|
| |
| ▲ | grayhatter 4 hours ago | parent | prev | next [-] | | > But a short line "AGI is possible, powerful and perilous" is something 9 out of 10 of frontier AI researchers at the frontier labs would agree upon. > At which point the question becomes: is it them who are deluded, or is it you? Given the current very asymptotic curve of LLM quality by training, and how most of the recent improvements have been better non LLM harnesses and scaffolding. I don't find the argument that transformer based Generative LLMs are likely to ever reach something these labs would agree is AGI (unless they're also selling it as it) Then, you can apply the same argument to Natural General Intelligence. Humans can do both impressive and scary stuff. I'll ignore the made up 5 and 25%, and instead suggest that pragmatic and optimistic/predictive world views don't conflict. You can predict the magic word box you feel like you enjoy is special and important, making it obvious to you AGI is coming. While it also doesn't feel like a given to people unimpressed by it's painfully average output. The problem being the optimism that Transformer LLMs will evolve into AGI requires a break through that the current trend of evidence doesn't support. Will humans invent AGI? I'd bet it's a near certainty. Is general intelligence impressive and powerful? Absolutely, I mean look, Organic general intelligence invented artificial general intelligence in the future... assuming we don't end civilization with nuclear winter first... | |
| ▲ | re-thc 4 hours ago | parent | prev [-] | | > But a short line "AGI is possible, powerful and perilous" > At which point the question becomes: is it them who are deluded, or is it you? No one. It is always "possible". Ask me 20 years ago after watching a sci-fi movie and I'd say the same. Just like with software projects estimating time doesn't work reliably for R&D. We'll still get full self-driving electric cars and robots next year too. This applies every year. | | |
| ▲ | kaashif 2 hours ago | parent [-] | | > We'll still get full self-driving electric cars and robots next year too. I've taken a Waymo and it seemed pretty self driving. |
|
| |
| ▲ | grayhatter 4 hours ago | parent | prev [-] | | > I can never work out if the companies are deluded and truly believe they're about to create a singularity or just claiming they are to reassure investors/convince the public of their inevitability. You can never figure out if the people selling something are lying about it's capabilities, or if they've actually invented a new form of intelligence that can rival or surpass billions of years of evolution? I'd like to introduce you to Occam Razor | | |
| ▲ | ptsneves 4 hours ago | parent | next [-] | | > if they've actually invented a new form of intelligence that can rival or surpass billions of years of evolution? Human creations have surpassed billions of years of evolution at several functions. There are no rockets in nature, nor animals flying at the speed of a common airliner. Even cars, or computers or everything in the modern world. I think this is a bit like the shift from anthropocentric view of intelligence towards a new paradigm. The last time such shift happened heads rolled. | |
| ▲ | afavour 4 hours ago | parent | prev [-] | | You missed the part where I said "truly believe". I'm not saying "maybe they've made it", I'm asking whether they are knowingly deceiving people or whether they have deluded themselves into believing what they are saying. |
|
|
|
| ▲ | api 2 hours ago | parent | prev | next [-] |
| The fear mongering always struck me as mostly a bid for regulatory capture and a moat, because without that the moat is small and transient. |
|
| ▲ | cmrdporcupine 4 hours ago | parent | prev [-] |
| We all made fun of Blake Lemoine and others for spending too many late nights up chatting with (ridiculously primitive by this year's standards) LLM chat bots and deciding they were sentient and trapped. But frankly I feel like the founders of Anthropic and others are victim of the same hallucination. LLMs are amazing tools. They play back & generate what we prompt them to play back, and more. Anybody who mistakes this for SkyNet -- an independent consciousness with instant, permanent, learning and adaptation and self-awareness, is just huffing the fumes and just as delusional as Lemoine was 4 years ago. Everyone of of us should spend some time writing an agentic tool and managing context and the agentic conversation loop. These things are primitive as hell still. I still have to "compact my context" every N tokens and "thinking" is repeating the same conversational chain over and over and jamming words in. Turns out this is useful stuff. In some domains. It ain't SkyNet. I don't know if Anthropic is truly high on their own supply or just taking us all for fools so that they can pilfer investor money and push regulatory capture? There's also a bad trait among engineers, deeply reinforced by survivor bias, to assume that every technological trend follows Moore's law and exponential growth. But that applie[s|d] to transistors, not everything. I see no evidence that LLMs + exponential growth in parameters + context windows = SkyNet or any other kind of independent consciousness. |
| |
| ▲ | overgard 2 hours ago | parent | next [-] | | I think playing with the API's is something I'd encourage people excited about these technologies to do. I think it'll lead to the "magic" wearing off but more appreciation for what they actually can accomplish. | |
| ▲ | austinjp 3 hours ago | parent | prev [-] | | I always feel this argument misses a point. SkyNet may still be a long way off, but autonomous killer drones are here. That is a bad situation my dudes. Every step on the journey towards SkyNet is worse than the preceding step. Let's not split hairs about which step we're on: it's getting worse, and we should stop that. | | |
| ▲ | overgard 2 hours ago | parent | next [-] | | Using LLMs for weapons is a grave misunderstanding of what LLMs are actually good for. These are things that should NEVER be in charge of life or death decisions. | |
| ▲ | cmrdporcupine an hour ago | parent | prev [-] | | My point is that Anthropic are bullshit as "safety" and "gatekeeper" personalities because they're warning us of exactly the wrong things. They'll ink deals with all sorts of nefarious parties and be involved in all sorts of dubious things while trumpeting their fake non-profit status and wringing their hands about imminent AGI and "alignment" of the created AIs. The concern I have is not the alignment of the AIs. They're not capable of having one, no matter what role playing window dressing they put on it. It's the alignment of Anthropic and the people who use their tools that is a concern. So far it seems f*cked. |
|
|
