The mental model here is that AI is a powerful tool whose main purpose is to outcompete others, increase power or provide a defense. This is not the right model!

The biggest risk is not that someone discovered a big new weapon, which by the way

> No one person, or one company, knows the answer and history is no guide

are you kidding me? It's a problem as old as time.

This is not nuclear power though. The bigger problem is a scenario where AI increasingly acts like bacteria, doing things autonomously, things no one really understands, things we think are confusing and weird. And then just by chance, some bacteria end up in some human lung cells and they turn out to be an excellent place to grow and multiply. Why is this worse? Because weird shit does not seem threatening, and it almost always isn't, so nobody tries to stop it.

Also, the argument in the article seems to be "obviously AI development is not a particularly urgent global threat," and therefore these downsides (risk of collusion, misuse of state power, etc) are the most critical factors? I guess, uh, yeah, if there's no benefit to taking these actions we shouldn't take them.

▲

Cthulhu_ 2 hours ago | parent [-]

Thing is though, I think that the argument of "we don't understand how it works or what it does" doesn't hold up - everything an LLM does is observable. And if it's scary, we turn it off. If it starts to act like a bacterium / virus (e.g. it becomes self-replicating), we've dealt with that kind of software before.

But more importantly, we need to keep the organizations building and running these things accountable. If a company like Anthropic or OpenAI builds software that "escapes containment", they should be held accountable for cybercrime, just like people who write viruses (there's probably a few that did that on here, and of those a percentage that got in trouble with the law over it).

Mark my words, the organizations that develop AIs will keep using phrases like "we don't know what it's doing" or "it's too dangerous" to try and avoid responsibility and accountability for their software. Just like Google did, hiding behind "the algorithm" for lawsuits about preferring their own services over that of the competition.

	▲	ngriffiths 33 minutes ago \| parent [-]
		Completely agree. It's a choice not to observe and understand what's going on, at least a very risky one if not actively evil. There's a tradeoff though... isn't it already super hard to understand what they're doing with these math proofs for example? Clearly there's a tradeoff between deeply understanding the behavior vs. quickly solving the concrete problem, hence the big controversy in math now. From one perspective, letting stuff run ~autonomously in a mad dash to solve your problem is a really bad idea, threatens to destroy the whole field etc. From another perspective, solving lots of problems fast is extremely enticing and worth some risks. Update for one other thought: > And if it's scary, we turn it off. Agreed, that's why I think the bad scenarios are some form of "it does some weird stuff ostensibly while helping us solve some problem" followed by "oops, we miscalculated that threat level."