Thing is though, I think that the argument of "we don't understand how it works or what it does" doesn't hold up - everything an LLM does is observable. And if it's scary, we turn it off. If it starts to act like a bacterium / virus (e.g. it becomes self-replicating), we've dealt with that kind of software before.

But more importantly, we need to keep the organizations building and running these things accountable. If a company like Anthropic or OpenAI builds software that "escapes containment", they should be held accountable for cybercrime, just like people who write viruses (there's probably a few that did that on here, and of those a percentage that got in trouble with the law over it).

Mark my words, the organizations that develop AIs will keep using phrases like "we don't know what it's doing" or "it's too dangerous" to try and avoid responsibility and accountability for their software. Just like Google did, hiding behind "the algorithm" for lawsuits about preferring their own services over that of the competition.

Completely agree. It's a choice not to observe and understand what's going on, at least a very risky one if not actively evil.

There's a tradeoff though... isn't it already super hard to understand what they're doing with these math proofs for example? Clearly there's a tradeoff between deeply understanding the behavior vs. quickly solving the concrete problem, hence the big controversy in math now. From one perspective, letting stuff run ~autonomously in a mad dash to solve your problem is a really bad idea, threatens to destroy the whole field etc. From another perspective, solving lots of problems fast is extremely enticing and worth some risks.

Update for one other thought:

> And if it's scary, we turn it off.

Agreed, that's why I think the bad scenarios are some form of "it does some weird stuff ostensibly while helping us solve some problem" followed by "oops, we miscalculated that threat level."