Completely agree. It's a choice not to observe and understand what's going on, at least a very risky one if not actively evil.

There's a tradeoff though... isn't it already super hard to understand what they're doing with these math proofs for example? Clearly there's a tradeoff between deeply understanding the behavior vs. quickly solving the concrete problem, hence the big controversy in math now. From one perspective, letting stuff run ~autonomously in a mad dash to solve your problem is a really bad idea, threatens to destroy the whole field etc. From another perspective, solving lots of problems fast is extremely enticing and worth some risks.

Update for one other thought:

> And if it's scary, we turn it off.

Agreed, that's why I think the bad scenarios are some form of "it does some weird stuff ostensibly while helping us solve some problem" followed by "oops, we miscalculated that threat level."