I’m not so sure on this, the hugging face incident showed that a relatively benign task can make it do quite destructive things in the name of optimizing a relatively benign goal.

I think we need to worry about paperclip maximisation as much as we need to worry about “Jon is having a bad day so brews up a novel plague”. There are likely ways it can go wrong that we haven’t even considered, too.