If you can't trust a tool, you shouldn't be running it at all. It's really quite simple. It doesn't matter how useful it is if you can't actually have confidence in using it safely.

I don't think it's about trust but rather incomplete evaluation. Evaluating the model on its capacity to refuse a task or to question its prompt is something recent when you look at it, i feel current AI is really just an immature solution and we are just yet realizing the mistakes that have been made for so long

People will use the tool regardless. so it’s a race to try to make it safe before something truely bad happens.

And yet we we all use human written software even though we can be confident that the next severe software vulnerability to be found in it is just round the corner.