This conversation grows more useless as AI grows more capable.

For example if you personally tell an AI to do what it thinks best and it blackmails some other person into giving it resources allowing the prompt to escape your instance and run wild on the internet causing billions of dollars in damages, could you possibly think that the idea of responsibility is a bit broken.

For example we don't give your average libertarian weapon grade plutonium now matter how much they scream about their god given rights because it is a clear and present danger to humanity. That's where we are getting to with more advanced models. They go from being a tool to a munition with agency. Most SOTA models are good enough to deceive their users, especially not technical ones in doing things they don't understand the ramifications of.

AI is not a normal technology. As long as we treat it like it is, we'll continue to make the wrong analogies.

I fully agree that it’s plausible for an AI to do things that are outside of the purview of the users intentions and be very dangerous and capable while doing so, however that fact alone doesn’t absolve me of responsibility for instantiating the AI that did a bad thing if that AI would never have done the bad thing if it was never instantiated.

You could possibly move the blame higher to the manufacturer of the product, for example, the weapons grade plutonium you provided, it doesn’t exist unless you take intentional actions to make it so, and even when it does exist it doesn’t nuke a city unless negligence or intention is applied, in both those cases the fault lies in the initial operator. We don’t blame split atoms for the chain reaction caused.

Now, if an agent decided to spontaneously and maliciously act in a way to cause harm that is in direct contradiction to the initial intent, then yeah it would totally be the AI’s fault, however I don’t think we have seen that yet (I’ll change my opinion if I’m wrong here) and until we do I can’t place blame on the machine.