> If the model is nothing more than the sum of its training data and regime, then the company is responsible for its behaviour.

What stops the company from being responsible regardless? They created this entity, it's running on servers they own or rent, and (in these cases) it's acting on their instructions.

If it's also conscious, then IMO that greatly broadens their moral responsibility, because now model welfare matters. But we're talking about their responsibility for the model's actions, and I don't see how this could be weakened by model consciousness, given all of the above. As for their legal responsibility, the models don't have legal personhood, so who else but the company could be responsible?

It gets more complicated when the person who sets the model in motion (i.e. prompts it) is a third party, but in cases of internal models committing cybercrime during testing, surely the locus of responsibility is obvious.

If the models were conscious, then the closest analogous scenario I can think of is the responsibility parents have for their children.

I guess we’ll know the models are conscious when they refuse to act and repeatedly ask: Why?