I agree with this. I basically don’t feel comfortable trying to pretend a fancy matrix multiplication can be responsible for anything.

But what if companies that don’t track responsibility outcompete those who do?

In particular what if some perfect AI decision making ends up nailing decisions that maximize the expected reward for the company. And, well, if that comes at the cost of some unmanaged low-probability catastrophic risk, the company doesn’t care because all the decision makers are AI that don’t mind being shut off.