This only works if there are negative consequences faced by the insured parties when things go wrong. If all the negative consequences are faced by society and there are no regulations that incur that burden on the companies building AI, then we'll have unchecked development.

We agree! Unchecked development could lead to disaster. Insurers can insist on adherence to best practices to incentivize safe practices. They can also clarify liability and cover most (but not all) of the risk, leaving the developer on the hook for a portion of it.