More realistically, a company that fails to properly evaluate this in ways that reflect actual market needs will fail in the marketplace.

The market used to be humans making choices. Why are you assuming it's not going to be flawed people using flawed AI? The rationality of the market was never a guarantee, and you think it's what's going to save us?