> What, exactly, is going to go horribly wrong here ...?

Operating anything at scale teaches you that the universe of “things that can go horribly wrong” has a much better imagination than you. That goes quadruple with complex and squishy reasoning models.

I’ve seen a couple of really good examples of what can go wrong in this subthread, the most straightforward of which is that abruptly changing a fundamental behavior can break existing use cases.