> What's stopping someone from just having the AI churn out garbage all day long?

In my case it's morality.

I would argue that making the company experience the consequences of its choice of metrics / mandates is in fact a moral imperative.

Interesting consideration, 'mandates' and all. Definitely in camp 'toss the output', here. I think I'll see 'morality' leaving when $EMPLOYER fires 'professional discretion'... forcing usage and, ultimately, debasing the position.

edit: Peer said it well, IMO. The consequences aren't really yours. Also: something, something, Goodhart's Law.