RL causes distributional collapse, it's how the models get consistent. Anyone who generated images with early gen (SD1.5-2) models will remember the wild variance between seeds, which newer models have mostly lost, and similarly GPT3.5/4 could produce weirder, more original outputs even if they were less consistently "good" in some sense.

It's worth mentioning that they do RL for aesthetics to some degree based on human expert feedback, but whatever the model tends to produce quickly becomes debased by its ubiquity. They could RL for output diversity, but it's less well studied and likely to cause minor regressions in coding performance, at least until the algorithms are dialed in.