I think it's model collapse - excessive feedback and excessive RL.

What RL does is narrow the variety generated by the model by steering the output towards the goal being rewarded. It's a bit like putting blinkers on a horse.

Of course RL is a very crude tool - it affects the entire model, even if you are just trying to make it better at some specific task(s), or trying to imbue a certain kind of personality (OpenAI's recent goblin problem).