You're using the fact the both parts of training affect the same weights to support your argument that they're making the system do something fundamentally different after RL?

Assuming you are saying that RL is changing the model from doing one thing to another, yes. RL is changing the nature of the model.