You can actually easily test and overcome this by training a model simultaneously on a massive of text and Atari while carefully balancing learning rates between the two.