A bit tough to say this, but transformers are trained the same way.

No. They're trained on the data, but there's no point in training at which they go through some exercise that involves creating copies of some of their input using the model being trained.

In any case, why is it "a bit tough to say this"? You thought your ability to learn was irreproducible?