transformers were first an image understanding technique, the text processing came later, it's all about the training data and gradient descent, and allegedly attention

It's actually the other way round - the Transformer architecture was introduced for text (machine translation) in "Attention Is All You Need" (2017). Vision Transformers, which apply it to images, came three years later in 2020: https://arxiv.org/abs/2010.11929