I would think that for training Google has optimized their entire setup for TPUs so I can’t see how this would be used for anything but inference

That's a good point. That makes it less likely that it's training, but who knows.