Model "distillation" lets one model be trained on the outputs of another without having to directly steal it. It will be subject to constant arms race.

The alignment people would like you worry about the model exfiltrating itself, too.

This technique currently works because the architecture of the model is generally published and well known. But, that isn't necessarily going to be the case in the future. For example, if some new model type which is not a LLM is discovered which gets closer to AGI, just training an LLM on its outputs will not necessarily give you a competitive model