> A language model is made to predict language

<pedantry>Isn't a language model made to predict the next token in a series, which just so happens to be good for predicting not only natural languages, but also formal ones (code and math)?</pedantry>

Also, similar to what nelox said, as long as language (or sequences of tokens or what have you) can be "about" something (whatever that means), then it's possible that LLMs are encoding information about that "something". I'm being deliberately vague because I think that trying to be precise (by e.g. referring to latent spaces and so on) makes it sound like we've figured something out when in reality we haven't even found the right words to ask the questions.