Hey author here. Your argument is completely valid, we only model physics implicitly and thus have no prove that the model "actually knows the physics". Practically, this might not matter much: If the model can predict the evolution of the system to a certain accuracy, the user won't care about the underlying knowledge. And even for modern physics (quantum / GR), we know we miss something and yet, the models we have are incredibly useful.

On a tangent, we cannot prove that LLMs actually know language, yet they can be incredibly useful. Of course, a true world model would be much nicer to have, I agree with that!

  > Practically, this might not matter much: If the model can predict the evolution of the system to a certain accuracyI'm 

  > the user won't care about the underlying knowledge. 

Read my example. People will care if you have a more complicated geocentric model. Geocentric was quite useful, but also quite wrong, distracting, and made many bad predictions as well as good ones.

The point is that it is wrong and this always bounds your model to being wrong. The big difference is if you don't extract the rules your model derived then you won't know when or how your model is wrong.

So yes, the user cares. Because the user cares about the results. This is all about the results...

  > we cannot prove that LLMs actually know language