I think we're talking about different kinds of models. I was referring to things like fluid dynamics equations that explain why gases and liquids move and how they act when changing states, as a basis for building weather models that predict how things will unfold in the future.

I'm also a fan of going the other direction: I've had a sideline working on code to evolve genetic algorithms for the past 20 years, and while the goal of that is to be predictive and profitable, it's often the underlying real-world dynamics my little mutants surface which are the most interesting and applicable in the long run. So I'm not saying there isn't a place for throwing everything at the wall until you see what sticks and then deriving a hypothesis from that (whether your interest is to predict the future, or merely academic, to explain the past). What I am saying is similar to you: We should not treat any model as an oracle. But I'm also saying that models can be built or they can be evolved, and if we only evolve them without understanding how they work, we are missing a crucial ingredient to knowing how well we should rank them. Overfitting and sample bias and data leakage are not problems when you want an equation to calculate airflow over a wing. If you began with an evolved equation which derived the results and didn't start from the base reality, you couldn't trust that equation to be airworthy even if it were right 99.99% of the time against the data it was trained on.