> I assumed numerical stability is not that beneficial compared to something precision heavy like physics simulation in HPC.

Yes, exactly.

For physics, there is a correct result. I.e. you want your simulation to reflect reality with high accuracy, over a long chain of calculations. Extremely tight constraint.

For deep learning, you don't have any specific constraints on parameters, except that you want to end up with a combination that fits the data well. There are innumerable combinations of parameter values that will do that, you just need to find one good enough combination.

Wildly different.