It is mumbo jumbo (also people hearing “significant” treat this as “effect/change is large and important” which is in no relation to the actual amount of change).

Low p-value basically means how surprising your data would be if there were actually no effect (ie less than 5% of the time you’ll get this due to randomness if there is no change — which is rather impossible)

Sample size matters heavily. With more observations, estimates become more precise, so increasingly small differences can become statistically significant. With a large sample, you can therefore get a tiny, practically meaningless effect with a very small p-value.

Eg effect of $1 can be statistically significant (not random) which does not matter in practical terms if average is like $10000.

So the key point here is not only to look at the p-value but also at an actual change. If a drug gives you only 0.01% more hair, it doesn’t matter to you that it is guaranteed.

One of the really awkward points of stats is that not only does sample size matter, but also model specification. Very small misspecifications can easily lead to infinitesimal P values over large sample sizes.

Similar things are true of Bayesian stats, leading to things like predictively oriented posteriors being studied nowadays.