Is there a danger of overfitting if you train something on a physics sim? How do you prevent the model to exploit the differences to real life? Surely there are some numerical errors and other idealizations that result in some stuff being a good solution but not working in real life, or is the sim that accurate?

This is an active area of research (look up papers that mention the "sim2real gap"). It is indeed a problem, but there are known strategies to deal with it.

You add variance to the sim parameters with the hope that the model learns to adapt to changes in environment