That has nothing to do with religion, beliefs or any validity of those. Just a phrase burned too deep into cranium over decades to dispose of it easily even if wanted. 'What the fuck' or other fuck variants are in same category.

I took it to mean that religious and non-religious scientists are drawn by the same sense of wonder at the cosmos.

I disagree with that. The phrase "Oh my God" expresses the surprise that there is something way larger, deeper and/or more powerful than the speaker imagined; this is the textbook definition of what God is.