Andre the Giant said he never worked out, he just wrestled. He had some kind of growth hormone disorder, if I recall.

Think about gorillas, who are pretty similar to us - they don't lift weights in the gym, do they?

Yeah, muscles are mostly about genetics, just like anything else. A mouse won't become a rhino by lifting. Humans are so incredibly genetically homogeneous that it can sometimes be tempting to ignore this, but even between humans the variability is quite large.