Does this mean in a general sense there are numbers that are harder to factor, or is it due to constraints? That some keys will be much harder to crack? If so, how can we know beforehand?

it's more that there are numbers that are easier to factor (e.g. 2^n-1). almost all numbers once you get past the tiny numbers are about the same.