Yes, this has always bothered me. IQ doesn't easily correspond to any measurable real-world quality.

For example, if we would postulate that height is gaussian, we could measure people's heights and just ordering them we could create a gaussian distribution. Then we could verify the hypothesis of height being gaussian by mapping the probability distribution function's parameter to a linear value (cm) and find that these approaches line up experimentally.

We could do the same thing with any comparable quantity and make an order of them and try to map them to a gaussian distribution, but we would have no knowledge if what we were making actually corresponded to a linear quantity.

This is a serious issue, as basically making any claim like 'group A scores 5 points higher than group B' is automatically, mathematically invalid.