Wouldn’t the racist claim be that because some group has a lower average IQ, they deserve fewer rights and freedom and less dignity?

The implication that if one group was shown to have a measurably lower intellect they are less “human” somehow is really worrying.

No, that would be ableist, or maybe eugenics.

And the way IQ is used in modern racist discourse is not this. Instead, the claims are "black people are not less well off on average because of racism in society, either old or current, they are less well off because they are dumber (have lower IQ) on average, so it's only a consequence of that, as dumber white people also tend to fail to get as much success". This is the argument being made, at least publicly, by most modern day scientific racists.

It is my understanding that it's only racist if the group somehow aligns with a race, but I don't know a good term for that belief when it doesn't align with races.

Is it racist, that we put gorillas in a zoo instead of a school because gorillas as a group have lower average IQ? Would we treat Gorillas this way if they had similar or higher average IQ?

What does it mean to be human?