They grouped performance, not intelligence.
If you don't think that what happened and is happening to black people affects our average performance on tests, you haven't eliminated racist assumptions, you've eliminated the entire black experience in America and any reason for a grudge or complaint.
Grouping a list of test performances by race shouldn't lead to an assumption of natural stupidity any more than grouping a list of incomes or wealth by race should lead to an assumption of natural laziness.
And the fact that you jump to that conclusion, and it makes you nervous, so you think it's bad to talk about the facts that remind you of that conclusion, does not help black people.
You could say the same thing about Hispanic economic migrants. You don't think that economic migrants from countries with good universities and healthy middle classes are going to score lower on tests? The ones that fly in are a different story, they're often elites, but the ones that walk in? They're bottom caste where they come from, and looking for a chance to change things.