I'm not denying there's a difference between the two groups, I'm denying that the difference is in phenomenal experience. If one group has a different way of describing their experience to others in general (e.g. perhaps because they interpret things much more literally than others), then they would likely describe specific things differently.

A quick demonstration I sometimes use to tease out this distinction: “imagine a six-sided die. Are you imagining it? Good. How many dots are on the side closest to you?”

If the phenomenal experience was the same and the difference is in the description, everyone should still be able to give a number as their answer, even if they have to use different processes to derive the number from their different descriptions. But aphantasiacs will visibly struggle to answer this question, or give a non-number non-answer like “no side is closest to me” or “depends on which way the die is facing”.