it's not clear to me how they determined that these were in fact different species and not just variation/evolution within the same species

When biologists, botanists, etc. use the term "species", they are not using it in a principled way. The distinctions are oftentimes quite arbitrary, I'm very sorry to say.

For what it's worth, there isn't even much agreement on what constitutes a genus or species among professional philosophers of science (i.e. academics who care about giving a principled definition for species/genus), even for those that share many of the same views otherwise (like ur-Platonists, Aristotelians, Thomists, Scotists, etc.).

This a broader fundamental question which you are better off asking AI to give you an overview. It involves various techniques like;

1) Paleo genomics i.e. DNA study of any surviving material.

2) Statistical study of Human Morphology i.e. variation w.r.t. current day human (and close primates?) biological structures. Eg. Current-day human cranium sizes vary within known limits. So if we find a fossil cranium which is outside these limits then there is a high chance it might be a different species.

3) Fossil dating shows different species (w.r.t. above data) occupying the same geography and within overlapping time periods.