> I thought the whole point was if it did exist the motion goes faster in one direction than the other.

No, the idea was that, in a space filled with the hypothetical ether, Earth's velocity through the ether should have been detectable by comparing light beams traveling in different directions.

The null result was very important -- it didn't prove the absence of an ether, it only showed that it wasn't a factor in light propagation.