I'd go a step further and say the ball/puck is not the interesting thing to watch.

Imagine if you couldn't see the players, and just saw the puck. Would that be interesting at all?

Think about tennis. There is the trope of people's eyes going back and forth following the ball, but I don't think they are following the ball directly. They are going back and forth looking at the person who is going to hit the ball.

I think you might be conflating knowing where the puck is with being able to fix your eyes on the puck at all times. The complaint is usually about the former. People are complaining that they don't know where the puck is.