> But It might be the right way to teach it to physics students.

Having studied physics, I would disagree rather strongly. I only really started understanding Special Relativity once I had a clear understanding of the math. (And then it becomes almost trivial.) Those of my fellow class mates, however, who didn't take the time to take those additional (completely optional) math classes, ended up not really understanding it at all. They still got confused by what it all meant, by the different paradoxes, etc.

I saw the same effect when, later, I was a teaching assistant for a General Relativity class.