> Now if there's not enough room to pass safely and silently I completely slow to the pedestrians speed and THEN calmly say excuse me. But I'm convinced that there is just no universally correct way to do it.

Anyone who is mad that you politely passed them at a safe speed is just too sensitive about these things. You're totally fine there. But "room to pass safely and silently" could still piss people off depending on your speed and distance.