Indeed it's rude, but since I suddenly found myself unplannedly having to use an iPhone I found that's it's hard to position the speaker so that my ear can actually hear the other person.. I have to carefully keep micro-adjusting the position until I can actually hear something, and then ask the person to start from the beginning. Speakerphone fixes that. Not that I would ever do that in public though.