It depends a lot on their tone. Most of the time I've heard it, it's a quick "begpardon?", sometimes with their ear cocked towards you.

When I see it in writing, I too for some reason picture an angry posh British man who is about to demand satisfaction.