In my experience, most of the time estimates are difficult because prerequisites and constraints are not clear or not fully known.

But, if you are enough experienced, with some fermi-style math, you will do good enough estimates most of the time.

Giving an estimate is not hard and shouldn't be a pain if if was truly an "estimate" PMs were asking for. But after a while you learn that when they say "What's the estimate for feature F?", they're in fact saying "What deadline you wish to impose on yourself for feature F?". Then the next interaction is most likely to be "Are you done yet?" and not "Is the estimate holding up?".