I always see SP a combination of time and risk. I think a lot of people do not include risk in the estimate.

So a story might be estimated at 3SP to implement but there's a high risk that it would blow out (e.g. idea was not fully proven in a PoC, work is in an area that is historically underestimated, reliance on a different team, etc.), so we set it to 5SP to include that risk. Maybe 50% of the time it does get finished in what a normal 3SP would finish in, but at least we've covered the 50% of time it blows out.

IMO it's best to welcome the addition of a risk premium, because if you don't, it'll creep in anyway, just in a patchy and inconsistent manner.

Over time it becomes "priced in" to the moving average, which is good, assuming that employee instincts are generally valid.

Of course, if someone makes the mistake of trying to peg points to time, they're indirectly creating a kind of inflation: Yesterday's "just in case" premium should not become today's "everything goes well" baseline.