Some adults indeed can't do it, but that doesn't mean it's difficult.

And it is certainly not "wear out a whole axe handle just to learn to swing" difficult.

You don't wear it out. You land the head long of your aim point, and splinter the handle on whatever you were trying to hit. It's certainly not hard to ruin a handle if you're learning to swing a sledge by driving steel splitting wedges.