Although by "average" you usually refer to the mean, not the median.

In common English, I would say that "average" as used in the quote is understood to be something closer to mode, hence why the joke works, because it trades on the ambiguity of "average" and the difference between mode and mean.