Quite the opposite. You're looking for a harmonious blend of flavors and textures which i does indeed tend to be the ~average, precisely because outliers are less likely to be harmonious. The average is not arbitrary at all. It's more like a "wisdom of the crowds".