Another aspect is contrast. We put such a lot of effort into getting adequate contrast between background and foreground, and then emoji skin tones destroy it.

On a light background, light skin tones are bad, lacking contrast between background and skin.

Dark skin tones are bad because they lack contrast between skin tone and other details in the emoji; and if on a dark background, dark contrast.

Yellow works well on near-white and near-black backgrounds.