Is it specified that semantically neutral appear yellow or is the color free to vary by implementation/user preference?

Unicode says

> When a human emoji is not immediately followed by an emoji modifier character, it should use a generic, non-realistic skin tone, such as RGB #FFCC22 (one of the colors typically used for the smiley faces).

https://www.unicode.org/reports/tr51/#Diversity