I feel like this is a dumb question, but why are we spending any effort on wine comparisons? Is there a way to get rid of experts on these to just be ones for code?

It's not like they're training on that. That's just a good showcase of how the model is good at generalizing & understanding (not benchmaxxed). You might want something similar to these experts for tasks like detecting anomalies or discoloration or whatever.

What if your coding project is related to wine? So you would want facts about wine to be inside the weights, not through web search.

Just looking through the benchmarks, the model seems to be very high on visual/spatial capabilities. So I assumed they just tried some hard tasks on it and was surprised by wine identification and thought it was surprising that it could identify wines.

you want your coding models to have good taste, so we have fed them a vast dataset of sommelier wine gradings.

we've also contracted hundreds of local record store guys with deeply knowledgeable music taste spanning post-punk, classic jazz, early hip-hop, and deep-cut experimental vinyl. This has resulted in the model returning judgmental responses up to 23% more often which we are working on addressing.