> the pelicans used to be an unrecognisable mess and now they’re unquestionably pelicans on bicycles, rendered poorly, from every model

You would not expect that to happen if the models trained on the unrecognizable mess, right?

> model capabilities across the board have only meaningfully improved in places where the labs are focusing their training efforts

And the labs clearly did focus on improving image rendering.

> they have a uniform style

SVG output from LLMs always looks like that. It looked that way from the beginning; no LLM ever produced a watercolor when asked for SVG output. They all render the prompted element centered in the picture. They all tend to draw things going from left to right, and so on.

I’m not suggesting Simon’s pelicans in the dataset are having a meaningful impact. I’m expecting that a company like ScaleAI has a product along the lines of “benchmax dataset: SimonW’s Pelican on Bikes test” which is a private curated series of well-drawn SVGs of animals riding vehicles for training and RL.