simonw, your pelicans are so famous now that the frontier labs can just RLVR for SVG images explicitly.

It's a simple manner of prompt...generate concept art with nanobanana, measure, llm creates svg with text, measure the distance between rasterized svg embedding to nanobanana art....do post-training RL on a 3 million such images and matching embeddings.

I swear this will be a canonical example of Goodhardt's Law in action.

You have to switch to a new benchmark that isn't so easy to run a verifiable rewards pipeline on...Perhaps a longer horizon task where it's easy to mess up midway.

▲

jameshart 8 hours ago | parent [-]

Why? Don’t you think the steady improvement in pelican bicycling SVG art is itself a worthy field of endeavor for LLMs to be optimized for?

Yes, Simon has made pelicans a benchmark and now models have grown very good at rendering them. Is there a reason to believe that this isn’t manifesting as useful skill in general in SVG generation?

	▲	wasfgwp 5 hours ago \| parent [-]
		The sample is too small. As long as the models are reasonable close enough to each other there is really nothing you can learn from this benchmark when its just one prompt. It would perhaps be slightly interesting if the prompt was run 100 times for each model. I doubt the pelicans would look identical.