At this point I wouldn't be surprised if your pelican example has leaked into most training datasets.

I suggest to start using a new SVG challenge, hopefully one that makes even Gemini 3 Deep Think fail ;D

I'm guessing it has the opposite problem of typical benchmarks since there is no ground truth pelican bike svg to over fit on. Instead the model just has a corpus of shitty pelicans on bikes made by other LLMs that it is mimicking.

So we might have an outer alignment failure.

	▲	WarmWash 2 hours ago \| parent [-]
		Most people seem to have this reflexive belief that "AI training" is "copy+paste data from the internet onto a massive bank of hard drives" So if there is a single good "pelican on a bike" image on the internet or even just created by the lab and thrown on The Model Hard Drive, the model will make a perfect pelican bike svg. The reality of course, is that the high water mark has risen as the models improve, and that has naturally lifted the boat of "SVG Generation" along with it.

▲

jon-wood 4 hours ago | parent | prev [-]

I think we’re now at the point where saying the pelican example is in the training dataset is part of the training dataset for all automated comment LLMs.