Remix.run Logo
lukev 9 hours ago

What if they’re not pelicanmaxxing, but svgmaxxxing in general?

Because otherwise using a LLM to generate complex svgs is pretty niche and what I thought made this a good benchmark when it was new - generalized programming and spatial knowledge.

Obviously image gen in svg format is not a particularly hard problem if tackled directly on its own.

qq66 8 hours ago | parent | next [-]

But that's a genuine worthwhile capability. It's like benchmarkmaxxing on a weightlifting competition by getting really strong.

throw10920 42 minutes ago | parent | next [-]

> benchmarkmaxxing on a weightlifting competition

If you stick to the benchpress, it's just "benchmaxxing".

Balgair 8 hours ago | parent | prev | next [-]

https://www.youtube.com/watch?v=jgYYOUC10aM

reminds me of this Key and Peele skit

ralusek 11 minutes ago | parent [-]

Alternatively Mitchell and Webb https://www.youtube.com/watch?v=_pDTiFkXgEE

sandcat_ 3 hours ago | parent | prev | next [-]

Yes, but the original purpose of the benchmark (simonw, please correct me if I'm wrong!) was to test whether new models were good at novel problem solving. Things they haven't been trained on. So yes, getting better at generating SVGs is great news (and it seems they have been) but this particular benchmark still strikes me as largely worthless now, unless SVGs happen to be what you care about in particular when a new model drops.

Instead, it feels like a more appropriate benchmark for the original purpose would be to come up with new, novel problems each time, and compare across all models (including previous ones).

Throwing my hat in the ring: Generate a pelican shaped crossword where all the clues are related to bicycles.

(Haiku 4.5: https://imgur.com/a/N112Nxo, I'm trying some others but it's very slow! Opus has been at it for about 20 minutes.)

TomatoCo 3 hours ago | parent [-]

I've mostly been running LLMs on my own hardware so the phrase "been at it for about 20 minutes" gave me pause. My first instinct was to ask "but on what hardware?" but I suppose one advantage of closed models is that there's a somewhat-consistent cognitive-effort-to-time ratio.

sandcat_ 3 hours ago | parent [-]

Fair, I don’t know if this helps but it was Opus 4.8 (high effort) via Claude Desktop, but through my org’s LLM gateway so hard to compare really. But it’s usually pretty quick.

(don’t tell my boss.)

ryukoposting 8 hours ago | parent | prev | next [-]

How useful actually is this? It generates SVGs of pelicans on bicycles, sure, and some of them are (almost) spatially correct. But, none of them look good.

AI image generation suffers from this more generally. You can generate pictures of pelicans, sure. Newer models clearly generate images with more pelican-ness than before. But all of it is still uglier than sin. Drawing things accurately is one thing, making results that someone might actually want to use (without embarrassing themselves) is something else.

derektank 7 hours ago | parent | next [-]

Just today I asked Claude Fable to take an SVG map and add a circle with a 150 mile radius around a specific city, cutting off the circle at the edge of certain boundaries. It came back a minute later with more or less exactly what I wanted, saving me maybe 5 to 10 minutes of photoshop time. It’s almost certainly not perfect, but I didn’t need it to be perfect, just a legible representation for a group of 20ish people.

cmckn 7 hours ago | parent | prev | next [-]

I used Claude to plan some landscaping, and it gave me SVG diagrams of flower beds showing where to place plants and their approximate mature size. It was pretty useful.

Kamshak 8 hours ago | parent | prev | next [-]

Great for charts, illustrations etc, can do much more with SVG than pretty pictures. Good spatial understanding in an SVG can be really nice for model to have

SirHackalot 5 hours ago | parent | prev | next [-]

I used it to generate some icons for my app recently, I found it a pleasantly useful feature already in its current state. Before this I would’ve had to close the code editor, open something like Illustrator, use something like a tablet (or struggle with the mouse and keyboard), and export the result. For something like a generic icon for a button, or a simple geometric figure, this always felt too tedious. On the other hand, my apps looked visually impressive before the AI and people valued my artistic ability more. So there’s that… Not every software engineer also had an eye for design. With AI generation, they can bridge that gap.

fiddlerwoaroof 8 hours ago | parent | prev | next [-]

AI can generate a fairly satisfactory SVG for a favicon now (programmer art quality at least).

zahlman 8 hours ago | parent | prev | next [-]

Even if the models don't break through any particular "uglier than sin" barrier, with a bit more work, presumably the SVGs could become importable into an editor that would let a human apply taste and discretion. Seems to me like a heck of a head-start.

As for conventional diffusion-model stuff, I happen to think there are some pieces of AI art that still look really good even knowing they're AI.

joshmoody24 4 hours ago | parent | prev | next [-]

Would it be useful if the models were actually good at it?

amarant 8 hours ago | parent | prev [-]

Vibecoding a SVG based metroidvania as we speak! This is gonna be lit!

OutOfHere 7 hours ago | parent | prev | next [-]

The fear is that the SVGmaxxing is limited to "X doing Y". If such 'template maxxing' exists, it will break for other templates, e.g. "X not doing Y", "X and Y doing Z", "X doing Y doing Z", etc.

nl 3 hours ago | parent | next [-]

I use LLMs for 3D CAD design in OpenSCAD.

There seems to be a very strong correlation between models that are good at SVG and models that are good at 3D CAD.

Anecdote I know, but there does seem to be generalization going on here.

zarzavat 4 hours ago | parent | prev [-]

If a model can improve at drawing "X doing Y" and that prompt wasn't in the training set then it means it has improved its internal mapping from text-to-spatial-to-text.

sysguest 8 hours ago | parent | prev [-]

well that holds IF svgmaxxing is 100% "code-writing-maxxing"

...which.. hmm I dunno if they are same or not

tsimionescu 8 hours ago | parent | next [-]

No, the point is that a general-ish ability to draw good SVGs is a useful ability in itself. People need SVGs for all sorts of purposes, and if AI can generate one for them, that's mostly useful (discussions about art and employment etc notwithstanding).

That said, I think this would correlate relatively little with general programming ability. They're not unrelated, of course, but being able to generate code that paints an accurate + esthetically pleasing image is quite different from generating code that achieves a non-spatial goal.

runarberg 6 hours ago | parent [-]

If people need to generate good-ish SVG why not simply use a specialized model for a much better result and for far cheaper and quicker?

Why do LLMs need to be able to do this as well, but worse, slower and more expensive?

nl 3 hours ago | parent | next [-]

For example I do SVG diagrams to explain architecture and specific data flows in a multi-tier app.

LLMs do a great job because they understand both code and SVGs well.

Edit:

An example for a synth I'm buulding: https://imgur.com/a/U694Ek7

The irony of having to post it as a PNG isn't lost on me...

an hour ago | parent | next [-]
[deleted]
runarberg 2 hours ago | parent | prev [-]

This is not a strong use case. I could have made this in Microsoft word when I was 13 years old.

Why on earth would you let an LLM do this when it would take you 10 min to do this in Figma or Inkscape or even just Word

simonw 27 minutes ago | parent | next [-]

I'd be very impressed by anyone who can produce that in Figma or Word in ten minutes.

runarberg 12 minutes ago | parent [-]

These are rectangles with some fill and strokes, textboxes lines and arrows. If the text has already been written, and the author has a clear idea of the flow all it takes are some a dozen or so points and click, and setting a couple of fills, then copy pastes, and finally some resizes, selects and moves.

This design is trivial. I admit it would be hard to achieve in Word (or at least for me because I don’t know how to make a diagram in Word any more) but Figma and Inkascape are made to do these things, and have optimized UI for that (personally I would have just used mermaid though).

I think you may have lost your faith in human capabilities just a little bit if you think drawing stuff like this takes any time or effort at all. Compared to designing and architecturing the system, drawing the diagram is trivial. Now I know that my parent did neither but it seems like they vibe-coded the whole thing. I’m sure they will end up with a fun little toy from the whole endeavor they can play with for 2 weeks before abandoning. Maybe the author will even feel bad about the carbon footprint of this whole exercise and buy some carbon offsets to make up for it.

nl an hour ago | parent | prev [-]

Because it reads the code and generates this in a single pass in 5 minutes.

I haven't read the code.

Why would I do it in a slower, more difficult way for something that's going to be outdated in 2 hours?

yorwba 6 hours ago | parent | prev | next [-]

Yeah, if you want to generate an aesthetically pleasing SVG, you'd be better off asking a pixel-based image-generation model for "vector art" and then deconstructing it into an equivalent SVG with something like LayerPeeler. https://layerpeeler.github.io/

exhaze 6 hours ago | parent | prev [-]

Agree directionally - even back in sonnet 3.5 days, I was helping some friends by showing them how to create intermediate representations for SVG building blocks mapping to parametrizable functions that can be used to make interactive SVG-rendered visualizations for various medical needs.

wasabi991011 8 hours ago | parent | prev [-]

I don't see why that's true. LLMs don't have to only be good at code-writing.

timClicks 7 hours ago | parent | prev | next [-]

If they're optimizing for SVG generation, then that's an excellent outcome in my opinion. Vector images shouldn't be "pretty niche".

knollimar 7 hours ago | parent [-]

Please it's almost my primary bench for drafting understanding.

I'm almost about to post that xkcd 810

Eggpants 2 hours ago | parent | prev | next [-]

I've been pretty happy with LLM svg based data plots I've asked, including log scaled axises and histograms. Definitely a first world problem of course.

brikym 8 hours ago | parent | prev | next [-]

Then it's great. A year ago I couldn't get any AI to draw a simple company logo in SVG or even convert from raster. No doubt the Pelicans put pressure on the labs to fix the awful SVG situation. Now I can even make a decent Peli in 3D.

simonw 7 hours ago | parent | prev | next [-]

Gemini have absolutely been SVGmaxxing. They've openly talked about it.

exhaze 6 hours ago | parent | prev | next [-]

Not an expert on this, so won’t speculate regarding what traits would create robustness specifically attributed to SVG visual representation capability, but felt you might find this paper on reasoning models trained on physical world video data becoming better at general reasoning interesting:

https://arxiv.org/abs/2210.05359

netsec_burn 9 hours ago | parent | prev | next [-]

Addressed in the article, in case you're curious.

Dylan16807 3 hours ago | parent | next [-]

Addressed by three inconclusive sentences. When someone goes into deeper detail on the topic, we shouldn't assume they failed to read the article just because they said "what if" instead of mentioning those three sentences.

lukev 9 hours ago | parent | prev [-]

Well, it’s mentioned as a limitation of the analysis, very much not ruled out (or in.)

That simonw is causing labs to do extra fine-tuning runs for this seems highly probable :)

6thbit 6 hours ago | parent | prev | next [-]

I think svg is a balanced test because of the level of indirection and the required 'conceptualization' of physical elements then expressed through code.

nbardy 4 hours ago | parent | prev | next [-]

Then they’re legitimately getting better at svg which is a valuable tool.

kaliqt 8 hours ago | parent | prev | next [-]

Funnily enough, not that niche, because I have tried many times to do it as part of a wider project.

kelvinjps10 7 hours ago | parent | prev | next [-]

SVG are really useful you can create images that don't have the AI look

charcircuit 9 hours ago | parent | prev | next [-]

I agree, other formats, both textual and binary should be tested.

delichon 5 hours ago | parent | prev [-]

> What if they’re not pelicanmaxxing, but svgmaxxxing in general?

Mission fucking accomplished. https://xkcd.com/810/