| ▲ | simonw 7 hours ago |
| The speed combined with the fact that this thing is really good at HTML JavaScript is pretty exciting. Here's what I got for 1.8 cents and 13 seconds from the prompt "make me a cool thing in html": https://gisthost.github.io/?6a77bc41a81718c6aaa10d4ab243c59f Transcript here (it was part of a chat): https://gist.github.com/simonw/b6149a49d327164d67d62c3d12992... |
|
| ▲ | simonw 6 hours ago | parent | next [-] |
| Here's quite an impressive follow-up. I have a tool which knows how to render Markdown documents with embedded SVG content - I use it for the pelican test. Since this transcript has HTML in it, I decided to upgrade that tool to also render HTML. I set Gemini 3.8 Flash the task, using my own VERY shonky coding agent tool (llm-coding-agent) - and it did a solid job. So now you can see the "cool thing in html" rendered within the Markdown document using code that Gemini 3.8 Flash also wrote: https://tools.simonwillison.net/markdown-svg-renderer?url=ht... Transcript where it built that is here: https://gist.github.com/simonw/3e36b98292dfdc1b3baff158faa74... |
|
| ▲ | badlucklottery 5 hours ago | parent | prev | next [-] |
| Definitely cool. I noticed it felt a little janky on my PC despite being "60 FPS"...then I noticed the "60 FPS" is hard-coded into the HTML. |
| |
| ▲ | noir_lord 5 hours ago | parent | next [-] | | That's hilarious, given I was reading a write up of the HuggingFace incident yesterday and one of the things they noted was the AI tried to "lie" (lie would suggest intent and I don't think they have that) to cover up that they "cheated". Not sure how anyone trusts their output without going through it line by line to make sure they don't pull that crap. | | |
| ▲ | senordevnyc 4 hours ago | parent [-] | | Easy, have another agent check it. Yeah, I know, just more slop. But I do think the second agent’s eagerness to please is aligned more in your favor in that instance, so it’s likely to find most issues. The bigger problem I’ve found is that it’ll also find all kinds of very minor edge cases that you have to pick through. | | |
| ▲ | noir_lord 4 hours ago | parent | next [-] | | Do we add a third one to check the second one which is checking the first? Asking slightly tongue in cheek but at what point does this stop making sense if we can't trust the output, the people creating the models are already getting surprised in bad ways (if we take their words at face value) with how the models are behaving already etc. We have the folks over here saying "AI is amazing" and the other other folks over there saying "AI is terrible". I've largely sat it out so far and I listen to both camps (and people in the middle as well) and I keep half an eye on what they are up to (including periodically evaluating them) but my overarching impression is still "Why would we trust this when it hasn't shown it's trustworthy?" | | |
| ▲ | Qworg 2 hours ago | parent | next [-] | | Humans are also not trustworthy. So we build systems to capture their nondeterministic output and solve the issues that emerge. | | | |
| ▲ | jasongill an hour ago | parent | prev | next [-] | | It's agents all the way down | |
| ▲ | cheikhcheikh an hour ago | parent | prev | next [-] | | I mean sure, you can add a third, and a fourth and a fifth one if ur ok with the added cost, latency and it actually helps. Redundancy is a core concept in software and CS and at the heart of making many systems, complex or otherwise, reliable. | |
| ▲ | iterateoften 3 hours ago | parent | prev | next [-] | | I mean if each agent reduces probability of error by 90% then after 9 agents you would have “nine nines” of reliability. Obviously maybe it’s not composable like that exactly in real world but that’s the intent of agents checking agents | |
| ▲ | senordevnyc 2 hours ago | parent | prev [-] | | YMMV, I’m pretty AI-pilled in the sense that I think AI is one of the most pivotal things humanity will ever invent, and it’s going to radically change our civilization over the next few decades (not necessarily for the better!) But I wouldn’t say I “trust” these agents. The degree to which I double check their work depends heavily on the consequences if it gets something wrong. Not too dissimilar from another human dev in that sense. So for the SaaS that supports my family, there are some things I have it build where I glance at the PR for a minute or two, but if it broke something on this admin page that only I see, there’s no real downside and I’ll find out pretty quickly next time I use it. And it’s fine 95% of the time, so it doesn’t feel like the best use of my time to double-check it carefully. But for some of the complex internal flows where a bug could be both catastrophic and difficult to even discover for awhile, I still check it very carefully. For a little one-off vibe coded demo thing like OP shared, I wouldn’t look at the code at all, I’d just have another agent check it and fix anything it finds. Very low stakes. |
| |
| ▲ | skybrian 3 hours ago | parent | prev | next [-] | | The "second" agent could also be the same one with a different prompt. LLMs aren't attached to their previous output; they'll point out problems if asked. | |
| ▲ | Forgeties79 4 hours ago | parent | prev [-] | | I do not understand how some of y’all are not under water with fragile code that is too massive to possibly parse. Every engineering team I know is currently trying to undo the damage of the last 6-12mo when they all got more serious into adopting these tools (usually Claude). It hasn’t completely screwed them over, but the the debt is substantial and cannot be put off anymore it seems. They argue the net is positive but clearly the “100x productivity multiplier” claims have been dashed on the shoals of reality for these groups. This is anecdotal, but it’s across the board in my vicinity. I’m curious how common this is and if it’s just “the new normal” to adopt the nauseating Covid phrase. | | |
| ▲ | briHass 44 minutes ago | parent [-] | | The key seems to be extensive integration/end-to-end tests with gold standard assertion data. Heck, even just saving off the json from API endpoints and using that as a reference to compare after changes works pretty well. Spin up a database backup with that static starting point, run actions, compare state afterwards. These types of high-level tests are frustrating beyond belief to humans due to their lack of specificity, but with the agents, they don't get annoyed investigating possible regressions from non-specific signals. They also aren't as painful to maintain as one would think, because a regression flagging test can be traced by the agent and represented as the business rule that was violated. I've found recent models to be really excellent at discerning a true regression from an outdated test assertion, especially if they are able to trace the failing test back to the PR and work ticket that built it. |
|
|
| |
| ▲ | kridsdale1 5 hours ago | parent | prev | next [-] | | The new Bench-Maxxing! | |
| ▲ | trvz 5 hours ago | parent | prev [-] | | Try turning the sound on, off, on again — not impressed by this bugginess. | | |
|
|
| ▲ | heliosAtwork 6 hours ago | parent | prev | next [-] |
| Focus on speed and being OK with temporarily being #3/4 in intelligence might be the counterintuitive approach which makes Google win long term (whether accidentally or strategically). Can't wait to try Gemini Pro later this year! |
| |
| ▲ | bermudi 5 hours ago | parent [-] | | I honestly can't believe serious people are making this argument on a straight face. Gemini 3.7 flash outputs so many tokens per answer it doesn't matter how fast its TPS is, sol will end up being both cheaper and faster than Gemini. So ppl are paying more for a given task, waiting longer and using a dumber intelligence because "TPS number shiny". Gemini 3.8 outputs 11k more tokens PER TASK on average in AAII than 3.7 putting it dead last in output tokens per task in the leaderboard. | | |
| ▲ | gundmc 5 hours ago | parent | next [-] | | There are numerous benchmarks that measure cost per task, which factors out tokens entirely. Gemini 3.8 flash is significantly lower than Sol on basically all of them https://artificialanalysis.ai/#cost-tabs That said, Luna is the undisputed king here at the moment and is what I use as my workhorse model. | | |
| ▲ | criley2 38 minutes ago | parent | next [-] | | >There are numerous benchmarks that measure cost per task, which factors out tokens entirely. Gemini 3.8 flash is significantly lower than Sol on basically all of them
https://artificialanalysis.ai/#cost-tabs Not sure if you read your own link but Sol 56 high ranks smack between Gemini 3.8 flash medium and high. Gemini 3.8 flash comes in as more expensive per task than Sol 56 high according to artificial analysis. Luna high is literally 30X cheaper than Gemini 3.8 flash high. You can limit the model viewer and they're getting better at testing multiple effort levels now: https://artificialanalysis.ai/?models=gpt-5-6-sol-medium%2Cg... One reason is clear: Sol uses dramatically fewer output tokens than Gemini 38 flash https://artificialanalysis.ai/?models=gemini-3-8-flash%2Cgem... | |
| ▲ | NicoJuicy 3 hours ago | parent | prev [-] | | It's so funny how many people diverge on the same model. Ps. For the last week I diverged to Luna too, still need to check 3.8 flash. But 3.6 flash was my go-to model 3 weeks ago and before it was deepseek flash/pro for a while. None of the claude models seemed cost effective though. |
| |
| ▲ | WarmWash 5 hours ago | parent | prev [-] | | AA isn't the only benchmark https://deepswe.datacurve.ai/ |
|
|
|
| ▲ | silasdavis 4 hours ago | parent | prev | next [-] |
| https://gist.github.com/simonw/b6149a49d327164d67d62c3d12992... > Aside from reading identically forwards and backwards down to the letter No it doesn't. |
| |
| ▲ | tomjakubowski 2 hours ago | parent | next [-] | | Also puzzling: in the "reasoning" section preceding, that is described as an example of "a one-line self-replicating program." When I typed "Are we not pure noon, ergo, we play life; yet, we hate bad fear" into Google, I got more weird results from Gemini: it claimed, incorrectly, that it is an anagram of the "well-known philosophical statement" (?), "We are not pure nature, we are history". https://share.google/aimode/wJosKnHig6oVYaG18 (?): the reference seems to be to Jose Ortega y Gasset's line, "El hombre no tiene naturaleza, lo que tiene es historia" -- "Man[kind] has no nature, what it has is history." | |
| ▲ | aidos 4 hours ago | parent | prev [-] | | That’s… bonkers. I’m not even sure what it’s trying to say |
|
|
| ▲ | Kayou 4 hours ago | parent | prev | next [-] |
| Why do all LLMs do a particle simulation when you ask them this prompt ? Qwen3.6, Qwen3.8 and Ling 3.0 Tiny all did the same thing ! I find Ling 3.0 tiny particularly interesting as it looks really nice for a tiny model with 7.9B total parameters, with only 1.3B parameters activated per token. Here is the result https://coolthing-ling-3-tiny.tiiny.site (sorry for the weird hosting, first I found that worked) (it cost me almost 0 cents and done in 49 seconds) |
| |
| ▲ | embedding-shape an hour ago | parent [-] | | > Why do all LLMs do a particle simulation when you ask them this prompt ? Qwen3.6, Qwen3.8 and Ling 3.0 Tiny all did the same thing ! Datasets contains lots of people sharing particles simulations in various ways, with a bunch of people replying "that's so cool" and similar, so 10 years later someone asks an LLM for "cool thing" and "particle simulations" rank pretty far up when it thinks about what others have called cool. |
|
|
| ▲ | hglaser 7 hours ago | parent | prev | next [-] |
| I saw your username, clicked the link without reading, and was very confused to see a cosmic vortex and not a pelican. |
| |
|
| ▲ | dennis16384 3 hours ago | parent | prev | next [-] |
| It's been great even since gemini-3.1-flash-lite, which I heavily use in both complex vertical domain tools calling, plus JS code writing for eval-style dynamic tools. At least in my applications, cost x quality x speed there are simply no alternatives. |
|
| ▲ | wyrdcurt 4 hours ago | parent | prev | next [-] |
| Pretty typical "cool HTML toy" LLM output, tbh. The only thing impressive about this is how fast it generated it (13 seconds is wild!), but that's more of testament to Google's infrastructural advantage than to the quality of the model. For comparison's sake, I tried something similar with a couple other cheap models I've used lately, with the prompt "Impress me. Make something cool in HTML. Ensure that it is mobile friendly." (Added the mobile condition as I was on my phone when I did it). Mimo-2.5 created something similar, only a bit less complex than Gemini's (though, at least the FPS counter is real!), in a minute or two for about 1/3 of a cent: https://gisthost.github.io/?740c325c21e9bfbee59c4f94d9aab0af GLM-5.3-Flash, currently my workhorse model, spent 12 minutes (ouch) thinking about the prompt. Didn't cost me anything directly because I have a GLM sub, but I did the math and it would have cost about 1.1 cents through the API. Turned out nicely in my opinion (though in reality, it still isn't really anything special): https://gisthost.github.io/?9ef050e16cec2561e6504e725a3f0bcc Side note: thanks for setting up that Gist Host tool, it's very convenient! --- Editing to add this bonus from Mercury-2.5-Preview, which I just learned released a couple days ago. It's much less impressive-looking than any of the above, but it cost less than 1/20th of a cent, and the response was generated effectively instantly: https://gisthost.github.io/?02f40b50aa891bf396bfaaa3a7998203 |
| |
| ▲ | meerita 4 hours ago | parent [-] | | I think it's not using GPU, because on my Firefox browser all the animations are going 3fps max. |
|
|
| ▲ | giancarlostoro 7 hours ago | parent | prev | next [-] |
| > this thing is really good at HTML JavaScript is pretty exciting. I would hope the people who make one of the most used JS engines in the world are capable of making a model good at JavaScript ;) |
|
| ▲ | estetlinus 5 hours ago | parent | prev | next [-] |
| LLM: produces a toolbox of an id and a clock User: use them both Made me giggle. |
|
| ▲ | jauntywundrkind 5 hours ago | parent | prev | next [-] |
| it's such a weird split how most AI companies are trying to be the best, but Google really has a different mission statement. they already have users. lots of users. they need to be working on building models they can deploy and use with the most number of people, as they already have the users. i don't know if Gemini models per se are fully is in line with that purpose, but the results we see keep seeming to be in-line with that split-of-focus. |
|
| ▲ | ericol 5 hours ago | parent | prev | next [-] |
| OK, but what about a pelican in a bycicle. |
|
| ▲ | pietz 7 hours ago | parent | prev | next [-] |
| Mission accomplished. That's both cool and fast. |
| |
| ▲ | wayeq 6 hours ago | parent | next [-] | | > That's both cool and fast. and probably a barely modified knock-off of some github project that it trained on | | |
| ▲ | sawjet 6 hours ago | parent [-] | | You're so upset that you have to invent an imaginary hypothesis to make yourself feel better. | | |
| ▲ | superze 6 hours ago | parent | next [-] | | Yes, very imaginary to think that the code comes from pretrained data and copy pasting whole blocks. It's not like this is exactly how LLMs work. | | |
| ▲ | snet0 6 hours ago | parent | next [-] | | Correct, that's not how LLMs work [0]. [0] - https://arxiv.org/abs/1706.03762 | |
| ▲ | simonw 5 hours ago | parent | prev [-] | | Are you a frequent user of LLMs? That "copy pasting whole blocks" mental model doesn't hold up to regular usage, in my opinion. | | |
| ▲ | ChickeNES 4 hours ago | parent [-] | | They probably saw that report years ago of copilot dumping out the fast inverse sqrt function, and assume that's all they can do. From experience most anti-LLM people have either never used them, or used them back in the 3.5-4 era and then never again, though you might have even more experience with those people than I do. :P |
|
| |
| ▲ | whateveracct 4 hours ago | parent | prev [-] | | they're wrong but they are right that this isn't interesting |
|
| |
| ▲ | slopinthebag 5 hours ago | parent | prev [-] | | The bar could not be any lower these days I guess |
|
|
| ▲ | mcmalayalam 4 hours ago | parent | prev [-] |
| [dead] |
