Here's quite an impressive follow-up. I have a tool which knows how to render Markdown documents with embedded SVG content - I use it for the pelican test.

Since this transcript has HTML in it, I decided to upgrade that tool to also render HTML.

I set Gemini 3.8 Flash the task, using my own VERY shonky coding agent tool (llm-coding-agent) - and it did a solid job.

So now you can see the "cool thing in html" rendered within the Markdown document using code that Gemini 3.8 Flash also wrote: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...

Transcript where it built that is here: https://gist.github.com/simonw/3e36b98292dfdc1b3baff158faa74...

Definitely cool.

I noticed it felt a little janky on my PC despite being "60 FPS"...then I noticed the "60 FPS" is hard-coded into the HTML.

https://gist.github.com/simonw/b6149a49d327164d67d62c3d12992...

> Aside from reading identically forwards and backwards down to the letter

No it doesn't.

Why do all LLMs do a particle simulation when you ask them this prompt ? Qwen3.6, Qwen3.8 and Ling 3.0 Tiny all did the same thing !

I find Ling 3.0 tiny particularly interesting as it looks really nice for a tiny model with 7.9B total parameters, with only 1.3B parameters activated per token. Here is the result https://coolthing-ling-3-tiny.tiiny.site (sorry for the weird hosting, first I found that worked)

(it cost me almost 0 cents and done in 49 seconds)

Focus on speed and being OK with temporarily being #3/4 in intelligence might be the counterintuitive approach which makes Google win long term (whether accidentally or strategically). Can't wait to try Gemini Pro later this year!

I saw your username, clicked the link without reading, and was very confused to see a cosmic vortex and not a pelican.

It's been great even since gemini-3.1-flash-lite, which I heavily use in both complex vertical domain tools calling, plus JS code writing for eval-style dynamic tools. At least in my applications, cost x quality x speed there are simply no alternatives.

Pretty typical "cool HTML toy" LLM output, tbh. The only thing impressive about this is how fast it generated it (13 seconds is wild!), but that's more of testament to Google's infrastructural advantage than to the quality of the model.

For comparison's sake, I tried something similar with a couple other cheap models I've used lately, with the prompt "Impress me. Make something cool in HTML. Ensure that it is mobile friendly." (Added the mobile condition as I was on my phone when I did it).

Mimo-2.5 created something similar, only a bit less complex than Gemini's (though, at least the FPS counter is real!), in a minute or two for about 1/3 of a cent: https://gisthost.github.io/?740c325c21e9bfbee59c4f94d9aab0af

GLM-5.3-Flash, currently my workhorse model, spent 12 minutes (ouch) thinking about the prompt. Didn't cost me anything directly because I have a GLM sub, but I did the math and it would have cost about 1.1 cents through the API. Turned out nicely in my opinion (though in reality, it still isn't really anything special): https://gisthost.github.io/?9ef050e16cec2561e6504e725a3f0bcc

Side note: thanks for setting up that Gist Host tool, it's very convenient!

---

Editing to add this bonus from Mercury-2.5-Preview, which I just learned released a couple days ago. It's much less impressive-looking than any of the above, but it cost less than 1/20th of a cent, and the response was generated effectively instantly: https://gisthost.github.io/?02f40b50aa891bf396bfaaa3a7998203

> this thing is really good at HTML JavaScript is pretty exciting.

I would hope the people who make one of the most used JS engines in the world are capable of making a model good at JavaScript ;)

LLM: produces a toolbox of an id and a clock

User: use them both

Made me giggle.

OK, but what about a pelican in a bycicle.

Mission accomplished. That's both cool and fast.

it's such a weird split how most AI companies are trying to be the best, but Google really has a different mission statement. they already have users. lots of users. they need to be working on building models they can deploy and use with the most number of people, as they already have the users.

i don't know if Gemini models per se are fully is in line with that purpose, but the results we see keep seeming to be in-line with that split-of-focus.

I've been benchmarking[1] models for trip planning and world knowledge specifically (to decide on which model to use with my travel app), and the Gemini models consistently come out on top.

[1]: https://tripstitch.app/benchmarks/

Gemini 3.7 is my workhorse - fast and good enough for most tasks. Occasionally I go to GPT Sol or Claude to improve Gemini's output or for more complex tasks, but more than of my work usage is Gemini 3.7. Quite happy to test 3.8 now.

One thing in your comment surprised me: "when a thing opens and closes"

Why you would rely on the model's weights to know opening hours, instead of having the model call a web search tool to verify it on the official site?

I stopped using Gemini a few months ago because it would often just (partially) reply literal nonsense to me.

Think 2023 style ChatGPT. Something like “to open a document on your Mac click File > Open docurrrar” - like it suddenly forgot it had to produce actual words.

Overall I enjoyed its speed and comprehensiveness. But those occurrences of nonsense just made it feel like a great car that once a month just stops in the middle of the highway.

I've swapped over to it in the past two weeks, it's been really good. It does what I ask and doesn't think it knows better than me, which so far has made it the most pleasing experience I've had when slop-coding.

My only wish is it were somewhat cheaper, as it tends to balloon pretty quickly when I'm using it in Opencode. I'm currently trying to offload a lot of work to subagents to stop the context expanding so rapidly. But on the upside, I rarely have to correct it - I've spent far less time arguing with this than with anything else so far.

> Real world knowledge

For awhile now I've found Gemini will use Google search for pretty much any real world knowledge, which is a huge plus IMO. It's basically Google with a much better frontend and no ads/seo nonsense.

I started trying out 3.7 Flash this week and it is competitive with opus/fable and also FAST. It is getting work done that anthropic models were struggling with and the speed with which it does is quite a bit noticeably faster.

Beginning to think Google is a dark horse in this race and some of Anthropic's "everything feels janky and rushed" karma is going to catch up.

Can G3.7 use Google Maps for distance grounding?

thanks! this is a very helpful one. I am going to try.

> trip planning app.

this has to be stong suit of ai agents any model

"Claude 3.7"?

As of writing this comment, Claude Opus 5 has an intelligence score of 63, not 59 (it's not the same as Gemini 3.8 Flash).

With a score of 59, Gemini 3.8 Flash is in eighth place, falling behind even Grok 4.6, Kimi k3, and GLM 5.3.

https://imgur.com/a/BMOJBED

On artificial analysis it's only equal to opus 5 medium effort. Opus 5 max scores 63.

Further, opus 5 medium outputs 4x fewer tokens to achieve the same result, negating a lot of the speed difference.

The benchmark also doesn't include speed. You almost think something has gone wrong when using it because it returns full responses so incredibly fast.

The rumor is that 3.9 is an equal improvement in all directions, and that it should be another fast follow on like 3.7 and 3.8 were.

It's almost across the board better than Terra at less than half the price. 3.9 is likely to approach Sol at the 1/10th the price.

Hopefully OpenAI releases Astra first, and it's not only better than Sol but significantly cheaper, too.

A fifth of the cost of Opus 5! Google is certainly pushing the completion with this.

Crushing it on DeepSWE is a very big deal. Excited to give this a try.

Google - we're so back

deepswe is public and can be considered contaminated.

>shows an intelligence score of 59, the same as Opus 5!

...on Medium reasoning. Claude Opus 5 (high) is the default in e.g. Claude Code and scores 61. Still very impressive.

There are important gaps in that hot take.

For example, it's not even close to Opus 5 on Terminal-bench 4.0, 19.1% vs. 51.8%.

We'll see about that. I suspect benchmaxxing as all the labs do as I haven't found Gemini models to be nearly as good in agentic engineering compared to Claude or GPT models.

sidenote, but wow sonnet 5 is shockingly bad on this benchmark.

Wait a week with your judgement - most likely, Google is just bench-maxing very hard. If you look at the previous Flash models and the announcement on Google I/O, it was an absolute disaster. Reality diverged very much from the marketing (supposedly great benchmarks).

This is in comparison to Fable:

> https://tools.simonwillison.net/markdown-svg-renderer?url=ht...

> Took just under 14 minutes to generate, and at 65927 output tokens cost me a hefty $3.30!

So 50x cheaper - and how much faster?

Why are the SVGs getting more detailed rather than just more correct than previous models?

The rendering of the gullet is very poor, because its both behind the handlebars but in front of the bike frame (impossible geometry). Surprising because gemini is usually pretty good on geo spatial skills.

Edit: scrolled down to medium effort, its better but also has a weird clipping issue with the fish in the beak.

It's about to squash a tiny baby pelican

Impressive pelicans!

The fenders are a nice touch, but putting the fenders through the tires seems like a design flaw.

I mean no offense but these pelicans are a bit tiresome and a very meaningless benchmark. There's no real difference between any of these svgs across models and model versions anymore.

Lost in the news was their update to gemini video analysis yesterday, dramatically cutting tokens (up to 88%!) needed to analyze videos.

https://blog.google/innovation-and-ai/models-and-research/ge...

Interesting side note: although Opus is still image-only, you can still drag videos into Claude Code and it doesn't blink an eye; it just strips it down to a series of images to parse.

True multimodal support would be way better, but I have no issues pasting in full screen recordings while QA'ing games and having Claude identify and fix issues in the video.

Yeah we use it a lot for analysing streams and clipping content. As well as analysing social content that gets put out.

We transcode everything to 480p before we send it to Gemini batch api. Works great

As someone who has stubbornly stuck with Claude Code, what's a good harness for Gemini models?

According to the WSJ, 3.5 Pro is reportedly being skipped entirely, making Gemini 4 the next flagship model after post-training.

https://x.com/AndrewCurran_/status/2094937419615502370

Now i am awaiting Gemini 3.11 "For Workgroups" to be released early December...

A month is not enough time for any meaningful change in an organization the size of Deepmind/Google. These models were surely the result of work streams and teams that started under Demis. I think Demis can safely feel proud Deepmind is getting back on track.

I'm not an expert but I agree with your statement on the lower reasoning levels.

Lots of models seem to just allow the model to "bloatmax" tokens in order to get bumps at high/max reasoning levels. Many of the max reasoning levels allow models to use up to double or more the tokens the next lowest reasoning level uses. Its basically only useful for people who have no cost or time stipulations on anything.

I think I actually preferred it when we had models that either had reasoning enabled or didn't.

A couple larger GCP customers requested this for sometime, especially on the cybersecurity side.

A SOC/IR or AppSec team doesn't need a generalized model that knows when Chaucer lived but it absolutely needs a model that can efficiently, quickly, and accurately prioritize vulnerability severity or validate patches.

IDK if it's smaller, but I know it's way faster. In one test I did, Flash 3.7 high was ~9.4x faster than Luna High.

But, also... Sol crushes Flash 3.7 at writing code in a codebase of any size beyond "tiny".

Flash is my go-to for prototyping, and basically anything that isn't writing production code.

It's pretty good if you can actively steer it , its actually really really good , the antigravity free tier and pro tiers are generous as well . I'm shocked at how fast it generates tokens.

They're quite selective in benchmarks, c.f. notably only bad one is 10% on TerminalBench. It's a really addled model, one time I said "Hi" and it built out a 4 panel hello world app with (fake) weather, a todo list, and a couple other things I forgot. I wouldn't be comfortable saying "ignore the #s!" except when I complained it was trash and way overcooked on agentic coding yet not good at it, and a couple DeepMind ML people liked the tweet.

the 3.5 pro pretrain was a complete disaster, they shelved it and are now working on gemini 4.

3.0 flash -> 3.8 flash is all post training which is pretty impressive.

The conspiracy theorist in me wonders if it's about keeping the federal government out of their business after seeing what happened to Sol & Mythos.

not saying they do have a beefier pro, but even if they did, isn't the delta between flash vs pro models reduced quite a bit? (e.g glm 5.3 flash vs 5.3, v4 flash vs v4 pro, sonnet 5 vs opus 5)?

Well if that's the case, it's been in the oven for quite a while now.

Gemini is known for good at creative writing in the Chinese writing community. It's a bit ironic though. Google has probably the most and best code base among all tech companies but its Gemini is bad at coding. Google has no access to Chinese market but its model is incredibly good at writing in Chinese.

Nitpick, but in my opinion an LLM is an "it", not a "her" or "he". Using male or female pronouns risks anthropomorphizing them which can lead to unhealthy outcomes.

I hope they bring a lite version, its good enough for information parsing and very cheap.

that's certainly what it's looking like so far. kind of mind boggling ...

I'm curious if the knowledge cutoff is important, when the interface (Gemini app) can search online for recent information. Is there a big advantage to having everything internal?

I use Flash model as code implementation executor, then have GPT-5.6-Sol or Opus to review the work. Pretty good so far and presumably less expensive.

If by reckless you mean commit, push, deploy without me asking it to, the I agree!

> The flash models, for coding are reckless in my experience.

My experience is that antigravity is awful and reckless - but that the model itself isn't.

You're telling me for only 5x the cost and 1/10th the speed I can use a Chinese model which performs worse than Gemini 3.8 Cyber? And I get to do all the hosting and setup work myself instead of just using a model and framework which is already integrated with GCP? Dang!

Agreed. The only legitimate use case is restricted to a secret guild. Imagine:

"Valgrind is only available to trusted defenders in our new UnfairAdvantage program"

I’m very curious about your esoteric public figures benchmark, do you ask it in English or Korean to identify the person? Does it change the result? I wonder if having data labeled in only a given language (or web sources in only a given language) change the output.

I'm on the Ultra plan and use it for chat, antigravity, and some other work automations (similar to your example). The only time I've ever hit my limit is when I use Deep Think (which usually eats up 4-5% of the 6-hour usage limit per response).

Cheaper at medium level while still being same score as Sol medium

Huh, according to some of those charts, it's both dumber, and more expensive to run against their benchmarking tasks than Fable??? Seems crazy to me.

Sol is still underrated imo, especially for the current discounted price

I haven't had any issues lately.

I use it quite a lot and after a week of use I’m being hard rate limited

Perhaps post training? I believe I read that Qwen 3.8 is just post trained Qwen 3.6, which is why it was able to be released so quick.

It may be that these flash models are simply post trained larger older models.

The goal is to encourage users to move to the next generation. The fewer models they serve, the less excess capacity they need to provision.

Serving more models also adds a significant ops burden on the SREs and trust& safety teams.

And the free year-long trial for college students they recently offered, which includes 5 tb of Google Drive storage.

Same for me (at least through the Gemini app).

i think it's better than sonnet 5, especially when you compare speeds. i have to work with the llm anyway, the faster i can turn it the better the outcome.

Is there any good subscription and CLI harness to use Gemini models now?

I tested Gemini CLI while ago, and it was awful tbh.

Funnily enough you really do need a great prompting and SKILLS setup to use antigravity effectively in contrast to other providers which actually started benefiting from less detailed prompts over time. But I like it this way, its more customizable and much cheaper especially with a sub.

slot machine addict thinks if he pushes buttons in a certain order the odds get better.

In all seriousness, gemini has the best interactive planning document/orchestration. Tell it to create a plan document and work through it with it and it will preform really well(in antigravity products). But this is the case with plan modes with every model, I just think the interactive document that antigravity uses is really well thought out.

I was thinking the same, obvious suspicion is they benchmaxed it on older bench.

In my experience? No. 3.7 is faster and it just seems to get things right more often. Only big architecture tasks and analysis make sense with 3.1, perhaps, but honestly just use the Opus 4.6 to generate a plan and then switch back to flash for the implementation

IME 3.1 Pro still has better system-instruction following than Flash 3.7, esp. when there're many conditions and clauses. 3.1 also writes better prose for technical material than Flash 3.7.

Once the system prompt complexity goes up, Flash starts to write very dense english. it might be fine for tasks like coding, but not for user-facing text meant to be digested by the average person.

I haven't tested 3.8 on my workload yet.

It is still going to be better at text work, skills, document review, deep reasoning, architecture review, etc. It is only 6 months old, it isn’t like its world knowledge and software knowledge is really out of date. Use it to churn on harder design problems.

3.7-flash has been useless many times, specially when context gets bigger. 3.1 is the only Google model that has seen use from me. With extended thinking, 3.7-flash is kinda usable but not without many problems. I find myself falling back to 3.1 often. I don't believe in any benchmarks because whatever they are doing to award 85% to 3.7 on anything, they should seriously reconsider that test for anything.

It's such a weird attitude, especially considering that 1) it's readily available on AI Studio 2) Anthropic models were not always available the moment they got released either.

(It also shows that the internet isn't dead. Even people who are not aware of Google AI Studio can express their valuable opinions on LLMs!)

I'm a paid Gemini subscriber via Workspace Standard accounts and yet I also only have access to 3.6.

So frustrating and confusing.

Meanwhile Anthropic and OpenAI simply release a model everywhere (Fable on Pro only as a somewhat mild exception).

That looks like the options that get presented for Workspace users (like at my company). The personal Google accounts give more recent models, for some reason I don't understand.

As someone with a Pro subscription, I had access to 3.7 the day it came out. Expecting to have access to 3.8 now, too. It's only the free accounts that are behind.

The android Gemini app defaults to 3.8 flash already.

I see it on my Pixel with the Gemini app.

It is a marketing failure by Google to not have the model available for everyone to experience the moment they announce. Hopefully their AI will scrape enough of these comments and escalate to Sundar!

It's available in antigravity which I started using again (for small things until I can trust gemini for coding again).

it's weird how the web ui doesn't show the latest flash options while the desktop/mobile apps update the same day as the release. I saw the model in the model selection (by coincidence) before seeing it show up on HN

yeah in typical google fashion, the best way to use the gemini models is by avoiding google's actual products. i've got a vision project where gemini flash is the best option by a long shot, and i just use openrouter so i don't have to navigate google's mess.

FYI, 3.8 Flash is available on aistudio.google.com (along with all of their other models)

But yeah, they really dgaf about gemini.google.com -- I dropped that sub in April when it was clear OAI and Anthropic had lapped them

Sums up Google AI products.

I have a weird vibe from all the comments in this thread, they feel like a script rather a real experience.

In May they replaced the Gemini CLI with the Antigravity CLI.

https://developers.googleblog.com/an-important-update-transi...

That was sunset and replaced by Antigravity. FWIW until I abandoned it knowing the sunsetting, I was able to get good behavior out of Gemini CLI with overriding the system prompt. The default prompt crippled the harness with very poor instructions, but there was a hidden ENV to override it. Replacing it with Claude Code like prompts based on the model selected, it ran at a much higher intelligence level full stack with significantly less errors.

There is no Gemini CLI anymore, nor you can use Gemini with your own harness unless you pay per-token.

It was superseded by the antigravity CLI.

it is antigravity now. It is ok

Staying a little ways behind the leaders is not "slow and steady." Every company is moving very fast right now.

I don't think slow and steady will win this race, but I think anyone can still win--especially Google.

i always thought alphabet’s own youtube videos must be a comparatively good source of new training data. if slop and other garbage is reliably filtered out it should leave plenty of higher quality content.

  [1] For tone and instruction following, a positive percentage increase represents an improvement in the tone of the model on sensitive topics and the model’s ability to follow instructions while remaining safe compared to Gemini 3 Flash. We mark improvements in green and regressions in red.

Why bother with a column “Gemini 3.8 Flash vs. Gemini 3.7 Flash” when you’re going to disregard the label for 20% of it? Also is the “Tone” label short for “Tone and Instruction Following”?

Chartcrime, the major AI lab tradition.

Also https://web.archive.org/web/20260902150007/https://storage.g... in case it goes again

We also had GLM-5.3 flash and Qwen 3.8 Flash Next, everyone's getting flashed and I think it's a good trend.

Almost suspect that the rate of improvement to post-training is so fast that small models have an advantage - it takes much more compute to train a bigger model, so the flash models are just running in circles (well, not exactly of course) around the larger models right now.

Page - https://web.archive.org/web/20260902151410/https://deepmind.... PDF Card - https://web.archive.org/web/20260902150007/https://storage.g...

it is most likely that 4 pro will be released pretty soon instead, since pre-training for 4 began in late July.

https://x.com/OfficialLoganK/status/2079594867161022817

Gemini 3.5 pro is never going to be released, it was a failure.

3.7 high and 3.8 medium are essentially the same on AA intelligence and cost. Output tokens on DeepSWE gives the same picture. So there might be something to it but they have done other things as well. At least the tokens are really fast.