new | show | ask | jobs Github

andai 7 hours ago

Wait, I didn't realize 3.7 Flash was already beating Sol on a bunch of the benchmarks. Isn't it a way smaller models?

▲

ipsod 7 hours ago | parent | next [-]

IDK if it's smaller, but I know it's way faster. In one test I did, Flash 3.7 high was ~9.4x faster than Luna High.

But, also... Sol crushes Flash 3.7 at writing code in a codebase of any size beyond "tiny".

Flash is my go-to for prototyping, and basically anything that isn't writing production code.

▲

ramon156 7 hours ago | parent | next [-]

The only company with a proper TPU set-up is bound to have the fast models, now add a market cap like Google to the mix.

▲

ipsod 7 hours ago | parent [-]

They've been my bet to win the AI race for a while. I was starting to doubt, but this 3.6, 3.7, and 3.8 arc has anchored me.

▲

Diederich 5 hours ago | parent [-]

Totally agree. When this current wave of GenAI really started heating up, I guess 2020-2021, my analysis was very straightforward. What are the high level inputs to long-term success? I basically came up with a couple of criteria:

1. Data. Lots of data.

2. Money. Lots of money.

3. Access to necessary hardware.

4. Business alignment/will to do it.

5. Access to talent, current and future.

This is certainly incomplete/naive. In my mind, though, Google was the clear answer.

On a more personal level, I've been deep into the Google ecosystem since I got diederich@gmail.com in 2005. (I actually paid 50 cents on ebay to get a very early invite.) There was no question in my mind that Google's AI work would deeply integrate into their whole ecosystem in very powerful and productive ways. (Yes, I can join you to discuss, at length, the various ways that Google's dominance is problematic/scary.)

Having said all that, I'm quite happy that there is, at the moment, a very rich competitive landscape. Indeed, not too long ago, with Gemini Pro 3.1 languishing, I moved most of my deeper thinking work to ChatGPT, which was, for me at least, clearly outperforming Gemini.

While I certainly didn't anticipate it, Google's strategy of making their fast/relatively inexpensive models surprisingly powerful has been a welcomed surprise.

	▲	ipsod 4 hours ago \| parent [-]
		Not only do they have access to the hardware - they've been developing it in-house for years.

▲

momojo 6 hours ago | parent | prev | next [-]

Same. Love oneshotting or sanity checks. Which fortunately is a lot of my workflow (lot of long tail stuff fits in one prompt).

▲

esafak 7 hours ago | parent | prev | next [-]

Luna is way slow. I don't remember an OpenAI model ever being this slow.

edit: I have a subscription; direct call.

	▲	dannyw 7 hours ago \| parent [-]
		Are you using direct or via OpenRouter? I think OpenRouter Luna always uses the `flex` tier, which is quite a bit slower.

▲

MrBuddyCasino 4 hours ago | parent | prev [-]

Its not good at not making mistakes, but what it produces is structurally quite nice, not over-engineered (looking at you Sol) and its personality isn’t annoying (looking at you Claude). A bit like Grok Code, but Grok is a better coder.

▲

realist_not 7 hours ago | parent | prev | next [-]

It's pretty good if you can actively steer it , its actually really really good , the antigravity free tier and pro tiers are generous as well . I'm shocked at how fast it generates tokens.

	▲	worldsavior 7 hours ago \| parent \| next [-]
		Some would say it's Google's TPUs.
	▲	MrBuddyCasino 4 hours ago \| parent \| prev [-]
		Can the free tier be used outside Antigravity CLI? Because its security prompts get old pretty quick.

▲

refulgentis 7 hours ago | parent | prev [-]

They're quite selective in benchmarks, c.f. notably only bad one is 10% on TerminalBench. It's a really addled model, one time I said "Hi" and it built out a 4 panel hello world app with (fake) weather, a todo list, and a couple other things I forgot. I wouldn't be comfortable saying "ignore the #s!" except when I complained it was trash and way overcooked on agentic coding yet not good at it, and a couple DeepMind ML people liked the tweet.

	▲	zuzululu 2 hours ago \| parent [-]
		i discount people who lean too heavily into benchmark as the authoritative truth when it comes to evaluation of coding capability of these models. experience tells me that those people simply have not used models for a long period of time specifically on coding and have run their own comparisons to someone who uses all vendors, the differences are very palpable and drives purchase decisions. also keep in mind Gemini and other labs have repeatedly done benchmaxxing, you must have your own benchmarks to evaluate these models.