I’m curious how you guys keep track of each model’s coding capabilities. The landscape keeps changing. I don’t suppose you benchmark all frontier models every other month, right?

▲ riddlemethat 5 hours ago | parent [-]

I use them. Daily. Gemini hasn’t been a contender by comparison for a long time.

▲ safog 4 hours ago | parent | next [-]

I wonder if it's a harness thing or a model thing at this point. I feel all coding models are quite capable for most tasks I want them to do.

Most of the time I don't need what the bench tests and I'm not really giving them completely ambiguous tasks without any refinement.

I only find marginal differences between models at this point and it almost feels like personality quirks in each model than anything.

▲ BenzeneDream 4 hours ago | parent [-]

When comparing OpenAI and Claude thats pretty much true, but not Gemini... And have you tried Antigravity? Yikes

▲ vrosas 3 hours ago | parent [-]

The CLI version of agy is great. Have you tried it?

▲ taylorfinley 3 hours ago | parent | next [-]

Do you dangerously allow permissions? I absolutely cannot use it until they ship an auto approver. As it is now I have it write one bash/python script to do everything it wants to, then I review that. Otherwise it is COMPLETELY unusable and it shocks me when I hear people are using it.

	▲	xnx 2 hours ago \| parent \| next [-]
		Sounds like they shipped some changes today that might reduce approvals: https://x.com/antigravity/status/2100001904969297980
	▲	andai 2 hours ago \| parent \| prev [-]
		`alias agy="agy --dangerously-skip-permissions"`

▲ VectorLock 24 minutes ago | parent | prev [-]

Compared to gemini-cli that they took out behind the woodshed, I hate it.

▲ baq 5 hours ago | parent | prev | next [-]

True but it has a niche in SQL reviews for me. Looks like Google has a lot of good sql in their corpus and in their RL digital lobotomy factory.

▲ le-mark 3 hours ago | parent | prev | next [-]

I did a test involving implementing cobol control flow in Java for a source to source translation project. Gemini was the only model to get the edge cases. Cobol is very peculiar in this regard.

	▲	robotmay 3 hours ago \| parent [-]
		It's very good at Elixir in my experience too. And it just does what I ask and doesn't wind me up like Opus. I don't think I've had to insult it more than once per day.

▲ adventured an hour ago | parent | prev [-]

I had a typical $20 Gemini plan that I just downgraded to their $5 plan (to keep access to some of the models). It had been so long since I let Gemini work on (or review) any code / design / html (anything) that I couldn't justify bothering to keep wasting money on it. It fell behind badly over the past year. Astra might as well be an alien super intelligence at code compared to Gemini. I enjoy talking to Gemini, it is very good at conversation, I get solid answers to everyday questions. I intend to keep the $5 plan indefinitely for basic use. I don't expect they'll ever resurface as a competitor in coding with Astra & Fable et al.