I love how there is a "Please do not discontinue gemini-2.0-flash[-lite], 2.5 is NOT an equivalent" from Feb 20th. Getting too attached to models is a smell.

▲

samuelknight 2 hours ago | parent | next [-]

It's not a smell. Why should these developers rebuild a core piece of their stack every few months. Switching out a model requires a new round of testing and validation when we should be able to rely on a piece of software the behave the same way since the last time we touched it.

	▲	recursive an hour ago \| parent \| next [-]
		> Why should these developers rebuild a core piece of their stack every few months? That's what they signed up for when established a hard dependency on an subscription online-only LLM model.
	▲	imhoguy an hour ago \| parent \| prev \| next [-]
		This is how development looks like for many years now, constant rewrite on the horizon. I think LLM development hype surpassed Blockchain and JS frameworks craze of decade ago.
	▲	vjsrinivas 2 hours ago \| parent \| prev \| next [-]
		Its almost a given considering how fast this field moves. Also, what kind of workflow structure would someone have that a single specific model is the only one that would perform acceptably?
	▲	askvictor 2 hours ago \| parent \| prev [-]
		It's kind of the same problem with cloud in general (though that moves much slower). If you want to be sure to be in control, then host it yourself

▲

dacox 2 hours ago | parent | prev | next [-]

the 1.5 and 2.0 flash models were absolute beasts. They were very cheap, and _very_ fast. We contemplated moving some of our fine tuned workloads to them because we would have gotten very substantial total latency reductions for our workloads.

However, they are aggressively deprecating them (OpenAI is as well), and replacing with newer models. These newer models are all reasoning models, and importantly, only bear the flash name. They are not fast. And they are very expensive!

	▲	imhoguy an hour ago \| parent \| next [-]
		I test workloads with multiple closed and at least one open model now. Good to have a backup on 503s or credits run out.
	▲	bjord 2 hours ago \| parent \| prev [-]
		have you experimented at all with the deepseek flash models?

▲

quentindanjou 3 hours ago | parent | prev | next [-]

In the post the issue is performance. Are you saying that getting too attached to performance is a smell? That sounds very odd.

It's not because a model performs better in some applications (often by fine-tuning to get better scores at specific tests) that it is better across the board or that we have to believe the company releasing the model with a high number 3 > 2 so that it is commonly accepted as better.

Pushing the reasonnning further: f you need an Opus level performance then not accepting GPT 3 isn't a smell.

	▲	hrpnk an hour ago \| parent [-]
		The smell is about folks being attached to the same model because they're used to it. Just like any UI change in an app that one uses everyday is annoying. What's often missing is the actual engineering. If you have evals for your use case, you use these to adjust to the new model. DSPy has great prompt engineering constructs it ships with. The smell is all about vibing, where a model feels better because the structure of the answers is more familiar to a person, instead of engineering where given constraints and input/outputs one is using/bending the system to fulfil the requirements.

▲

ekidd 2 hours ago | parent | prev | next [-]

We have benchmarks for our use cases, and every generation after Gemini 2.0 Flash has been a grim hit on price/performance. Costs have gone up, throughput has gone down, and performance has improved very slightly (and regressed on a few things).

▲

all2 3 hours ago | parent | prev | next [-]

That's like saying 'getting attached to locked dependencies for your app is a smell'.

But this could be framed as 'getting attached to an API revision when a new one is available'...

I can see it both ways, tbh.

	▲	3 hours ago \| parent [-]
		[deleted]

▲

data-ottawa 3 hours ago | parent | prev | next [-]

I built some BigQuery workflows on 2.0 and 2.5 flash lite that are something like 6x more expensive with 3.1 flash lite.

I tried 3 flash for months and it didn’t work using Googles own vertexai integration because it’s been in preview mode for months.

Not wanting to pay significantly more and do a bunch of rework isn’t a smell.

They left a large gap in their new pricing vs the prior generation, and if you had a working use case that sucks. The model is >99% reliable for my use case so there’s nothing to gain from a smarter model.

▲

2 hours ago | parent | prev [-]

[deleted]