Dude it’s not trivial to switch because the behaviors are different!

You’re clearly not building a product based on an LLM.

I’m still using various old Anthropic and OpenAI models for products I’ve built and released because I can’t risk the behavior changing in unpredictable ways and the users being pissed.

It’s much easier to switch out some deterministic software than an LLM which you’ve spent a ton of time on testing and benchmarking and understanding its nuances. Changing it is like replacing an employee who’s critical to the business.

Anthropic has discontinued models in as little as 13 months from launch so if you do business with them switching can't be that big a deal?

I've been building multiple products with LLMs, and they are in fact interchangeable for the most part.

In fact, most benchmarks show this! Most benchmarks have similar performance for the same classes of models.

On top of this, there are tools like open router, or even the openai SDK which trivially allows you to swap endpoints for the LLM!

If you're using the agents SDK from openai or something, then yeah it's not interchangeable but that's you doing it wrong

The behavior of a single model and version can and does change. There’s not only built-in stochasticity, but closed hosted models like Claude are tweaked and changed all the time.

For the public facing consumer functionality I have Gemini Flash running on guardrails directed by a state machine that calls it statelessly everytime. For that, it's strictly locked to a version. I can't afford to suddenly get responses that the SM is not tuned for.

As for which model does the building... I'm not at all attached. Enough logic, and CI gates/tests live outside the whims of the LLM to be able to hotswap them any time.

I don't think they are saying it's trivial but compare say for example switching an organisation from Office or Windows the example that started this. They are not even in the same ballpark.

Can you give specific examples on what differences make it hard to switch?

Because this claim is counter to my experience as well.

Makes sense but honestly if you've spent more time testing and working around the nuances to build consistent experience doesn't it mean you actually need more standardization to easily switch models if/when your trusted model is not viable for you provider?