If you're developing on top of LLM APIs directly, this is definitely not true. There are differences in how context caching works, in what's available through native harnesses, the types of tools you're fine-tuned on (GPT uses apply_patch while Claude uses edit, with different formats), the API surface (Agents SDK, Responses API, Managed Agents), cost structures, and best-practice guidance all around.

Not to mention the meta of account limits, billing, ZDR contracts, etc.

It really depends on what you're doing, but most LLM usage and agentic runs are pretty interchangeable in my experience, and it's usually trivial to switch.

If anything, you're better off supporting multiple LLMs as backup because most model providers have been so inconsistent with working all the time

just use your agents to do the migration, that's what it's good at.

Exactly, as in, really, will they? Where and at what price, especially across an actual enterprise that needs to deploy them to lots of devs? There's much more than just the actual model.

Of course my numbers are a sample of one and I am not spending a lot of money or time on it. Just lazily trying things on my "happen to have this" hardware. But basically trying out the Claude Code I'm used to from work but locally with a bunch of open weight models.

I can run super tiny models on my 8GB NVIDIA card. They all suck (I have to use <=~5GB models if I want "usable" ~250k context that doesn't need to use system RAM and CPU (which makes things super slow).

I've also tried a GLM 4.7-flash, which even though it's super slow (in comparison) with ~250k context and it just doesn't cut it vs. the Claude Sonnet or Opus I get to use at work. All the while these are all touted as "totally usable, Claude/ChatGPT killer!" replacements.

It's just not "there" with tool use or building software for that matter. Like, just a simple Claude "web search" fails with it. So I asked it to build itself its own "web search" functionality and it just couldn't. It made so many mistakes its just not funny any more. And it couldn't recover from them either. I retried a few times (as I didn't have python installed and it wanted to implement it using that - this happens to be new system - never mind other attempts). I spent as much time doing this (and failing) as I spent building an actual full feature at work last week w/ Sonnet.

If it can't build itself a simple web search to .md file tool/skill, how am I supposed to trust this with actual coding? I'm used to being able to point Claude at our large code base and essentially work with it like a junior doing my bidding. Maybe 5.2 is a killer game changer vs. what I was able to try out (if slowly) but you really have to show me to convince me at this point. And not with synthetic benchmarks. In those, all of the models I tried are supposedly super awesome.

This is a meme and massively over-complicating what is ultimately quite simple.