Currently we are building agentic systems for various business functions like Legal and Finance. How these models perform in those tasks is paramount, and things like cost per run is only a part of the equation. It can become very subjective, and things like brand recognition comes into play. The cost of a counsel or a VP finance overshadows the savings you get by micro-optimizing for a cheaper model. We tend to use the best-in-breed LLM for the various tasks - mainly Claude and Gemini, and sometimes GPT.

> How these models perform in those tasks is paramount,

Then you have no choice but to use your own hosted model because that's the only way you can be sure the model isn't going to be removed or quantized/nerfed behind your back.

If your system can widstand the fluctuations of closed models over time, then it can most likely support switching between multiple providers in the first place.

If you ever need to scale to millions of customers, then cost per task is paramount. If it’s a one-off vibe coded custom project for one client, then best-in-breed is mandatory without it falling apart.

Ur case actually strength the argument for open weight self hosted model. You would have precision control over not only the technical issues, but also the legal issues.

Once the open weight models are good and cheap enough, there is no reason not to use them

Spending money in the most effective places is smart... and then there are the stupid businesses.