It is changing so often that to keep the costs and quality at a pareto level you need to experiment with a lot of different providers. And the pain you are describing should be a normal process in AI. I use model pools where the tasks are optimized for multiple AI providers. For example I was experimenting with GPT Luna and it turned out that the model is good but tool shy so I had to improve the instructions. Now this model is my main model for the chat in my app. Next week it can be a different model.

It might just not be for you. I use OpenRouter because I do like being able to quickly check whether a new model works better, but mine is a “human in the loop” dev process so it won’t ruin a day of batch processing or anything.

I do like being able to try eg GLM without having to set up a new account. It’s also nice that I don’t have to top up per-model accounts. I think I have a couple accounts with $7 in API credits sitting around.

Probably not something I would do for actual business processing, where the stability is dramatically more important than tinkering with new models.