Have you tried a package like LiteLLM so that you can more easily validate and switch to a newer model?

The key seems to be in curating your application's evaluation set.

I'd love something like litellm, but simpler. I'm not provisioning models for my organization, I don't need to granularly track spend, I just want one endpoint to point every tool or client at for ease of configuration and curiosity around usage.