https://marginlab.ai/trackers/codex/

By and large they don’t. I have seen this drop a few times, eg before fable came out opus dropped a lot probably due to less compute available.

My guess is it’s a combination of getting used to the new cliff models fall off on and forgetting that model performance drops significantly when context fills up.

So new model comes out, people try it and it’s amazing on a task or two. Then they start using it, context window fills up and it gets a lot worse.