People seem to think LLMs are sentient and not tightly coupled to server infrastructure OAI are in full control over. They literally have every single message sent and action taken, if they can’t monitor that during test runs they should find another job.

It’s like running a benchmark and forgetting to tune it and then wondering why production is running slow.