Interesting — I hadn't tracked the hours on my side. A small community tool to collect this would help. The hard part is separating "the model got nerfed" from "my prompts don't fit the new behavior anymore". Think downdetector for LLMs, but based on real metrics instead of user reports. Opt-in client wrapper, anonymized telemetry, public dashboard. Does it exist already? I just searched and couldn't find anything.

I'm not aware of a tool that does this yet. If you find it or build it, please let me know as I would love a tool with that functionality.

And I'm sure I'm not the only one.