Remix.run Logo
▲ SoftTalker a day ago

I'd start with "what has changed" but surely they've looked at that.

▲NetMageSCW 6 hours ago | parent | next [-]

They did look at recent changes but none seemed to be related, but it turned out recent network changes triggered a failure mode in a much older network change. Dependent or cascading issues remain hard to diagnose or find.

▲jamesfinlayson 21 hours ago | parent | prev [-]

Could be something adjacent - the last time I majorly blew up a release was because something had changed (which was fine) but it caused a list of items to be broken over two pages, but only in prod. Me and a team mate were scratching our heads for the longest time because "what had changed" was quite mundane itself.

▲sire-vc 14 hours ago | parent [-]

The classic 'database is smaller in testing than in prod' problem which is responsible for some outages IME. Often the 'query was not optimised but that didn't show until prod' one.