> for example, I've seen N+1 problem sneak past because at the time the performance was good enough but once there was enough data it crumbled

This is a super-common problem I've had to help with many times. It generally happens with teams working on single-tenant (or single-database-tenant) products, and basically always comes back to the dev team working on a database with hundreds of things, and there's a handful of customers using 10,000+ things when it starts getting slow.

You acquire a bunch of small customers and everything is great, and it can months later before you really start to see performance impact. And often with these types of problems it just gradually slows down until finally the system hits a wall where the DB can no longer compensate and suddenly performance tanks. "It must be caused by something we just added, but it's weird, because this problem is in a totally different part of the system."

It's one of the things where I've found code review to be the most helpful remedy. If everything gets reviewed then N+1 queries can be rejected before they even hit the main branch. They're usually pretty obvious if you're looking for them.