Remix.run Logo
▲ colinrand 3 hours ago

Completely agree. Things are moving too fast and so are breaking.

For example, we updated a GCP managed base image of Ubuntu that was released with 350 CVEs patched, some were as high as 9.8. In doing so, they (GCP) changed the DEFAULT setting for hardware GRO from off by default to on. That's a massive networking change packed into a security update.

We only caught it AFTER it started causing issues in production. We didn't have the OS pinned because (decently valid reasons) or catch it in non-prod (decent reason + human mistake).

This is on a modern highly automated tested resilient well architected platform.

I can only imagine these other platforms are dramatically more exposed that we are, hence these common large scale outages.