Remix.run Logo
cadamsdotcom 2 hours ago

Worth mentioning CI is not part of production outages!

You should never fix an incident by deploying another fix as it risks making things worse especially while everyone is panicking. Better to roll back - and spread non-backwards compatible work across multiple deploys that each are backwards compatible - always giving yourself a known good state to roll back to.

Rollback should not done be via your CI either! Set up some low dependency thing - a separate tool or script or system.

Once you've rolled back and got the system stable, then produced a fix at your leisure, that's when CI comes back into the picture :)