Remix.run Logo
jaggederest 2 hours ago

That's a problem for circuitbreakers on these kinds of actions, not lying on health checks.

Something like healthcheck fails -> restart -> healthcheck fails -> restart -> healthcheck fails -> circuit breaker trip, alarm raised, give up until manual intervention or X minutes have passed

deathanatos an hour ago | parent [-]

That circuitbreaker exists, by default. It is "CrashloopBackoff", here, and TFA covers it. (& it's an "until X minutes have passed" kind, by default.)

dilyevsky 39 minutes ago | parent [-]

backoff is only applied to individual pods/containers not across pods. the point is at scale it's easy to get into a situation where it's not possible to recover without (usually manual) full service drain