|
| ▲ | dilyevsky 2 hours ago | parent | next [-] |
| 1. was already mentioned in sibling - cascade failures 2. you'll have massive number of restarts for various flake reasons and missing things that got papered over with restarts until you hit 1 and everything is broken. another popular version of this is "just restart when memory leaks too much" |
|
| ▲ | arccy 2 hours ago | parent | prev | next [-] |
| Thundering herd / cascading outages. You take out a large enough portion of your fleet, and the remaining load overloads your remaining nodes one by one as they restart, so you can never have enough healthy nodes. |
| |
| ▲ | erulabs 2 hours ago | parent | next [-] | | SRE team debates correctness versus availability for the 540th time this year You're both correct, of course! | | |
| ▲ | atmosx 2 hours ago | parent [-] | | What this guy said :point_up: My personal take-away is this: whatever you choose, make sure it's consistent across services (not serviceA behaves like X and serviceB like Y) and make sure eng teams know _how_ these are configured and what can go wrong. They'll figure out the rest. |
| |
| ▲ | jaggederest an hour ago | parent | prev [-] | | That's a problem for circuitbreakers on these kinds of actions, not lying on health checks. Something like healthcheck fails -> restart -> healthcheck fails -> restart -> healthcheck fails -> circuit breaker trip, alarm raised, give up until manual intervention or X minutes have passed | | |
| ▲ | deathanatos an hour ago | parent [-] | | That circuitbreaker exists, by default. It is "CrashloopBackoff", here, and TFA covers it. (& it's an "until X minutes have passed" kind, by default.) | | |
| ▲ | dilyevsky 20 minutes ago | parent [-] | | backoff is only applied to individual pods/containers not across pods. the point is at scale it's easy to get into a situation where it's not possible to recover without (usually manual) full service drain |
|
|
|
|
| ▲ | figmert 35 minutes ago | parent | prev | next [-] |
| My favourite: misconfigured Linkerd setup that causes CA certs to rotate every month :) Definitely worth restarting on that |
|
| ▲ | connicpu an hour ago | parent | prev | next [-] |
| The better solution is to not have too many critical upstream services :) |
|
| ▲ | peterabbitcook an hour ago | parent | prev | next [-] |
| What are your feelings about using initContainers and wait-for-it to skirt the thundering-herd problem? |
|
| ▲ | cmckn 2 hours ago | parent | prev | next [-] |
| > what's the problem with restarting? Exponential backoff can delay recovery up to kubelet’s maxContainerRestartPeriod (default 5m). |
|
| ▲ | javier2 2 hours ago | parent | prev [-] |
| cascading failures on upstream services. then you get 20 different services failing instead of the single one. |
