SRE here, Strong disagree with do not fail readiness and liveness checks on upstream dependencies failing. There are several reason to do so and unless you have extreme start up time, what's the problem with restarting?

Maybe DNS has changed on you but you are stuck with bad local cache because you poorly respect TTLs (Looking at you Java), reseting the process will clear that cache away.

Maybe TCP connections are in stuck weird state, resetting the process generally helps with that.

Maybe someone gave you bad ENV VARs and you cannot connect to database, by refusing to progress the rollout, no outage generated.

So yea, if you are not ready to do work including critical upstream dependencies, don't lie to system and say you are.

▲

dilyevsky 2 hours ago | parent | next [-]

1. was already mentioned in sibling - cascade failures

2. you'll have massive number of restarts for various flake reasons and missing things that got papered over with restarts until you hit 1 and everything is broken. another popular version of this is "just restart when memory leaks too much"

▲

arccy 2 hours ago | parent | prev | next [-]

Thundering herd / cascading outages. You take out a large enough portion of your fleet, and the remaining load overloads your remaining nodes one by one as they restart, so you can never have enough healthy nodes.

▲

erulabs 2 hours ago | parent | next [-]

SRE team debates correctness versus availability for the 540th time this year

You're both correct, of course!

	▲	atmosx 2 hours ago \| parent [-]
		What this guy said :point_up: My personal take-away is this: whatever you choose, make sure it's consistent across services (not serviceA behaves like X and serviceB like Y) and make sure eng teams know _how_ these are configured and what can go wrong. They'll figure out the rest.

▲

jaggederest 2 hours ago | parent | prev [-]

That's a problem for circuitbreakers on these kinds of actions, not lying on health checks.

Something like healthcheck fails -> restart -> healthcheck fails -> restart -> healthcheck fails -> circuit breaker trip, alarm raised, give up until manual intervention or X minutes have passed

▲

deathanatos an hour ago | parent [-]

That circuitbreaker exists, by default. It is "CrashloopBackoff", here, and TFA covers it. (& it's an "until X minutes have passed" kind, by default.)

	▲	dilyevsky 41 minutes ago \| parent [-]
		backoff is only applied to individual pods/containers not across pods. the point is at scale it's easy to get into a situation where it's not possible to recover without (usually manual) full service drain

▲

figmert an hour ago | parent | prev | next [-]

My favourite: misconfigured Linkerd setup that causes CA certs to rotate every month :) Definitely worth restarting on that

▲

peterabbitcook 2 hours ago | parent | prev | next [-]

What are your feelings about using initContainers and wait-for-it to skirt the thundering-herd problem?

▲

connicpu an hour ago | parent | prev | next [-]

The better solution is to not have too many critical upstream services :)

▲

cmckn 2 hours ago | parent | prev | next [-]

> what's the problem with restarting?

Exponential backoff can delay recovery up to kubelet’s maxContainerRestartPeriod (default 5m).

▲

javier2 2 hours ago | parent | prev [-]

cascading failures on upstream services. then you get 20 different services failing instead of the single one.