Remix.run Logo
toast0 an hour ago

> Services where uptime matters tend to be designed so they can tolerate the reboot of a single node for other reasons besides kernel maintenance. I can't imagine a situation where I can't tolerate the downtime of a reboot but I would be willing to risk the system locking up with brain surgery gone wrong.

I've run systems with live code updates for userland, and would have considered live kernel updates if it was reasonable on our systems.

The thing is you typically build your system to tolerate reboot or unscheduled stop of a single node. Scheduled stop is nicer, but systems sometimes lock up even when you're not doing risky behaviors, so you know.

But just because the system can tolerate a reboot or restart doesn't mean it's not disruptive. A lock up / etc during hot load is also disruptive, of course. But when you can push code without having to stop anything, with limited impact on users, it makes it easier and faster to do updates. You can use whatever rollout pattern you like to contain risk too; same as you would for an upgrade with restarts.

For us, we have servers with hundreds of thousands or millions of tcp connections from mobile clients. Restarting a server would make all those clients have to reconnect and connecting is expensive. Restarting all the servers would result in many clients reconnecting several times. It was better to avoid it when possible.