Homelab =/= Production systems

The gulf between these two insofar as what approach, technologies, and due-diligences are necessary is vast.

I think we've gone a little nuts defining "production system" these days. I've worked for companies with zero-downtime deployments and quite a lot of redundancy for high availability, and for some applications it's definitely worthwhile.

But I think for many (most?) businesses, one nine is just fine. That's perfectly doable by one person, even if you want, say, >=96% uptime, which allows for 350 hours of downtime per year. Even two nines allows for ~88 hours of downtime per year, and one person could manage that without much trouble.

Most businesses aren't global. Downtime outside regular business hours for your timezone (and perhaps one or two zones to the west and east of you) is usually not much of a problem, especially if you're running a small B2B service.

For a small business that runs on 1-3 servers (probably very common!), keeping a hot spare for each server (or perhaps a single server that runs all services in a lower-supported-traffic mode) can be a simple way to keep your uptime high without having to spend too much time or money. And people don't have to completely opt out of the cloud; there are affordable options for e.g. managed RDBMS hosting that can make maintenance and incident response significantly easier and might be a good choice, depending on your needs.

(Source: I'm building a small one-person business that is going to work this way, and I've been doing my research and gaming it out.)

▲

PaulKeeble 2 days ago | parent | next [-]

One thing that AWS, Google and Azure do that your own systems don't is release their updates whenever it suits them, often taking down your business down in the middle of the day with their own problems. You can't fix it, you can't rollback what you just did and get back up and running you just have to sit and wait.

That is quite different to a business that turns off its boxes for an hour at 0100 Sunday morning to do updates and release new software. The downtime isn't equivalent because it really matters when it is and if that hurts your use case or not. Your own system might be down for more hours a year than AWS, but its not down Monday to Friday on an evening when you do most your sales because you refuse to touch anything during that period and do all the work outside that and schedule your updates.

▲

immibis 2 days ago | parent [-]

It also feels like AWS (or Azure) isn't really that much more reliable than your own thing. But half the internet is down at the same time so you don't get blamed as much.

	▲	PaulKeeble 2 days ago \| parent [-]
		Its the "No one gets blamed for going IBM" thing in the modern era. They are making it someone elses fault and absolves the blame. The problem is if your competitor is still up you could be loosing customers on average mid day outage, even if they are down for 3x as long its not when it matters.

▲

1dom a day ago | parent | prev [-]

I agree. "production system" isn't a useful term without context.

Either it's production system in the context of "it's a business serving customers", in which case, there are many homelabs out there which have received paying traffic, or it's a production system in terms of functionality, downtime, technical features etc. Again, there are many homelabs out there that can tick all the same technical and performance boxes as "production system".

In the context of the original comment I was responding to ("...running and managing numerous DBs, k8s clusters, ci/cd pipelines and stateless container systems is simply impossible with a team of 1-2 people.") you're more evidence for my comment that it's perfectly possible for 1 - 2 people to run a small production system, just as it's possible for 1 nerd to run a medium - large homelab with the same technical features.

▲

1dom a day ago | parent | prev [-]

Your production is not my production. 5 9's of businesses don't need 5 9's, and in my experience, people who talk like they do are talking from a position of managing 1 or 2 production systems in a single domain.

My experience is that the largest regulated production systems I've run have clearly been larger and more complex than my homelab, but my homelab has been significantly more resilient, featureful and robust than many of the smaller production systems I've been responsible for outside of regulated domains.