You expect to not be responsible for what happens to the software you put into production?

(… and I'd like to avoid distracting arguments that amount to "my company does on-call badly" — yeah, those problems do exist and we should strive to fix them. But if I'm to not categorize the argument here as the baby with the bathwater, then we need something to replace on-call with. Prod goes down on a Saturday afternoon; are you going to tell management "tough cookies" until Monday?)

> You expect to not be responsible for what happens to the software you put into production?

First: IT seems to be rather the exception - most professions have no on-call. Eg. even if my car mechanic screws up a service job, they'll have me bring the car back into the garage during their normal working hours, regardless of how and where stranded I am in the middle of the night.

A second comment: I'll be responsible for anything I have created in my own way. The reality of software development is that we implement functional requirements we've been given with which we disagree, we implement non-functional requirements which don't achieve the goal, we are made to use frameworks and tools we're not familiar with, on a short timeline, a low budget and inadequate infrastructure and we're supposed to take responsibility for code our co-workers wrote.

My boss recently started an on-call rotation for us. None of the code I have written is customer facing. If everything I wrote breaks at 5:01pm on Friday, external customers will feel 0 impact if I wait to fix it until I show up again on Monday. Worst case, someone internal has to wait to work on something they’ve probably been putting off for months anyway. There are other things they can work on. If it was a constant problem, I’d get it, but a rare instance can be forgiven when no outside impact is felt.

I am responsible for my code, but we need to be realistic about the impact. Not all outages are created equal.

I used to work nights watching over the hardware, operating systems, and applications running in it. We’d do upgrades and break/fix stuff. Some things were worth waking someone up for, but a lot of things weren’t. We’d do what we could do fix it on our own, but for a non-prod environment, it could wait until morning if we couldn’t do it on our own. This idea seems to be lost on people now. I get that 100% uptime of 100% of the systems would be nice, but not at the expense of your employees sanity.

I haven’t actually been called yet with the new rotation, but any week I’m on-call I’m a bit on edge. In the past I had some pretty horrible on-call experiences that pushed me close to quitting, which I won’t get into, so I’m preparing for the worst. I worked my ass off to get into a position where I didn’t need to be on-call and put in my time working nights so other people could sleep. Being back on-call feels like a demotion.

“Devops” traded less bureaucracy for more accountability.

Have a generalist ops team that is staffed 24x7, or has paid on call as part of the job. They get run books to respond to whatever goes on.

I’ve set this up twice. The first time, we had a team in the Philippines that would cover overnights.

They could start and rollback deployments and do most stuff via the runbook they were provided. Most callouts (5% of escalations) to product teams were due to bad or missing documentation.

The US based team did similar work, just during the day. Both could escalate quality issues for the product team to fix.

The other model was all US, on-call based. We used junior and low-skill folks, who had rotating on-call. They were paid 20% of hourly rate for standby pay and had a minimum pay threshold when they got called. All of that hit the cost center of the offending product or service, so there was both a financial incentive to not get calls, and a human incentive as the engineers didn’t want to get called for escalations. Again, documentation is key.

>> You expect to not be responsible for what happens to the software you put into production?

I'm responsible for the software I put into production from 9 AM to 5 PM for about 200 days a year. At 3 AM, I am responsible for taking care of myself by getting a good night's sleep.

If you need 24 hour coverage, taking into account vacations and weekends, you need 5 or 6 people.

I don’t put things in production, the company does. And it’s the companies responsible to deal with problems that show up.

24/7 coverage is expensive and mandating someone is on call 24/7 don’t actually provide it.

I am capable of writing very good software, testing it, and putting it into production, but I am not capable of being responsible for what happens at 3am on a Sunday. Whether that deal is okay is up to you. I'm okay if you don't want to pick me. There are other jobs I can get. I write good software though.

If the customer is awake at 3am on a Sunday, it's the customer's problem that they were awake at 3am on a Sunday. If it's a social network, I frankly couldn't care; the customer should go to bed. If it's going to be deployed in the emergency room, fine, we should care, but YOU, management, should find people who are actually willing to take that shift (for extra money, or are based in other time zones).

Why do I give a single shit about the software having an issue at 2am? I don't own the company. I don't care. If they care they can hire night shift triage.

I expect to be very responsible during my working hours.

I have never worked for a company where people building the software and people supporting it when it is critical were the same. The idea is weird to me.

Plus any large enough company should have team in spread out timezones eliminating the need for on call if it’s correctly managed.

Only a disfunctional company would rely on the programmer who wrote the code.