| |
| ▲ | Geezus_42 a day ago | parent | next [-] | | We do for clients who have 24/7 users during the week. | | |
| ▲ | selcuka a day ago | parent | next [-] | | I don't think the hesitation to push to prod Friday afternoons has anything to do with the working schedule of your customers. It's more about whether you'll be able to roll back on a weekend. | | |
| ▲ | james_marks 14 hours ago | parent | next [-] | | We avoided Friday deploys because we didn’t have enough usage over the weekend; traffic was oriented on M-F working hours. It meant that issues would sit undiscovered over the weekend until Monday morning, and lead to a week starting in chaos. Much better to get fast feedback while the deploy could be monitored against prod traffic. | | |
| ▲ | hinkley 12 hours ago | parent [-] | | We did Monday morning IST, on a build we declared valid on Friday afternoon - no pressure to declare premature victory there. The cherry on top was sprint demos first thing in the morning PST so when I first joined the project the demos got interrupted on a regular basis. I don’t think most people fully appreciated how much work a handful of us did to make sign off faster and deployment less error prone. |
| |
| ▲ | DANmode a day ago | parent | prev | next [-] | | or want to have to. | |
| ▲ | Geezus_42 a day ago | parent | prev [-] | | Why wouldn't you be able to rollback on a weekend like any other day? | | |
| ▲ | ezfe a day ago | parent | next [-] | | cause I don't want to work on the weekend. Why would I introduce that risk? | | |
| ▲ | kevin_nisbet a day ago | parent | next [-] | | Or if you need more resources, can be a lot harder on a friday evening or weekend to get ahold of someone else to help. | | |
| ▲ | hinkley 12 hours ago | parent [-] | | Assuming they don’t have an actual life and aren’t somewhere touching grass. I’ve worked with several people who either confessed to or were accused of having taken up pastimes that would guarantee they weren’t available during times they shouldn’t be expected to be available. Management gets rewarded for getting more work out of us and there’s always someone pushing boundaries trying to leach a little more work out of the staff they have. These same people seem to think burnout is a big mystery, because to admit they caused the problem would be to admit they did something wrong. |
| |
| ▲ | Geezus_42 17 hours ago | parent | prev | next [-] | | Because you are patching a critical system that can't be down during the week. | | |
| ▲ | ndsipa_pomu 14 hours ago | parent [-] | | In which case, why wouldn't you deploy it during the weekend? That way, you're already prepared to be working several hours on a Saturday (presumably earning overtime rates) and so it's no major issue to rollback if it goes wrong. Friday deployments make no sense if you have teams that work Monday-Friday. | | |
| ▲ | Geezus_42 7 hours ago | parent | next [-] | | Friday afternoon is not unusual if you are facing a weekend deployment. Might as well try to get it done Friday afternoon an not have to worry about it the rest of the weekend. | |
| ▲ | hinkley 12 hours ago | parent | prev [-] | | Several hours. That number goes up quickly when the bus number for something that broke unexpectedly is unavailable. While your coworker who actually knows what that error means is at the zoo with his girlfriend or with his kids. The whole point of WLB is that we don’t live like monks. | | |
| ▲ | Geezus_42 7 hours ago | parent [-] | | Why do you only have a single co-worker that knows that error? Why hasn't anyone written shared documentation? | | |
| ▲ | hinkley 6 hours ago | parent [-] | | Outages are caused by change. Sometimes user traffic, but more often your coworkers. Do you know how every feature toggle in your system works? Do you know the potential blast radius of those toggles? Can you speculate on the surprise surface area that might be additionally exposed by impedance mismatches and side effects (ie, unanticipated shared state)? Hell does every engineer who wrote a feature toggle know this about their own code? You know who I’m talking about. If you’re having problems in production that can be anticipated, why are you still having those problems? How long should those be tolerated before someone gets fired? |
|
|
|
| |
| ▲ | Rapzid 19 hours ago | parent | prev | next [-] | | Thursday is pretty close to the weekend. Too risky on Thursday? | | |
| ▲ | hinkley 12 hours ago | parent | next [-] | | Scrum originally recommended people run sprints from Wednesday to Wednesday so you were doing deployments in the middle of the week. I’ve never worked anywhere that did a Scrum right, and I don’t mean that in the “well then you’re doing it wrong” sense. I used Kanban before Scrum and man is that a downgrade. | |
| ▲ | tuwtuwtuwtuw 18 hours ago | parent | prev | next [-] | | I'm old, I do prod changes Monday to Wednesday for that reason. I'm not paid to work on weekends. Sure, you can argue that people should be competent enough to never release software with a bug, but I sometimes fuck up. | | |
| ▲ | Geezus_42 17 hours ago | parent [-] | | Or just have a good rollback plan like a snapshot. | | |
| ▲ | tuwtuwtuwtuw 15 hours ago | parent [-] | | If the system I work on needs to be restored from a snapshot then I'll definitely be spending the weekend working. There are systems you can't just rollback and restore a snapshot and all is fine. | | |
| ▲ | hinkley 12 hours ago | parent | next [-] | | And then there’s the Tower of Hanoi of breaking major changes into tiny pieces that can increment without making rollbacks impossible. Always two more steps than you would think you need and requires patience that management often doesn’t possess. Example, got a feature with a new required field? Add empty field, synthesize a default value for existing rows, start accepting data, validate that data, require the data, use the data. If you’re doing deployments once a week, you end up juggling two stories so you’re not sitting on your hands the whole time, and two stories divided your attention, so the odds of fucking it up go up anyway even though you’re being careful on paper. Meanwhile one of your coworkers is doing the same dance on another feature, so you start tripping over each other as team size increases. | |
| ▲ | Geezus_42 15 hours ago | parent | prev [-] | | So when you roll a patch out on a Tuesday morning and it breaks, how much revenue do your client's lose? I'm all for telling a client to eat it because I'm not working on the weekend for some BS, but there are critical systems that CAN'T be offline during the week. |
|
|
| |
| ▲ | ndsipa_pomu 14 hours ago | parent | prev [-] | | Thursday is a good choice, especially if Fridays are less busy than other days as that gives the team a day to work on the issue if necessary and minimises the business impact if it goes bang. |
| |
| ▲ | dzhiurgis 21 hours ago | parent | prev [-] | | It’s ok, claude can do it for you. |
| |
| ▲ | latentsea a day ago | parent | prev | next [-] | | You can, but it also means you might have to. | |
| ▲ | schiffern a day ago | parent | prev [-] | | Not a family man, I see? | | |
| ▲ | Geezus_42 17 hours ago | parent [-] | | That has nothing to do with it. I don't like working weekends anymore that anyone else. However, every now and then it is necessary for certain systems. Also, if your rollback procedure is go, it isn't issue. | | |
| ▲ | schiffern 15 hours ago | parent [-] | | >I don't like working weekend [sic] anymore [sic] that [sic] anyone else.
So why the feigned confusion above? It seems you already knew the answer to your own 'question.' :P | | |
| ▲ | Geezus_42 7 hours ago | parent [-] | | My confusion is people acting like it is blasphemous to deploy on a Friday when I have laid at multiple reasons why it may be necessary. |
|
|
|
|
| |
| ▲ | asveikau 7 hours ago | parent | prev | next [-] | | I think it's not just inconvenience for users but also availability of engineers to look at failures. Many organizations do have round the clock staff but maybe they have the most staff during 9-5 on a particular time zone. So end of day Friday is guaranteeing you'll get a different shift of worker diagnosing failures, i.e. not the author of the deployed change, and possibly fewer available people. | |
| ▲ | hdgvhicv 20 hours ago | parent | prev | next [-] | | We have systems who’s main use in at the weekend. Other systems are only used overnight. Some the quietest part is 0930-1230 on weekdays. The reason you don’t want to push on a Friday, or overnight, or weekend, is your experts and the top bosses tend to work m-f 9-5, so pushing early in the day and early in the week ensures more people are available to fix problems and run comms. | | |
| ▲ | Geezus_42 17 hours ago | parent [-] | | There is someone on call at all times at every level in my company and we are not very big. It's rare I see anything I can't fix, but if I do, it's easy to click that button that makes the alarms go off and they'll be on the call in minutes. | | |
| ▲ | fn-mote 12 hours ago | parent [-] | | Is the guy on call your best troubleshooter? No. That guy doesn’t work weird hours because you’d have to be paying him $50k more to put up with that BS. It’s easy to think things aren’t going to go to hell, but look at the huge companies that screw things up. Amazon. GitHub. It could happen to you, too. | | |
| ▲ | Geezus_42 7 hours ago | parent [-] | | Actually, we ALL work on call. No one gets out of it just because they have seniority. Plus, we have an excellent team of smart people who are able to figure most things out on their own through use of their own knowledge and troubleshooting skills, aided by both internal and external documentation. It is rare that escalation is required. I never said things don't go to hell sometimes. That's why you build a good team and train your people. |
|
|
| |
| ▲ | pferde 20 hours ago | parent | prev [-] | | So... it's 24/5, then. | | |
| |
| ▲ | mlhpdx a day ago | parent | prev | next [-] | | It happens. My company has done a couple dozen production rollouts today, including minutes ago. | |
| ▲ | camcil a day ago | parent | prev | next [-] | | In public utility we only did deployments on Fridays after 5PM | | |
| ▲ | shadowgovt a day ago | parent [-] | | There's a bifurcation in best practices. For industries that serve customer bases that are down-cycled on weekends, the dominant effect is "Deploy on Friday to minimize customer disruption chance." For industries that operate with a 24/7 uptime expectation, the dominant effect is "do not deploy on Friday so you don't have to pay for your expensive software engineers pulling extra hours on a weekend. Much better to have disruption while the folks most qualified to fix it are in-house and alert." |
| |
| ▲ | ElProlactin a day ago | parent | prev | next [-] | | Team America: World Police | |
| ▲ | saturn8601 a day ago | parent | prev [-] | | Uh I'd imagine any team that mainly has their user base using the product on weekdays would deploy Friday evening so if something goes wrong they have the weekend to fix it. | | |
| ▲ | paulhebert a day ago | parent [-] | | Only if the team wants to work weekends? If something goes down on Saturday there’s a good chance I’m out of cell service | | |
| ▲ | shadowgovt a day ago | parent | next [-] | | It is interesting to see this bifurcation in action. Hey software engineers - you know that working weekends is unusual, right? Like... please tell me someone has at least negotiated your right to only work 5 days a week so if you're working a weekend you get Monday and Tuesday off. If not... Has anyone told y'all you should unionize? | | |
| ▲ | hdgvhicv 20 hours ago | parent | next [-] | | Many people work shifts, it’s quite normal. What seems to be unique in American companies is you’re expected to work shifts or nights but also work m-f 9-5 at the same time, which is really weird. Same with of call, if you get called at 3am on an on/call rota for a specialised third line escalation many companies still expect you to work the next day, and they might not even pay you for being on call?! | |
| ▲ | saturn8601 a day ago | parent | prev | next [-] | | I mean, you make it sound like releases fail often. For our team I think we have only seen one failure in the last two years needing a fix after hours (going into the weekend)? The other option is to fail during the workweek. If the software is directly generating money for the users and the developers are judged on what actual business value they provide then failure during business hours is worse. | |
| ▲ | cowlevel a day ago | parent | prev [-] | | Did you know that when you use your 2 days off to go shopping, someone has to work at the shops? | | |
| ▲ | shadowgovt 14 hours ago | parent [-] | | Yes. That's why people work staggered schedules. Folks who work weekends at shops usually (and should) have two other days off in the week. People literally fought, bled, and died for our right to not do capitalism 24/7/365, because with no back-pressure that's what the efficiency engine will demand of us. | | |
| ▲ | cowlevel 13 hours ago | parent [-] | | And part of that is software engineers doing deployments after-hours for their customers. If your company or country is any good, they'll give you double time off during normal hours. |
|
|
| |
| ▲ | saturn8601 a day ago | parent | prev [-] | | Such a smug privileged answer given in a world where the industry is shrinking its workforce and pretty much all other industries have flatlined for years(meaning we are still better off despite this realities). | | |
| ▲ | paulhebert 13 hours ago | parent [-] | | Wanting to have a weekend is smug and privileged now? People fought hard for the 40 hour work week and 2 day weekend. You think that should be given up? | | |
| ▲ | saturn8601 6 hours ago | parent [-] | | Oh, I know all about that history. Having been a volunteer for Bernie Sanders back in 2015, I admired him as being one of the remaining few people who actually remembered this history due to him being part of the Silent Generation and continue to fight for it decades later. >You think that should be given up? Not really what I said. My point was that in an era of transition where we started with a 40 hour workweek and a two day weekend, the positive outcome could be a 30 hour workweek and a three day weekend or more. That transition period from one to the other is extremely precarious for everybody and is what we are in currently. It isn't going to transition until a great deal of people make it so, and they only do that when the pain reaches home for them. In the interim, your advice is counterproductive. My take away from this past election: the overwhelming majority are not going to change until it gets real for them. Your advice is to light yourself on fire in the meantime. | | |
| ▲ | paulhebert 5 hours ago | parent [-] | | I really don’t understand how you got that from my post. I mentioned the fact that I’m often out of cell service on weekends (I like to hike and camp) and so it’s a bad time for an outage. Are you saying labor conditions need to get worse for everyone in order to get better? If so I disagree and think we should be advocating for better labor conditions for everyone while protecting things like weekends |
|
|
|
|
|
|
