| ▲ | sweswas2 2 hours ago |
| Anybody familiar with these systems know why they would do an update like this at 3pm on a Friday? This would be extremely unusual in any software job I have ever had. For commercial aviation specifically which does have pretty well defined peaks and valleys in terms of traffic over the course of a day this seems like an odd time to choose |
|
| ▲ | salt-thrower 2 hours ago | parent | next [-] |
| I did web dev work for an airline and we couldn't even release new updates to the loyalty benefits website during a weekday (even though there was no connection to operations or flights). The fact that this company would reboot a critical comms server at 3 pm on a Friday is indeed bizarre. |
| |
| ▲ | Rapzid 8 minutes ago | parent | next [-] | | I believe not deploying on Friday is an anti-pattern for most software teams. | |
| ▲ | ocdtrekkie 14 minutes ago | parent | prev | next [-] | | IT in corporate is actually more panicked than IT in life safety, because life safety invests more redundant design. | |
| ▲ | 2 hours ago | parent | prev | next [-] | | [deleted] | |
| ▲ | luxuryballs an hour ago | parent | prev | next [-] | | the update caused the reboot but could that mean the reboot happened due to a prior update? or literally the reboot was FOR the update hmm | |
| ▲ | NooneAtAll5 2 hours ago | parent | prev | next [-] | | > we couldn't even release new updates during a weekday so you were forced to release updates on weekends? isn't that worse? | | |
| ▲ | gdavisson an hour ago | parent | next [-] | | It depends on the schedules of the people who'll have to fix it if something goes wrong. I don't mind working weekends, so when I was in the sysadmin game I always tried to make significant changes late friday/early saturday, so if something went wrong I had the weekend to fix it & clean up. But if the people needed to fix a problem take weekends off, pretty much the opposite schedule applies. | |
| ▲ | 80fj0348i an hour ago | parent | prev | next [-] | | Flights generally peak Mon-Thu due to business flights, weekends are usually lighter unless it's a long holiday weekend. | |
| ▲ | margalabargala an hour ago | parent | prev [-] | | Worse for the employees. Better for the clients. |
| |
| ▲ | ransom1538 an hour ago | parent | prev [-] | | "3 pm on a Friday is indeed bizarre" Most likely, they HAD to do it. | | |
| ▲ | wombatpm an hour ago | parent [-] | | Probably was scheduled for Thursday evening but was delayed due to the Outlook issues. |
|
|
|
| ▲ | maerF0x0 31 minutes ago | parent | prev | next [-] |
| It should be fine to update at any moment. Why? Because systems like this ought to be hot spare or blue/green models that have a full working copy at any given time, and a failover/failback. |
| |
| ▲ | Waterluvian 21 minutes ago | parent | next [-] | | A reasonable deployment plan considers the reality that the deployment could fail. One mitigation is to choose ideal conditions for a possible failure. You’re correctly pointing out one of the mitigations, but it doesn’t render the others unnecessary. | | |
| ▲ | bombcar 19 minutes ago | parent [-] | | You can also have reasonable updates done at reasonable times that don’t show issues until unreasonable hours - or something else fails. But what I’m hearing is that one company “L3Harris” could take down the airfristructure anytime it wanted to. That’s not great. |
| |
| ▲ | sweswas2 19 minutes ago | parent | prev | next [-] | | I mean sure and we have that in finance but there’s no amount of time saved that could make me deploy something during market hours | | |
| ▲ | logifail 9 minutes ago | parent [-] | | Airspace certainly has busy and quiet(er) periods, but the skies are open all day every day. |
| |
| ▲ | skort 15 minutes ago | parent | prev [-] | | Well clearly they don't have a hot spare or blue/green models, or working failover. |
|
|
| ▲ | mainmailman 2 hours ago | parent | prev | next [-] |
| Maybe patching a critical vulnerability? I’m also at a loss for why they chose Friday afternoon. |
| |
| ▲ | cjbprime an hour ago | parent | next [-] | | That could be their least busy day, for all we know. Weekends are busier than weekdays for airline travel. | | |
| ▲ | alistairSH a minute ago | parent [-] | | Friday and Sunday are the busiest days. Business and pleasure overlap. Sat is mostly pleasure. Mon-Thu are mostly business. |
| |
| ▲ | 2 hours ago | parent | prev | next [-] | | [deleted] | |
| ▲ | dietr1ch 2 hours ago | parent | prev [-] | | I'd wager management wanted this by the EoW | | |
| ▲ | EvanAnderson an hour ago | parent | next [-] | | Arbitrary day-of-the-week and date-based deadlines piss me off so much. I worked for a person who had a fixation on "neat" deadines: the "first of the month", "the end of the week", "the start of the quarter", etc. It'd be one thing if the dates were tied to legal compliance (tax filings, etc), but that usually wasn't the case. They were chosen because they were "tidy". I'd time meetings looking to maximize time for work. You try to meet late in the day Wednesday because an early Tuesday meeting might get an "end of the week" deadline. A late Wednesday meeting was more likely to garner a "end of next week" deadline. Meeting Thursday or Friday guaranteed an "end of next week" but you lost Thursday and Friday work time. Wednesday was a gamble, though, because it was still early enough to get you an "end of the week". Meeting after lunch helped with that. You always had to be aware of the numeric calendar day, though. Gunning for an "end of next week" deadline when next Wednesday was the end of the month (or worse, end of the quarter) almost guaranteed you an earlier deadline than you wanted. All of this deadline-setting happened with little regard for the amount of time work was going to take. It was immensely fun, when setting a deadline in this person's absence, to get a group to agree on a mid-week deadline. You could see it make this person's skin crawl hearing that the work would be done on Tuesday or a Thursday. | |
| ▲ | nik282000 an hour ago | parent | prev | next [-] | | Weird how all epic stupid decisions come from the people who are supposedly the trained 'decisions makers.' I spend at least 1/3 of my fridays answering all requests with "Read Only Friday." | |
| ▲ | a012 an hour ago | parent | prev [-] | | RCA could be in just 2 words: incompetent people |
|
|
|
| ▲ | stridTastic 17 minutes ago | parent | prev | next [-] |
| Rolling out updates on a Friday afternoon is still practiced in various industries. |
|
| ▲ | vlovich123 37 minutes ago | parent | prev | next [-] |
| A security patch would be a reason not to wait. |
| |
| ▲ | Rapzid 6 minutes ago | parent [-] | | And because a security patch can't wait no deployment should wait because it's Friday. That's how you stay ready; you're always ready. |
|
|
| ▲ | asdfman123 an hour ago | parent | prev | next [-] |
| > Anybody familiar with these systems know why they would do an update like this at 3pm on a Friday As an insider to the technology industry, I would surmise the technical reason for it is that they didn't want to work over the weekend. (Yes, this comment is tongue in cheek, but not everyone follows the "don't deploy on Friday" rule, even people in mission critical fields.) |
| |
| ▲ | takeda an hour ago | parent [-] | | > I would surmise the technical reason for it is that they didn't want to work over the weekend. Isn't that exactly what they did? If you do major change on Friday at 3pm you likely will work over weekend. | | |
|
|
| ▲ | bitbangarang 13 minutes ago | parent | prev | next [-] |
| This is why you don't push PROD on fridays... |
|
| ▲ | _heimdall an hour ago | parent | prev | next [-] |
| Nothing like a good Friday afternoon did lot to ruin a weekend. |
|
| ▲ | thephyber 2 hours ago | parent | prev | next [-] |
| Perhaps. But the entire first months of the Iran War was started on Friday after the markets closed. It took a few weeks for most Americans to catch on. My money is on some mundane ops issue like "it takes lots of time to get sign off for the upgrade from lots of different stakeholders and the necessary upgrade was falling days or weeks behind, leading to a key IT person to decide to YOLO the upgrade/deployment". I've heard that Sec Transport Duffy has been pushing FAA upgrades faster than is safe, so maybe the employee rushed it so this upgrade wouldn't hold back some other larger timeline. There have been several articles recently about how the FAA is full of people who are "too risk averse", suggesting that there might be an effort to push riskier projects faster. Having a nationwide outage is one of the likely symptoms of raising risk tolerance. |
| |
| ▲ | hydrox24 an hour ago | parent | next [-] | | > It took a few weeks for most Americans to catch on In what sense? It was well reported at the time, so presumably most people know about it. I'd be interested in what makes you think the time and day of week chosen would delay market or public reaction be _weeks_. | |
| ▲ | panarky an hour ago | parent | prev [-] | | My money is on DNS, but if it is enemy action, will the FAA even know that's what it is? | | |
|
|
| ▲ | rdevilla 2 hours ago | parent | prev [-] |
| [dead] |