| ▲ | ArcHound a day ago |
| They are turning it off and on again. I don't even. If this works, our industry is cooked as the kids say. |
|
| ▲ | onefiftymike a day ago | parent | next [-] |
| I worked on multi-million dollar airplanes. If a pilot called me with an issue and hadn't rebooted the plane (on the ground), I wouldn't even talk to them. Rebooting isn't always a bad solution to technology. |
| |
| ▲ | ArcHound a day ago | parent | next [-] | | It's not a bad solution, it definitely should be tried. But I'm still rather shocked to see it in official notes, especially this "late" in the timeline. | | |
| ▲ | mapontosevenths a day ago | parent | next [-] | | It's not a solution at all. It's a symptom. | | |
| ▲ | NetMageSCW 9 hours ago | parent [-] | | It’s not a symptom, it’s a work around until a more permanent fix can be deployed. | | |
| ▲ | mapontosevenths 3 hours ago | parent [-] | | > it’s a work around until a more permanent fix can be deployed. If it "fixes" your problem, then you didn't fix you problem you identified it. It's another symptom of a bloated industry that can't or won't innovate, and I think it's more "won't" than "can't." For example, we still can't patch running Windows, and most of the information technology professionals I know are still convinced it's impossible. Billions upon billions of dollars in time have been lost, probably enough man hours to send men to mars several times over. Yet, you can just buy a version of Windows Server that already does it. It's not only possible, they already have the code. They just don't care enough to integrated it into the product most people use. |
|
| |
| ▲ | a day ago | parent | prev [-] | | [deleted] |
| |
| ▲ | NamTaf 14 hours ago | parent | prev | next [-] | | In rail, we call it BOBO - battery off, battery on. | | | |
| ▲ | olyjohn a day ago | parent | prev | next [-] | | If they rebooted it, and it started working again, would you just walk away? Or would you try to figure out why the fuck it needed to be rebooted in the first place? | | |
| ▲ | onefiftymike 8 hours ago | parent | next [-] | | The vast majority of the problems were a system that missed the time window to report it's self test. A reboot got things in order. It was a small fleet, so we had a pretty good idea if the same problem kept popping up so we could look more into that system. A few fault codes were well known enough that if you called with that one the answer was to shut down and find a hotel, you aren't going anywhere and no reboot will take care of it. (bad detectors in the engines for about 10 years until a redesign) | |
| ▲ | SoftTalker a day ago | parent | prev [-] | | Thales or Collins or Honeywell or whoever provided the system would. The technician at the airport would only follow a procedure listed on a checklist, which probably includes rebooting the system. | | |
| ▲ | cyphar 15 hours ago | parent [-] | | Ironically there have been a fair few cases of aircraft incidents where components were reported as bad by pilots but the issue couldn't be reproduced on the ground and "it worked" after a reboot and self-test so they just kept using it (with this cycle repeating multiple times). Often there were clear physical problems (cracked solder joints, stuck microswitches, broken insulation) that were only discovered after a catastrophic failure but could've been discovered with a more inquisitive engineering culture. I would argue a "just reboot it" culture leads to more problems -- engineers should be empowered to diagnose problems before they become catastrophic. In the world of software we even have mechanisms for extracting data from failed systems to root cause them later, and yet most people do not take that step before "just reboot[ing] it". [1]: AirAsia 8501, Lauda Air 004, and SmartLynx Estonia 9001 come to mind. | | |
| ▲ | electroweak 12 hours ago | parent | next [-] | | But these intermittent failures lead to delays and worse. The sheer cost of repeat events leads to action. If deferrable at all the standard troubleshooting is mostly remove and replace. The repair and reliability processes identify these rogue units fairly fast and the actual rates of that in a well functioning airline are very low. | | |
| ▲ | SoftTalker 9 hours ago | parent [-] | | Yes, I remember a flight that sat at the gate past departure time, the explanation to the passengers was that some essential piece of equipment wasn't functioning, they had tried restarting it, and it was going to be replaced. About 30 minutes late, we were on our way. |
| |
| ▲ | hgomersall 13 hours ago | parent | prev | next [-] | | Right, rebooting only really makes sense if you understand why the reboot is going to fix the problem. As in, corrupted volatile state is reasonably fixed with a reboot. Any hardware issues or persistent state problems could well get worse. | |
| ▲ | Merrill 13 hours ago | parent | prev [-] | | The following is a few decades old -- A physicist, an engineer, and a computer programmer are driving down a steep mountain pass when the car’s brakes fail. The car speeds out of control around sharp turns, nearly flying off the edge before they miraculously manage to skid to a stop at the bottom. Shaken up, they discuss what to do next: • The physicist says: "We need to calculate the friction coefficients in the brake pads and model the temperature rise to figure out why they failed." • The engineer says: "I have a few tools in the trunk. I’ll take a look underneath and see if I can fix the mechanical failure." • The programmer says: "Why don't we get back in and see if it's reproducible?" |
|
|
| |
| ▲ | thisismyanon2 a day ago | parent | prev | next [-] | | Rebooting is also the only solution for some problems. looks at the Boeing 787 | |
| ▲ | boomlinde 18 hours ago | parent | prev | next [-] | | What makes sense for something that is used for 20 hours at a time is different from what makes sense for a service that literally has "365" in its name. | |
| ▲ | taftster a day ago | parent | prev | next [-] | | (on the ground) is probably pretty key, though. Ha ha. I can imagine the conversation if this was in the air. | |
| ▲ | cyberax a day ago | parent | prev [-] | | Yeah. It works, almost always: https://en.wikipedia.org/wiki/Indonesia_AirAsia_Flight_8501 | | |
|
|
| ▲ | justonceokay a day ago | parent | prev | next [-] |
| In maintenance, sometimes the dumb solution is the best. The servers that ran amazon.com had a memory leak and were subsequently restarted. This hack continues for years because it was the most cost-effective thing to do. Put another way, if they didn’t turn it off and on again, I would think less of them because that’s the first thing you should always do unless you know better |
| |
| ▲ | jimmaswell a day ago | parent | next [-] | | I suppose you might want to first rule out a break-in, and if there is one, recover evidence that won't survive a reboot. | |
| ▲ | jamesfinlayson 20 hours ago | parent | prev [-] | | Yeah, a place I worked at years ago had a cron job to restart some critical service at midnight because it leaked memory all day. That was there for years. | | |
| ▲ | ninalanyon 13 hours ago | parent | next [-] | | We had a mail server that ran on an old DEC minicomputer. I rebooted it every six months because it had a slow memory, or possibly resource, leak somewhere. It was the only thing running on that box so it wasn't a big deal to reboot. | |
| ▲ | Rantenki 19 hours ago | parent | prev [-] | | Another real world example: to this day, it's a good idea to limit requests per PHP process using the MaxRequestsPerChild(apache)/max_requests(php-fpm) config option to some sane limit. After that configured number of requests, the PHP worker is terminated and restarted to... "fix" memory leaks. I remember setting that in the late 90s on Apache... I am happy to no longer use PHP for anything. | | |
| ▲ | jamesfinlayson 19 hours ago | parent [-] | | Oh, it's been a while but I remember that. I've used PHP in two previous jobs and certainly the stuff that I was in control of had it set to 0 because we made sure we didn't leak memory. Other departments... I'm sure at least one had the limit set to something non-zero. |
|
|
|
|
| ▲ | icameron a day ago | parent | prev | next [-] |
| Powercycling is a designed method to bailout Cisco routers. If you botch a configuration command and lose connection, you can have someone powercycle it to restore you last good configuration. Its not just a joke it really works more often than I'd like to admit. |
| |
| ▲ | macintux a day ago | parent [-] | | ~25 years ago I was a firewall administrator, and the Cisco PIX we used was notorious for gradually running out of memory, effectively cutting off the entire company from the Internet when it went down (and it was a very big company). We absolutely should have scheduled reboots during off hours. Lessons learned. |
|
|
| ▲ | bawolff a day ago | parent | prev [-] |
| Amateur hour. Everyone knows professionals say "power cycling" instead to make it sound complicated. |
| |
| ▲ | red-iron-pine a day ago | parent [-] | | the layman understands both of those words and their meaning together. we "bounce" the servers when on calls with customers. |
|