| ▲ | fransje26 8 hours ago |
| > searching for the cause of a bug let you discover multiple "forgotten" servers, ETL jobs, crons all interacting together. And no one knows why they do [..] And then comes the "beginner's" mistake. They don't seem to be doing anything. Let's remove them, what could possibly go wrong? |
|
| ▲ | HelloNurse 6 hours ago | parent | next [-] |
| If you follow the prescribed procedure and involve all required management, it stops being a beginner's mistake; and given reasonable rollback provisions it stops being a mistake at all because if nobody knows what the thing is it cannot be very important, and a removal attempt is the most effective and cost efficient way to find out whether the ting can be removed. |
| |
| ▲ | Retric 4 hours ago | parent | next [-] | | > a removal attempt is the most effective and cost efficient way to find out whether the ting can be removed Cost efficient for your team’s budget sure, but a 1% chance of a 10+ million dollar issue is worth significant effort. That’s the thing with enterprise systems the scale of minor blips can justify quite a bit. If 1 person operating for 3 months could figure out what something is doing there’s scales where that’s a perfectly reasonable thing to do. Enterprise covers a while range of situations there’s a lot more billion dollar orgs than trillion dollar orgs so your mileage may very. | | |
| ▲ | HelloNurse an hour ago | parent [-] | | If there is a risk of a 10+ million dollar issue there is also some manager whose job is to overreact when they hear the announcement that someone wants to eliminate thing X, because they know that thing X is a useful part of the systems they are responsible for. In a reasonable organization only very minor systems can be undocumented enough to fall through the cracks. | | |
| ▲ | Retric 7 minutes ago | parent [-] | | In an ideal world sure, but knowledge gets lost every time someone randomly quits, dies, retires etc. Stuff that’s been working fine for years is easy for a team to forget about. |
|
| |
| ▲ | amalcon 2 hours ago | parent | prev | next [-] | | I have had several things over the course of my career that: 1) I was (temporarily) the only one still at the company who knew why it was there 2) I only knew myself because I had reverse engineered it, because the person who put it there had left the company Now, some of those things had indeed become unnecessary over time (and thus were removed). Some of them, however, have been important (and thus were documented). In aggregate, it's been well worth the effort to do that reverse engineering to classify things properly. | |
| ▲ | chrisweekly 2 hours ago | parent | prev [-] | | Well, maybe. See Chesterson's Fence^1 [1] https://theknowledge.io/chestertons-fence-explained/ |
|
|
| ▲ | Mtinie 3 hours ago | parent | prev | next [-] |
| If it’s done in a controlled manner with the ability to revert quickly, you’ve just instituted a “scream test[0].” ____ [0] https://open.substack.com/pub/lunduke/p/the-scream-test (Obviously not the first description of the technique as you’ll read, but I like it as a clear example of how it works) |
|
| ▲ | notTooFarGone 7 hours ago | parent | prev | next [-] |
| I've fixed more than enough bugs by just removing the code and doing it the right way. Of course you can get lost on the way but worst case is you learn the architecture. |
|
| ▲ | xnorswap 4 hours ago | parent | prev | next [-] |
| The next mistake is thinking that completely re-writing the system will clean out the cruft. |
|
| ▲ | fragmede 7 hours ago | parent | prev [-] |
| that's a management/cultural problem. if no one knows why it's there, the right answer is to remove it and see what breaks. If you're too afraid to do anything, for nebulous cultural reasons, you're paralyzed by fear and no one's operating with any efficiency. It hits different when it's the senior expert that everyone revere's that invented everything the company depends on that does it, vs a summer intern vs Elon Musk bought your company (Twitter). Hate the man for doing it messily and ungraciously, but you can't argue with the fact that it gets results. |
| |
| ▲ | ljm 6 hours ago | parent | next [-] | | This does depend on a certain level of testing (automated or otherwise) for you to even be able to identify what breaks in the first place. The effect might be indirect several times over and you don't see what has changed until it lands in front of a customer and they notice it right away. Move fast and break things is also a managerial/cultural problem in certain contexts. | |
| ▲ | mschuster91 5 hours ago | parent | prev [-] | | > It hits different when it's the senior expert that everyone revere's that invented everything the company depends on that does it, vs a summer intern vs Elon Musk bought your company (Twitter). Hate the man for doing it messily and ungraciously, but you can't argue with the fact that it gets results. You can only say with a straight face that if you're not the one responsible to clean up after Musk or whatever CTO sharted across the chess board. C-levels love the "shut it down and wait until someone cries up" method because it gives easy results on some arbitrary KPI metric without exposing them to the actual fallout. In the worst case the loss is catastrophic, requiring weeks worth of ad-hoc emergency mode cleanup across multiple teams - say, some thing in finance depends on that server doing a report at the end of the year and the C-level exec's decision was made in January... but by that time, if you're in real bad luck, the physical hardware got sold off and the backup retention has expired. But when someone tries to blame the C-level exec, said C-level exec will defend themselves with "we gave X months of advance warning AND 10 months after the fact no one had complained". | | |
| ▲ | faidit 4 hours ago | parent [-] | | It can also be dangerous to be the person who blames execs. Other execs might see you as a snake who doesn't play the game, and start treating you as a problem child who needs to go, your actual contributions to the business be damned. Even if you have the clout to piss off powerful people, you can make an enemy for life there, who will be waiting for an opportunity to blame you for something, or use their influence to deny raises and resources to your team. Also with enterprise software a simple bug can do massive damage to clients and endanger large contracts. That's often a good reason to follow the Chesterton's fence rule. |
|
|