| ▲ | linsomniac 3 days ago | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
SysAdmin related: I was once talking to a fire chief and I asked about how much water the fire engines carried. He said that they carry about enough to put out the typical house fire. The first engine on scene immediately jumps to fighting the fire. The second engine on scene hooks the first engine up to the water supply before going on to fight the fire. I've often thought about that when there's a work crisis: If I'm the second on the scene, what can I do to support those fighting the fire right now, before jumping in. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | dreamcompiler 3 days ago | parent | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Our engine holds 1200 gallons. It goes in first* and starts putting the wet stuff on the red stuff. As the engine drives in it drops a 3" hose along its path. Next is our big tender with 3000 gallons. It stops at the street and connects to the dropped hose to pump more water up to the engine. The tender also has a drop tank -- think about a portable kids' wading pool but much larger and deeper. Shuttle tenders refill the drop tank while our big tender draws from it to continue supplying the engine. We don't have fire hydrants, so this is the dance we have to do. * It's very important to park the engine close to the fire but not too close. Ask me how I learned this. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | WarcrimeActual 2 days ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
>I've often thought about that when there's a work crisis: If I'm the second on the scene, what can I do to support those fighting the fire right now, before jumping in. Great! Now I'll have to see this quote over an image of a sweaty firefighter on LinkedIn every 3 weeks for eternity. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | WarcrimeActual 2 days ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
https://i.imgur.com/zCjoWIZ.jpeg I feel gross. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | BuildTheRobots 2 days ago | parent | prev [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> I've often thought about that when there's a work crisis: If I'm the second on the scene, what can I do to support those fighting the fire right now, before jumping in. A lot of it depends on the size and skill-set of your team and the escalation routes available to you, but in general (and off the top of my head): - Get the first people on scene to give a summary of the problem as they know it. Make sure everyone actually agrees on what the problem is and what symptoms have been observed. Understand what areas people are currently investigating and make sure they aren't trampling over each other or actually making the situation worse [1] - Make sure the situation hasn't evolved whilst the first on scene have been investigating the initial symptoms. It's easy to get lost in the weeds digging into a handful of monitoring alerts only to look up and realise there's now 300 and the original problem is only a small part of what's going on. - If there isn't one already and you're not better doing something else, become incident commander. When done right it's an extremely important and useful role.
Being at the coalface when it's on fire is a very different view of the world to watching other people panic and singe their fingers. It's also very easy to get lost in a chain of technical problems [2] when it's mostly irrelevant to the wider picture.If you get a moment, it can also be a good time to assess how useful your monitoring is during an actual event. [1] "Hey, server x has flagged on monitoring and my ssh session is hung waiting for a login prompt!" I've been round the houses enough to know this is probably OOM and if I just wait, I'm likely to finally get in. I also know that saying this in a room of 20 technical people, means the server is now processing 22 new ssh sessions and now no one is getting anywhere. [2] The famous Malcolm in the Middle intro where Hal is tasked with changing a lightbulb and ends up repairing the car. Except in my example the bulb is actually fine and there's a power cut we missed. https://www.youtube.com/watch?v=AbSehcT19u0 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||