One of my favorite interview questions is to ask the candidate about the most difficult technical outage that they have been a part of. I learn a lot about the individual based on how they describe their response to the situation. On occasion, the individual asks the question back to me. I always have the same response. The most difficult technical outage I have ever been a part of was when my organization’s file server crashed.
In short, we made changes to a core system in an effort to improve it’s reliability. Everything was validated and the system ran without issues for two days straight. Suddenly, backups started to fail. We restarted the server and it never came back online.
The team and I came into the office at 10pm to congregate and troubleshoot the issue. We worked through the night to mitigate the impact of the issue. We ended up rolling back our changes. After being awake for 36 hours, I finally went home.
I was upset about the scope of the outage and took it especially hard. The lack of sleep didn’t make things any better. I told myself and others that I would need to work my butt off to repair my reputation. I was devastated.
After I slept for a few hours, I woke up to a voicemail message from Ben Burgett. He told me a few things that I needed to hear…
First off, Ben didn’t feel that my perception wasn’t based on reality. Ben thought that people within the organization knew the outage wasn’t caused by negligence. The impact was minimized due to the quick actions of the team. The overwhelming consensus was, “stuff happens, that’s awesome those guys stayed overnight at the office to fix it”.
Ben didn’t want me becoming hyper sensitive to all of my actions because I assumed that people thought I needed to right a wrong that they likely weren’t even thinking about. He said that this attitude usually causes more issues than the original problem itself. He thought that people knew where my heart was and that I had a good track record.
The last piece of advice that Ben had was for me to understand my emotions. I really needed to think about why I was going through them. By gaining a deeper understanding of my emotions, I could try to channel them into something positive. I still reference this advice often.
This situation reminded me of an experience I had when I was in high school. At one point, I wrecked our family’s vehicle. Fortunately, nobody was hurt and the car was in good enough shape to drive home. My Dad made me drive the car home.
I thought at the time that he was just being difficult but he was trying to teach me a valuable lesson. Sometimes when you fall off the horse, you need to jump right back on.