It was a random Sunday. I started to see some alerts on my phone that I had never seen before. Something didn’t feel right. I tried calling one of our main numbers and it rang disconnected. My heart sank.
Our team immediately sprung to action. We forwarded our lines to a 3rd party answering service. It was a Sunday and our volume was minimal. However, this could not bleed into Monday. To add insult to injury – our new VP for the contact center’s first day at IGS was the following day.
We felt that our troubleshooting would be more effective if we all met in the office. I ordered some pizza and picked up a stockpile of caffeinated beverages. All logic pointed to the issue being caused by a particular system. After an hour or so, we were waiting for the manufacturer to isolate the cause of the problem.
I started to talk to the team about how we should split up who stays at the office and who goes home to get some rest. We needed some fresh minds for the next day in the event that this issue crept into the evening. We mapped out our plan of action in case the resolution of this issue looked like it would take hours instead of minutes.
Despite the fact that most of our team had taken a break to let the manufacturer work the issue, Chuck Campbell (Network Engineer) kept digging. It truly felt like a dog and its bone. He wouldn’t let go. Out of nowhere, I heard a wonderful sound.
Chuck’s desk phone was ringing with a call from the outside! I can’t articulate how much joy I felt in that moment. Chuck had isolated the cause of the issue. We had restarted one of the critical systems earlier that day but it hadn’t “let go” of a failed session. Chuck found a problem that the system manufacturer had yet to even identify.
Chuck fixed the issue. We eventually circled back with the vendors, business partners and manufacturers. We had a lot of work to do to ensure that this issue didn’t happen again. However, we would live to fight another day.
I learned a lot from Chuck that day…
- Don’t shift blame
- Don’t give up
- Think outside the box