Leadership Lessons from Chuck Campbell: Keep Digging

It was a random Sunday. I started to see some alerts on my phone that I had never seen before. Something didn’t feel right. I tried calling one of our main numbers and it rang disconnected. My heart sank.

Our team immediately sprung to action. We forwarded our lines to a 3rd party answering service. It was a Sunday and our volume was minimal. However, this could not bleed into Monday. To add insult to injury – our new VP for the contact center’s first day at IGS was the following day.

We felt that our troubleshooting would be more effective if we all met in the office. I ordered some pizza and picked up a stockpile of caffeinated beverages. All logic pointed to the issue being caused by a particular system. After an hour or so, we were waiting for the manufacturer to isolate the cause of the problem.

I started to talk to the team about how we should split up who stays at the office and who goes home to get some rest. We needed some fresh minds for the next day in the event that this issue crept into the evening. We mapped out our plan of action in case the resolution of this issue looked like it would take hours instead of minutes.

Despite the fact that most of our team had taken a break to let the manufacturer work the issue, Chuck Campbell (Network Engineer) kept digging. It truly felt like a dog and its bone. He wouldn’t let go. Out of nowhere, I heard a wonderful sound. 

Chuck’s desk phone was ringing with a call from the outside! I can’t articulate how much joy I felt in that moment. Chuck had isolated the cause of the issue. We had restarted one of the critical systems earlier that day but it hadn’t “let go” of a failed session. Chuck found a problem that the system manufacturer had yet to even identify.

Chuck fixed the issue. We eventually circled back with the vendors, business partners and manufacturers. We had a lot of work to do to ensure that this issue didn’t happen again. However, we would live to fight another day.

I learned a lot from Chuck that day…

  • Don’t shift blame
  • Don’t give up
  • Think outside the box

Leadership Lessons from Paul Bahler: Plan for the Worst, Hope for the Best

I’ve been working with Paul Bahler at IGS for almost 4 years. During a majority of that time, Paul has lead our IT Operations team which includes Database Administration and DevOps. I’ve seen Paul lead our department of over 125 talented individuals through major accomplishments and incredibly difficult situations. Regardless of the outcome, I manage to learn something from Paul’s years of experience.

When I think back about the key lessons that I’ve learned from Paul, the first story that jumps out was when we were in the process of launching a new website. This launch involved using technology that the team was unfamiliar with and we had encountered several unforeseen issues. In short, we were behind schedule and in danger of missing a critical deadline that had been communicated externally.

In a situation where deadlines will likely be missed, people have a tendency to still move towards the happy path. One of the key lessons I learned from Paul during this issue was how quickly he forced all of us to consider what failure would look like. Instead of simply pressing forward and hoping for the best, Paul asked us to come up with a contingency plan in case we were unable to hit the deadline. Based on the information we discussed, we realized that failure was not an option for our team but we learned that we could make some concessions in an effort to reduce some of our workload until after the launch.

The prospects of missing a deadline and/or cutting corners increased the tensions among the team. At the first inkling of conflict and finger pointing, Paul immediately squashed it. He didn’t dismiss the fact that we needed to make improvements to ensure that we didn’t find ourselves in this same situation during our next big project. Instead of looking back at the root cause of our problems before even we completed the project, Paul suggested that we hold a formal “retro” at the conclusion.

Fortunately, we launched the website on time and without any major hiccups. The project team met at a restaurant a few weeks after the site went online. We discussed what went well and most importantly the areas we could improve. I remember walking away from that meeting feeling a lot closer to the team along with a sense of what I could do better next time.

While I hope I don’t have to use these lessons in contingency planning and continuous improvement anytime soon, I’m really glad I have gained this perspective from Paul and look forward to working with him for years to come.

FedEx Days at IGS Energy

Google is known for allowing their employees to devote a certain amount of time to work on projects that might not pertain to their day-to-day responsibilities. This practice gives Google team members an opportunity to flex their creative muscles while providing them with the freedom to work without limits. It has lead to the creation of several successful products (including Gmail). Following this same model, FedEx days (also referred to as ShipIt days) are typically a 24 or 48-hour period where employees are allowed to create and innovate with very little parameters or instructions.

Several months ago, it was decided that the IT department at IGS Energy would hold their own FedEx days. The idea was announced to our entire team at a department meeting. In the days and weeks leading up to the event, additional details were shared with the group. These details included a few rules.

IGS FedEx Days Rules…

  • We could work on anything we wanted as long as it’s not part of our regular job.
  • We could work in our system or work area, but not on items in our current backlog
  • The official clock started on a Thursday @ 2pm
  • We couldn’t start creating prior to 2pm but we could research
  • We had to show what we created or learned by Friday @ 4pm by creating a 2-minute video highlighting our work and outcome
  • Not everything will be a success, but often failure teaches us the most.

I wasn’t sure if I was going to participate at first. I had a lot of work to catch up on and could have used some time to focus on a few tasks. Fortunately, I was encouraged by a few co-workers to participate in the event. That left me with a  very important decision…what should I create?

About 10 months ago, I found myself in a situation where we needed all hands on deck to help solve a production issue. Since the issue occurred on a Sunday afternoon, it took me roughly 45-60 minutes to call everyone and explain the situation while discussing our next steps. I decided to spend my time calling instead of sending emails and text messages because texts/emails can be easily overlooked on a weekend. Afterward, I thought about ways that I could automate that process but I never really had time to devote to solving that particular problem. I decided this would be my challenge for FedEx days.

Before joining IGS, I wrote Python apps and lengthy PowerShell scripts but it had been about 2.5-3 years since I had even touched a piece of code. The idea of writing and deploying an app in 36 hours was really exciting. However, I wondered if I could actually get the work done in that short of a time-frame. I ended up partnering up with one of our Directors of Application Development (Matt). It had been about 10-15 years since Matt had written a line of code as well. We knew we had our work cut out for us but figured we would start by identifying our application’s requirements.

The requirements were really pretty straightforward…

  • The program should automatically call a list of people and play a brief message for them
  • The message should notify them there is a critical issue and that they would receive a follow-up communication with further details
  • The program should send out a text message & email with information about how to dial-in to a conference bridge
  • The program should be easy and quick to execute
  • The program should send out a Tweet when executed…because…why not?

Over the last few months, I had seen blogs about how people have creatively used Amazon Dash buttons to automate tasks. In fact, Amazon now sells AWS IoT Buttons that can be easily connected to AWS. This seemed like a fun way to kick off our process. Matt was responsible for connecting the button to AWS and ensuring that it could execute my python script.

A FedEx days project should celebrate failures. I would have been perfectly content if I gave this a shot and didn’t produce a working product. I was happy to just spend some time writing code and solving a problem with technology. Fortunately, thanks to some extremely helpful Python libraries and technical blog posts, we had a working script in just a few hours. After a few additional hours of tweaks, we even had the script executing from AWS after pressing our IoT button! We knew that there were potential improvements to make but we had solved our initial goal!

When it was all said and done, we had the following components in our solution…

  • AWS IoT Button – used to kick off our process
  • AWS Lambda – used to host and execute Python script
  • Python Script
    • Trello – used to make a phone call and send a text message
    • Tweepy – used to post a Tweet to my personal account
    • SendGrid  – used to send an email
  • Digital Ocean Web Server running Apache – used to host XML configuration file that was leveraged by Trello

We didn’t end up filming a video. This is mostly because Matt and I had to fit in a few meetings leading up to the final presentation. It also was a result of us making code changes up until 15 minutes before the projects were due. The lack of a video left us with a much more exciting yet nerve-wracking opportunity…a live demo.

We hadn’t tested the IoT button outside of Matt’s office and were a bit nervous about the recent code changes that we had made. We ended up configuring the script to call 16 people that we knew would be in the room during the presentations. When Matt pressed the button, I braced myself for the fact that the indicator light might turn red and the script wouldn’t execute. Sure enough, the light turned green and phones around the room started to ring. My Twitter account even sent out a test message (below). It was awesome.

There were a bunch of other great projects ranging from mobile applications to unique experiments involving spare laptop batteries. I was really proud of everyone that participated, particularly those that I work closely with on the Infrastructure team. Overall, I had a great time, learned a lot and I’m really looking forward to our next FedEx Days.

Kanban for Infrastructure and Security Teams

I work at IGS Energy in Dublin, Ohio. My team and I are responsible for Information Security, Network Engineering, Systems Administration, Telecommunications and Technology Support. We work in a fast-paced environment with frequently changing priorities and deadlines. A few weeks ago, I set a goal to improve and streamline our project communication/collaboration.

We work in a department comprised of several agile development teams. From the moment I started at IGS, I was very impressed with the open and transparent communication involved with running an agile team. In the back of my mind, I always wondered if I could make that work for the Infrastructure & Security teams at IGS. Our team already held daily stand up meetings and worked very closely with the development groups. Could we take it a step further?

We ended up attempting to leverage agile for an Infrastructure-related project. The goal of the project was to replace the portion of our network that is leveraged by our end-users. Unfortunately, we struggled to leverage agile for this project. We found that the work didn’t always fit into standard delivery time slots and often had to be delivered continuously due to inconsistent maintenance windows, etc. We also found that we had to frequently change the schedule to allow us to occasionally focus on daily maintenance and keeping the lights on. Finally, we noticed that attempting iterative delivery of this specific infrastructure project was actually hindering the user experience and negatively impacting the goals of the project.

We ended up trying other methods of project delivery and communication. Everything from sending email status updates to holding individual project meetings. We even tried using Microsoft Teams to document status updates. These solutions were good but we wanted something great.

We really just needed a few things:

  • Open/transparent communication
  • Continuous delivery
  • Clearly articulated priorities

I ended up asking friends and colleagues for advice. I reached out to App Dev Managers, Business Analysts, Infrastructure Leaders and even our VP of IT. I ended up getting some great advice that lead me to the Kanban Methodology. I ended up doing some research and found that several Infrastructure & Operations teams had successfully implemented this form of project flow.

I ended up deciding that it’s worthwhile for us to test out Kanban and see if it works for us. At that point, we had to make a few decisions. Should we have a physical or virtual Kanban board? It made a lot of sense for us to use a virtual board but we still had to decide what product to use. Our organization already had subscriptions to Jira and Office 365 which both offered tools for virtual Kanban boards. However, a member of the team was already using Meistertask and after exploring the UI, we decided to give that a shot.

We ended up starting out with 5 categories:

  • Backlog – Tasks that haven’t been assigned. I was given some great advice to purge items from this category if they remain dormant for an extended period of time. If they’re dormant for that long, clearly they aren’t a priority. An extended backlog of tasks that are never prioritized could turn the board into a source of anxiety.
  • Hold/Blocked – Tasks that have been assigned but can’t be completed at this time. This could be as simple as a task being contingent on the completion of another task.
  • Work In Process (WIP) – Active tasks. Our goal is to limit the amount of WIP to items that can be focused on at that point in time.
  • Validate – Tasks that are being tested.
  • Complete – Completed tasks. It’s important that we take time to recognize these items and celebrate success.

After discussing this with the team, there were a few questions that ended up causing us to make a few adjustments to the process. For example, he team already receives a high volume of requests/tasks through ServiceNow. Should all of those flow through Kanban as well? We made the decision that only items that will take more than 3 business days will be added to the Kanban board.

The team also discussed who could actually create items for the Kanban board. It’s important that we had a clear sense of priorities and didn’t introduce too many cooks in the kitchen. We decided to add an extra category for “brainstorming”. This is an area that anyone on the team can add an item to. During our weekly project review meeting, we will discuss all open brainstorming tasks and decide whether or not they are added to the backlog.

We plan on communicating the status of our tasks during Daily Stand Up Meetings, Weekly Status Meetings, and a Monthly Retro. During our retro, we will review what went well, what we could have done better and what our next steps are. We will also hold retros for any unplanned downtime.

I’m looking forward to seeing how this works out. Have you implemented Kanban for an Infrastructure or Operations team? If so, let me know how it went!

Why I believe in i.c.stars

After finishing grad school in 2015, I had a strong urge to volunteer. I wanted to make an impact. Unfortunately, I didn’t quite know how to devote my time. Should I go to a soup kitchen? Meals on wheels? I reached out to Jen Bowden (Director of Community Investment at IGS) and she gave me some wonderful advice. She told me to find a problem that I was passionate about fixing.

The more and more I thought about it, I was frustrated by the unnecessary barrier that was preventing talented individuals from entering the IT workforce. I was fortunate enough to have an opportunity to invest time and money into receiving a formal education at Ohio University. My formal education served as a great way to demonstrate my work ethic and desire to learn. It ultimately got my foot in the door of a great company and helped kick-start my career. In my mind, there had to be a way to help less fortunate individuals demonstrate those same capabilities while gaining some practical experience. After all, I learned more about business working at our family-owned shoe store in High School than I did receiving a Minor in Business Administration.

Rather than sit back and complain, I decided to try and fix the problem from the inside. I started teaching classes at Franklin University. As much as I loved helping the students, it wasn’t enough. I wanted to do more to remove the barrier preventing talented individuals from entering the IT workforce.

Fast forward a few months and the aforementioned Director of Community Investment at IGS introduced me to Ryan Frederick. Ryan spoke to us about a program that had started in Chicago called i.c.stars. The program identifies, trains and jump-starts technology careers for low-income young adults who, although lacking access to education and employment, demonstrate extraordinary potential for success in the business world and for impact in their communities. Candidates that enroll in the i.c.stars immerse themselves in the program. They go through a 4 month training cycle and spend 60+ hours a week working on a real project for organizations.

The numbers from the original office in Chicago speak for themselves…

  • 300+ total alumni
  • 95% initial placement rate
  • $9,915 average annual earnings before the program
  • $57,240 average annual earnings 30 months after program completion

Honestly, even after seeing those statistics, I was skeptical. I didn’t think that i.c.stars could prepare individuals with little-to-no technical experience for a career as a Business Analyst, Project Manager, QA Analyst or Developer in 4 short months while also managing to deliver a working product. The team at IGS decided to serve as the first project sponsor for i.c.stars Columbus. We saw a partnership with i.c.stars as an opportunity to give back to our community and change lives for the better.

The i.c.stars group was split into several teams that worked to build IGS a dashboard to display information about our EDI transactions. I can definitively say after serving as the organization’s first project sponsor that I had no reason to be skeptical. I am still amazed by the growth that I witnessed during the initial 4 month program.

If you’re interested in learning more about i.c.stars, contact me and I will introduce you to some of their awesome team members. In fact, along with Josh Miller (Training Manager at i.c.stars), I’m establishing a Mentorship Committee at i.c.stars in an effort to help recruit candidates around Columbus to help mentor future classes. We’re looking for folks with experience in Software Development, Database Administration, Business Intelligence, Engineering, Project Management, Leadership, Information Security, IT Infrastructure, QA Testing or Technical Support. The time commitment to mentor could be as much or as little as someone is willing to contribute. If you’re interested in joining the committee and helping recruit mentors for the organization, please let me know. Also, click here if you’re able to spend some time serving as a mentor.