Information Security Experience

Information Security


Role Description

My job description expanded with the added role of managing the division’s information security operations team. This was a global role responsible for 30,000 Unix/Linux systems across 51 sites. I retained my core job but was able to add the additional responsibilities since I had developed my local team into a strong, self-sufficient group that could manage themselves.


Background

In one fell swoop after an extended vacation to Nepal, my boss met with me and told me that I was now put in charge of managing our organization’s information security team. I knew nothing about information security and told him so. He encouraged me to figure it out… fast! The role was to implement security measures to our Unix/Linux environment by direction of our division’s information security team. I quickly assembled a team at our multiple sites and began organizing the work we needed to do. In order to succeed, I took several important steps to build the team.  They were:

  1. Create a team charter
  2. Develop ground rules and team norms to set expectations on how team members should work, interact, and perform.  It was both a top down and bottom up exercise.  I needed to set expectations as a leader, but the team needed to contribute and buy in to these rules.
  3. Create clear goals.  This required pressing the divisional team to do the same.
  4. Develop roles and responsibilities so that team members and their manager knew what was expected of them. Most did not know what they were supposed to do at all in this area since the work required was unfamiliar to them and the division was too far removed to engage with them effectively.

Figuring out what the goals were from the division was not easy. As we began communicating with them for direction, we found huge gaps in how we were supposed to conduct our work. Requirements were fragmented and instructions were non-existent. We were still held to deadlines, but the quality of work and assistance provided from the division was weak and nearly useless. I made several complaints and gave numerous suggestions over the course of a few months in order to improve the situation. Division sensed that I had a strong grasp of what needed to be done and how to do it, so they eventually appointed me in charge of the program. This was a big step in my career and my responsibility level skyrocketed instantly.


Scope

I was given 21 Unix administrators for my global team that were located in nearly every part of the world.  Those representatives were located at the sites below.  These administrators supported 51 sites in total.

North America:

  • Hillsboro, OR.
  • Dupont, WA.
  • Folsom, CA.
  • Santa Clara, CA.
  • Chandler, AZ.
  • Hudson, MA.

Asia:

  • Cavite, Philippines
  • Bangalore, India

Europe, Middle East

  • Copenhagen, Denmark
  • Haifa, Israel

Implementation

In order to move such a scattered group forward, I needed to establish effective communication and approachability with the team members as a group and as individuals.  I set up meetings with each one of them, got to know them and their daily challenges, and then began piecing the team together. I also met with their managers in order to get their buy in since I would need to take more of their employees’ time and focus. Without this, the administrators would inevitably get pulled into local issues that always crowded out global ones. This was not an easy task. Upper management was good at creating programs that required results, but did not create a system of accountability to make it happen. I had to do that on my own to get results.

Specifically, I did two things that aided this cause:

  1. Embedded my goals into the administrators’ quarterly objectives
  2. Presented status regularly to upper management about my program’s efforts

Both these steps helped me gain visibility and commitment from my many stakeholders. As a result, information security became a formal part of my team members’ jobs. Middle management gained scrutiny by our divisional director when I would report out in their staff meetings on my progress. If some regions failed to get work done, these middle managers were told to get committed and devote resources to the work. I leveraged another important piece of the puzzle, that of working with a staff manager as my sponsor. Without her, my job would have been much tougher when trying to deal with the middle managers who preferred to worry about their own priorities.

I did the same thing as I did for my regional team, i.e. creating a team charter, ground rules, goals, etc. and then set out to get the work done. Still, there were more challenges to overcome. Many of the administrators had little experience in information security themselves and were frequently unable to implement the measures recommended by the security architecture team. The architecture team was made up of extremely skilled security experts. They did not have any patience to help my people out and possessed no sense of how to enable people to perform their duties. I had to leverage them in a productive way and so I pushed to become an advisor on their team. Upon joining the team, I pushed for two important things, 1) Get my team trained properly, and 2) Get the uppity security experts to write documentation and teach my people. To get the training, I investigated the various resources available that an industry consortium provided. I managed to secure numerous training books from them and also flew in my team members to the U.S. to take an extensive training course. This brought us very far, but I still needed cooperation from the security experts. I met with them individually and made my case for how they could do some upfront investment in documentation and teaching that would create a more self-sufficient team. Their constant complaint was how inept my team was, so if they could take a few steps to fix this problem, then they could get past this irritant, and move on with their desired duties. Slowly but surely, they began their work to document how to do a long list of measures that I had collected into a master plan. Simultaneously, I would go to the divisional staff meeting and explain clearly the scope of work needed to be done and what I needed to get it done. Soon, I was getting support from above and a growing cooperation from below. I kept a tight measurement system going about what items we had completed in each region and each site. As I checked off more and more items, the security experts could see that we were making real progress and became more helpful in continuing the momentum.

I made an organizational change that also aided my work. Running meetings with a global team was extremely impractical due to time zone differences. Those far away were having the most difficulty achieving goals due to communication problems from the center. I split my team into two so that the meeting times were more agreeable to international team members and attendance and attention improved. I also appointed team leads for each of these groups who helped me keep discipline and communication going within the team. These two leads sat in time zones that could communicate better with my remote sites. I also gave them more training and an open door to contact the security experts. They relished their roles and improved drastically the functioning of our larger team. To supplement the attention needed for the remote sites, I stayed online my entire waking hours so that anyone could call me at home or at work for help.

We made tremendous headway, but we still had a number of measures that never quite got done. I needed to increase the awareness of these gaps to upper management, so I tried a new method. I wanted to put some peer pressure into the system so that the different regions could feel a sense of competition with their fellow regions. I created a program called Site Certification to do this. Since I had a comprehensive list of everything needed to be done at all 50 sites, I developed a “certificate of completion” for each of them. As each site completed their deliverables, they would receive an award in the divisional staff meeting. Each region owned numerous sites, so seeing other regions achieve certification could compel them to work hard to outdo their peers. The program was approved by division, but by this time, we had really gotten closer to achieving our goals. I had been regarded as an extremely successful program manager and was ultimately given a brand new mess to fix. Site certification didn’t proceed as formally as I hoped, but it lived in spirit. When I left the team, my leads kept it alive at a lower level for the months to come. They achieved all the goals eventually, so though it didn’t happen as I had planned, the work got done and the overall program was a huge success.


Key Learnings

Out of this experience, I gained several new skills and lessons. First, I learned how to lead a global, multi-cultural team effectively. I learned the best ways to communicate with people from far away and to understand how to motivate them and remove barriers for them. I also gained great experience in setting a direction, tracking progress carefully, and developing a very clear understanding for my team and stakeholders of the work that needed to be done. I also sharpened my skills in working with multiple levels of management, getting buy in, and putting goal pressure into the system. Additionally, I learned to work with very arrogant, prickly people, turn them around, and get them to cooperate and help. I also gave all my team members a sense of purpose, mission, and pride. I worked with very enthusiastic people and I think I created the environment that cultivated it. I also gained valuable experience in getting tasked with huge amounts of ambiguity. I didn’t know anything about the subject matter, I didn’t know how to motivate people across the world, and I didn’t know how to get such a large and vast body of work completed. I worked hard to learn these things, (granted, by getting a lot of bumps and bruises along the way) and got results. I think this was one of the best achievements in my career.


Performance Review for my Security Efforts

  • Dave became the leader of the EC Security Operations Team (SOT) and really focused on driving process with the SOT and keeping the team on track.  To bring structure to the team, Dave developed the team’s charter, job description, and set clear expectations.  Dave communicated with each member’s direct manager to gain their support and commitment for security activities.  He set up meeting structure to ensure full participation (East/West coast) and worked against quarterly objectives.  He also developed two members as team leads who are responsible for running weekly meetings and drive completion to action plans.  Dave grew the SOT from 12 people to 21 and increased the number of sites owned from 35 to 51.  Dave worked with the Security Architecture Team (SAT) to try to define process and process and procedures for requesting and working with the SOT members.  Dave played a key role in trying to manage the time the SAT is requesting from the SOT for new projects as well as keeping them on track with their regular security operational tasks.  EC improved Unix audit scores from 85% to 90%, moved sites to nearly 100% ssh and chpasswd usage, drove central logging to 36 sites, and conducted numerous audits.  The impact is that the EC organization has moved to an unprecedented high level of security compliance which significantly improves the protection of Intel IP.

Asset Management Experience

Asset_Lifecycle_Management_Services2


Role Description

After leading the information security team successfully, I was asked to lead another program that was in sore shape.  The Asset Management Team was in desperate need to account for all their assets in a short period of time.


Background

Due to Sarbanes-Oxley requirements, Intel had to account for all of its assets. Our division was struggling to complete this task, so I came in to make it happen. We needed to physically track every computing asset that we supported worldwide. The number was over 30,000 systems. It sounds a bit funny, but all these systems were accounted for on the network, but we simply didn’t know where all of them actually resided. We had to look in every nook and cranny, under desks, in data centers, etc. until we could find every one, bar code it, scan it, and then make sure it populated into our asset database. Once we achieved that, we had to dump the information into a master database for the entire company. Each step created challenges.


Getting It Done

I was given a team of about 16 people that represented all 50 sites we supported worldwide. I had to take over from the division’s asset manager who simply became overwhelmed by the task. She stayed on the team to help. Sometimes that was good, sometimes it wasn’t. She was tirelessly dedicated to her job and lived and breathed this type of work. She just didn’t have the skillset to take on a global project. She was a bit resentful of me taking over, but did manage to handle the situation maturely. The only real obstacle was that she had a very narrow view of how to get the work done, narrow in the sense that her ideas were too big for what was required. She was ex-army, rough around the edges, and had no political skills. It was going to take some special care to keep the project focused properly. I had to balance the raw objective of getting the assets tracked with her desire to create a comprehensive asset management process. Normally, I liked to do that sort of thing, but we had deadlines and I had to choose wisely when to leverage her ideas and when to cast them aside.

One luxury we had was an actual database administrator who could customize our asset tool on the fly. He could make bar code scanners work better with the tool and could manipulate the system so that it could upload our data to the master system. He was generally a reliable order taker, so he made my job a lot easier. For the rest of the team, I got a representative or two from all regions to pound the pavement and track the assets. In comparison to my information security program, this was on nearly equal footing in global reach and scale, but it was still a simpler job. It took less time but still managed to create a lot of headaches. I didn’t need to study asset management or train my staff. It was a fairly straightforward task, but the enormity of the asset base and a zillion little, pesky problems required a large percentage of my time.

Once again, I used my experience in starting up teams and programs and set the charter, ground rules, responsibilities, and goals. As before, this work was considered a second priority for my team members despite the fact that we had a legal requirement to get the job done. I had an entirely new stakeholder to deal with, i.e. “Mother IT” who loved to control everything and was a huge morass of bureaucracy. It was hard to even find who was responsible for what and to know who mattered and who didn’t.

To the regional middle managers, this was a necessary but mundane task to provide resources for. It was therefore even more important to make my goals visible to upper management in order to get commitment. The team members were reasonably motivated and ultimately enjoyed contributing to the effort.

I did have to do a little homework. There is such thing as an IT Asset Life Cycle. My asset manager was hopelessly devoted to it, so I tried to humor her by learning it and trying to implement the steps suggested in this model. My memory of this portion of the program has faded significantly. I know we analyzed a lot of flow charts and wrote a lot of documentation, but I can’t remember how it all contributed to the effort, honestly. I think it did help to have this as a foundation, since it assisted us in dealing with asset procurement and making sure anything new that came in the door was tracked properly. It also helped establish a process for retiring assets and making sure they were removed from the system. As it turned out, these two parts of the chain were what caused us to get into a mess more than any others. Bottom line was that this program was more than just hunting down assets, it was intended to be a platform for making sure all assets got handled properly throughout their lives.

After the launch of the team, I needed to find ways to measure status and progress. There were already a few thousand assets tracked properly, but the reporting was not very robust or reliable. I had to give this priority to our DB admin beyond most other duties. Most people were hot and heavy about getting items scanned. I had to refocus the team to make sure we had a useful metrics system. No metrics, no management support. Once we did get our first run of useful reports, it showed that some regions (like the asset manager’s region, for instance) were in great shape and mostly needed to clean up some stragglers. Other regions were a complete mess, especially Europe and Asia (and my own region!). I had to learn the right methods and then extrapolate them across the entire division. Some regions didn’t even have bar codes and scanners, so buying those and distributing them was one of the early orders of business. Once we knew where we stood, I gave my first presentation to divisional staff about where we were. I got good attention to the matter and commitment was less difficult this time around since it was less complex, could utilize less-skilled technicians, and had Mother IT breathing down their necks. Still, some regions continued to lag behind, so I had to increase awareness by sending out regular reports to divisional and regional staff.

As mentioned above, we had to get the data in front of people in a clear and concise way. We really hit the jackpot when my French representative on my team created a masterful looking e-mail interface for our report. Every week, we sent out a beautiful, clean, and useful report on what percentage complete each region was. It was our best and most effective weapon. I hardly had to do any convincing to middle management since our divisional director got the same report and could see plain as day how his management team was performing to goal.

Along the way, we had several irritating problems. Sometimes the bar code scanner didn’t work, sometimes the assets wouldn’t load into the database properly, sometimes we knew we had assets on the network but couldn’t find them, and sometimes we just couldn’t get all the data to transfer to the IT database correctly. There was a lot of painstaking work to prioritize the work, get the right fields to populate, and get the regions to follow procedures we put in place. Bit by bit, the database admin got closer and closer to making the tool work more reliably and the screaming from IT became less and less.

The other difficult problem to overcome was to make sure the entire division embraced the asset life cycle process. Technical support engineers were constantly buying or borrowing computing assets from every direction and plopping them on the network. They’d also take them off the network, stick them in a closet or send them to another site and do nothing to track their physical locations. We had to create a thorough process that wasn’t too overbearing but could close all the gaps that humans create for themselves. To this end, we documented a very solid process via a great amount of feedback from engineers at the sites. We also had to put together an enforcement policy so that the process would stick without having to chase people around all the time. We had to automate everything we could. This is a difficult task when talking about physical locations of items that can be moved willy nilly at any time. It was especially difficult in Europe since many of the sites were small and didn’t have the personnel to watchguard the process. Once we did get a reasonable process in place, we set up training sessions across the globe to make sure people were doing things properly. We also set up a way to cross-check the assets we showed on the network with the assets we showed in the asset database. Whenever we had a mismatch, we required my asset management team to investigate it and work with the local support engineer to account for the system properly. Gradually, we closed more and more holes and the accounting of our assets improved quickly.

In some cases, we had to fenagle odd computers so that they would report all the data that IT required. Some systems would easily report out information such as serial number, model number, operating system, hardware configuration, etc. quite easily. Others didn’t. We had to enlist savvier engineers to develop scripts to pull that data out on a case by case basis. Again, this was a painstaking process that often seemed to be a situation of managing exceptions rather than managing the norm.

The last step in the chain, that of dumping our data into IT’s central repository, became a pain in the ass, too. Suddenly, things became political since IT was objecting to our division having its own database. They just wanted us to load our system information into their repository directly. There were too many reasons why we couldn’t do that beyond just the time required to shift gears entirely. I had to get upper management to fight that requirement off. I also worked hard to solve our data transfer problems so that it would be one less thing for IT to complain about. If we could make the transfer work flawlessly, then they would likely leave us alone. Ultimately, that strategy worked. Once we fixed our irritating database problems, the data transfer became routine, they saw our data loading and catching up with all the other divisions, and what was done was done.

In the end, our weekly status reports revealed where regions weren’t performing and enough pressure was put on them to address the problems. I didn’t have to be as much of a bad guy as I did during the information security days. The data spoke for itself. We grew closer and closer to the 90-100% goal in place for tracking our assets and the urgency and hysteria in the program began to wind down. The timing was good since I reached a point where the program could manage itself and my division could prepare itself to see me go on to Russia.


Lessons Learned

As usual, I had to learn how to work with a prickly personality. There’s always one in the bunch, it seems. I found the right balance in working with her. I really wanted her to feel like her opinon mattered and that some of her grandiose plans could be realized. That kept her motivated and supportive. If I had lost her support, she could have brought the whole program down. I had many leadership and management techniques that I could teach her so that she could see how to get things done without acting like a drill sergeant. I think she appreciated that and made it much easier to get her on board.

I also reminded myself how valuable it is to get something to measure as soon as possible. Data draws attention to itself and requires a lot less manual intervention to convince people what goals and behaviors are needed. Getting the team to contribute instead of just having them take orders is another valuable lesson. I can’t tell you how pleased I was to have my French team member create that report. It was a striking moment for the program and I will always encourage people to use their creativity and inherent desire to help as much as possible.

Finding the right people who have the right skill at the right time is an invaluable instinct to have. I seem to possess this skill fairly well and so it’s always good to reach out to people you don’t expect can help to come in and save the day.

Lastly, despite the grandiose plan my asset manager had, I was very glad I didn’t skip over it too much. The IT Asset Life Cycle created a great foundation for our work and it gave us more credibility when people saw that we were using an industry model as our process. Sometimes, people get turned off by this, so you need to use this wisely. But in this case, using a tried and true way with some theory behind it gave us that extra strength to convince people we were doing the right thing.