Patrick Kelley
Cliffside Home
Contact Info

Honey, I Blew Up The Data Center 

Patrick Kelley

It is human nature to look ahead to better times.  Since disasters occur infrequently, we fool ourselves by assuming that disasters will never happen to us, helping us to live happily.  When an organization, public or private, takes the steps necessary to anticipate and plan for disaster, it heartens me.  However, planning is only a partial solution - exercising continuity plans is essential to ensure plan viability and integrity.

What is the best way to test a plan?  There is no one-right answer, but taking a thoughtful, consistent approach to plan testing goes a long way to obtaining reproducible results. By using a continuous quality improvement approach, exercising contingency plans can ensure operational readiness and well-trained participants.  I have found that running a "Full Scale" exercise immediately after the plan is completed is a great way to ferret out plan deficiencies!  Sometimes it takes drastic action to get management's attention.  Also, I find stronger support among managers for full-scale exercises than for other types of exercises.  Full scale exercises can be more easily justified, as exercises take personnel away from day to day productive activities.

The worst way to test a contingency plan is to wait for an actual incident.  This approach supplies only haphazard test coverage, at best.  An ongoing disaster is not conducive to gathering data unemotionally and rationally.  Not to mention that using an untested plan during a real world disaster could be career foreshortening.  Somebody might get hurt, or worse!  Finally, do not underestimate the legal liabilities associated with this approach.

My company's management recognized the need to improve their disaster preparedness plans.  Once on board, I relied on our profession's accepted practice of gaining upper management support.  This involved the creation of a critical department manager committee, an executive oversight committee, and the addition of disaster related agenda items to the existing department director committee meetings.  My intent was to create a business continuity exchange up and down the management chain.  My hope was that this debate would survive after the initial "new plan development push" and so continue disaster recovery discussions when developing new business processes.  Of course, I also had to revalidate the old business impact analysis.  Once this was accomplished, and compromises negotiated, it was apparrent that the current plan had several shortcomings:

  1. The evacuation plan was inadequate.  Training of essential personnel had deteriorated.  For example, many employees did not even know where they were to assemble during a building evacuation.
  2. The data center itself was a "single point of failure," the hub of the entire operation, but not recoverable in case of total disaster.  Even partial operation at remote sites was not assured.  In essence, the company had chosen to "put all their eggs in one basket and watch them."  This is not necessarily  bad or wrong if it is a conscious decision with the risks well understood.
  3. The growth plans for the company did not include considerations to ensure survivability in case of  a major disaster.  Processes that were now included to accomodate growth would require significant review to coincide with the business continuity planned capabilities.

My dilemma was to present my analysis to management without sounding like Chicken Little.  How was I to present my case without coming across as smug, or worse, embarrassing my management team  the authors of the then current plan.  Being a recent addition to the company, I needed a way to prove plan shortcomings and get confirmation from objective third parties who had no "axe to grind."  From my experience with developing exercises for military installations, counties, cities, and the Red Cross, I know that nothing turns heads and churns plans better than a (simulated) full scale disaster!  I also knew that building exercises would be a difficult, costly and time consuming process.  My time constraints were that management needed an exercise plan by the end of the second quarter of 1997, less than six months away.

Now, for all the "experienced types" out there:  think of reestablishing a disaster recovery environment, completing a Business Impact Analysis, composing a "workable" disaster continuity plan, and completing a full scale exercise in six months for a company that has grown five times larger since they wrote the last plan, a company that now has a presence in fifty states, and a current annual gross income approaching $900 Million.

The solution to my exercise development problem came to me at the Disaster Recovery Journal's Business Continuity Conference in San Diego, in March 1997.  Claiming to be the automation tool I needed, Plan AHEAD is the world's first commercially available exercise development software system.  On my initial review of this product, I was impressed that it utilized the industry accepted standards recommended by the Federal Emergency Management Agency (FEMA), the American Red Cross, and even Pacific Bell Telephone.  I quickly realized that if this product did half of what it was supposedly capable of, it was just what I needed.

It may sound strange to hear me tout a product, but you should have seen my manager that day in San Diego.  I practically dragged him to Cliffside Software's booth to show him what I believe to be the best software product in the entire disaster management arsenal.  The presenters from Cliffside could not get a word in edgewise, while my manager could not believe his ears.  He had never heard such a sales story from one of his own employees before, and was flabbergasted that I would wax poetic about a product that I had only gained a cursory knowledge of earlier in the day.  After dealing with disasters for many years, I am generally not that excitable. Yet I was really over the moon about this application, the price was right, so I obtained the product when it first shipped in April of 1997.

I quickly realized that Plan AHEAD was the right software product to fulfill my needs.  Although I have been performing and authoring emergency and disaster exercises for close to three decades, this product really helped me to get organized.  The embedded tutorial cured some fuzzy thinking I had about whether something was a need or an actual exercise objective.  The product gave me a place to write and edit events, messages, the narrative and other exercise elements.  Through its MSEL (Master Sequence of Events Listing), I could ensure that the right messages and expected actions were correctly matched with events.  To have an event / message diagram automatically created so one can ensure proper order is very useful when designing a full scale exercise.  Without it chaos is likely.  This software is definitely a far more powerful exercise design tool than any word processor could ever be.

The evaluation creation process alone is worth the price tag.  The software automatically takes each exercise objective and creates a blank evaluation form.  Measurement criteria can be presented as scalar, time related, multiple choice, or even a grid of questions.  Its ability to automatically format a printed and electronic evaluation form is exceptional.  However, to have these evaluations tied into corrected actions with an audit trail capability is absolutely outstanding.  Plan AHEAD is so flexible that I have even tailored it to help perform business impact analysis!

I was faced with putting together a full scale exercise before the end of Q2-97.  The purpose of the exercise included these components:

  1. Test  the ability of the company to respond during the Initial (Emergency) Response Phase.
  2. Test the ability of the company to transition into the Recovery Phase.
  3. Validate the new business recovery plan.
  4. Provide a base measurement for future improvement.
  5. Provide a "friendly" environment for participants to learn.

Nothing is funny or "cute" about a disaster, but it is one facet of our discipline that simulations can have a macabre humor about them.  Establishing the scenario for an exercise is about the only "funny" part of my job.  So what was the disaster scenario for our exercise?  Since the data center had all the eggs in one basket, I broke the basket by "destroying" it within thirty seconds of the exercise beginning.  It seems a terrorist bomb was "miss-sent" by the post office.  Although a terrorist event may be improbable for a company such as ours (we are eyeglass insurers), it added drama and an element of anxiety.  The point was to pick a hazard that would take out the data center and management's reliance on it. 

The exercise involved close to eighty participants over a day and a half, and since we planned to feed everyone both breakfast and lunch, logistics became a primary challenge.  Plan AHEAD helped facilitate this part of the exercise development process.  It provides a place to document equipment and supplies needed, at various locations, everything from passenger vans, to cell phones, modems and file servers.

I couldn't have asked for better or more informative results from the exercise.  The building evacuation, while meeting Fire Marshall specifications, pointed out deficiencies in our plans.  The destruction of the data center dramatically exposed the difficulties our company would face even obtaining a minimal amount of data processing capabilities from our remote site.

We performed the exercise in July 1997, and by November 1997 management had essentially carried out all of my recommendations.  The most important lesson learned centered on the decentralization of data.  The exercise helped change management's perception of the data centralization problem.  Also, the exercise lessons ensured that the data processing growth plans now consider data management and recovery plans.  Finally, many of our department's budget proposals were also justified thanks to the exercise.

A second lesson learned  the inability to declare a disaster properly, exposed the need to improve management's familiarity with this crucial procedure.  Exercising creates a a safe "makebelieve" situation (free of embarrassment or attribution).  Now management can try their hand at various critical functions, and demonstrate the relative importance of these procedures, without suffering real costs or damage.

For me, the greatest lesson learned was discovering how effective Plan AHEAD is when developing a thorough exercise, helping me capture the results and quickly creating management reports.   I had thought the main use of the product was to assist me in building exercise materials, but I soon realized that it had far greater impact on my entire exercising process. For example, I never would have believed the Plan AHEAD could have been so useful with my Business Impact Analysis.  By using the product I am able to get far more information out of a given exercise.  By analyzing Plan AHEAD's Corrective Actions, I developed a graphic method of critical and support team capabilities in a spreadsheet.  This enabled me to graphically show the company's executives the status of the teams regarding planning, equipment, and training, thus justifying change.

The most profound realization is that I now have a tool which helps me gradually integrate exercising into our standard planning process.  Building an "exercise culture" is a longterm commitment on the part of the entire organization, often taking three or more years.  In my current company, I know that care must be taken to not move too quickly or overwhelm various business units with any one exercise.  Instead I must gradually introduce exercising as a familiar (if not frequent) business activity.  More than just quickly building exercises, with Plan AHEAD I can manage a strategic long term approach to the process of exercising.  I can recall messages, objectives, and functional requirements for various types of exercises.  In essence, I do not have to "reinvent the wheel" with every exercise.

It's true that disasters (even simulated ones) don't happen often.  But it is equally true that without testing your disaster preparedness plans, you'll only find out if they work on the day the big one hits.  By using Plan AHEAD, I was able to get a substantially greater return on the investment my company made in our exercise.  Exercising takes away from production, yet because of the quality of the exercise, I was able to make lasting and significant improvements to my company's disaster preparedness plans and processes.

All within the time line established by the company!!


Patrick Kelley has worked within the disaster response/recovery profession since 1972.  He started in the military and eventually became a Disaster Preparedness NCOIC.  He has been an American Red Cross Disaster Response volunteer for the past twelve years.  He currently works as both the volunteer Chapter Disaster Services Chair at the Sacramento / Sierra Chapter, and as  Reserve Logistics Assistant Officer for the American Red Cross.  His full time position is Disaster Recovery Planner for a vision insurance plan company.  Pat is a current member of the Association of Sacramento Area Planners (ASAP) and the California Emergency Services Association (CESA).


[Cliffside Home] [Plan AHEAD] [Contact Info]

Please send comments to the  Webmaster.