It is human
nature to look ahead to better times. Since disasters occur infrequently, we fool ourselves by assuming that disasters will never happen to us, helping us to live happily. When an organization, public or
private, takes the steps necessary to anticipate and plan for disaster, it heartens me. However, planning is only a partial solution - exercising continuity plans is essential to ensure plan viability and
integrity.What is the best way to test a plan? There is no one-right answer, but taking a thoughtful, consistent approach to plan testing goes a long way to obtaining reproducible results. By using a continuous
quality improvement approach, exercising contingency plans can ensure operational readiness and well-trained participants. I have found that running a "Full Scale" exercise immediately after the plan is completed
is a great way to ferret out plan deficiencies! Sometimes it takes drastic action to get management's attention. Also, I find stronger support among managers for full-scale exercises than for other types of
exercises. Full scale exercises can be more easily justified, as exercises take personnel away from day to day productive activities.
The worst way to test a contingency plan is to wait for an actual
incident. This approach supplies only haphazard test coverage, at best. An ongoing disaster is not conducive to gathering data unemotionally and rationally. Not to mention that using an untested plan
during a real world disaster could be career foreshortening. Somebody might get hurt, or worse! Finally, do not underestimate the legal liabilities associated with this approach.
My company's management
recognized the need to improve their disaster preparedness plans. Once on board, I relied on our profession's accepted practice of gaining upper management support. This involved the creation of a critical
department manager committee, an executive oversight committee, and the addition of disaster related agenda items to the existing department director committee meetings. My intent was to create a business
continuity exchange up and down the management chain. My hope was that this debate would survive after the initial "new plan development push" and so continue disaster recovery discussions when
developing new business processes. Of course, I also had to revalidate the old business impact analysis. Once this was accomplished, and compromises negotiated, it was apparrent that the current plan had
several shortcomings:
- The evacuation plan was inadequate. Training of essential personnel had deteriorated. For example, many employees did not even know where they were to assemble during a building evacuation.
- The data center itself was a "single point of failure," the hub of the entire operation, but not recoverable in case of total disaster. Even partial operation at remote sites was not
assured. In essence, the company had chosen to "put all their eggs in one basket and watch them." This is not necessarily bad or wrong if it is a conscious decision with the risks well
understood.
- The growth plans for the company did not include considerations to ensure survivability in case of a major disaster. Processes that were now included to accomodate growth would require significant
review to coincide with the business continuity planned capabilities.
My dilemma was to present my analysis to management without sounding like Chicken Little. How was I to present my case without coming across as smug, or worse, embarrassing my management team the authors
of the then current plan. Being a recent addition to the company, I needed a way to prove plan shortcomings and get confirmation from objective third parties who had no "axe to grind." From my
experience with developing exercises for military installations, counties, cities, and the Red Cross, I know that nothing turns heads and churns plans better than a (simulated) full scale disaster! I also knew
that building exercises would be a difficult, costly and time consuming process. My time constraints were that management needed an exercise plan by the end of the second quarter of 1997, less than six months away.
Now, for all the "experienced types" out there: think of reestablishing a disaster recovery environment, completing a Business Impact Analysis, composing a "workable" disaster continuity plan,
and completing a full scale exercise in six months for a company that has grown five times larger since they wrote the last plan, a company that now has a presence in fifty states, and a current annual gross income
approaching $900 Million.
The solution to my exercise development problem came to me at the Disaster Recovery Journal's Business Continuity Conference in San Diego, in March 1997. Claiming to be the automation
tool I needed, Plan AHEAD is the world's first commercially available exercise development software system. On my initial review of this product, I was impressed that it utilized the industry accepted standards
recommended by the Federal Emergency Management Agency (FEMA), the American Red Cross, and even Pacific Bell Telephone. I quickly realized that if this product did half of what it was supposedly capable of, it was
just what I needed.
It may sound strange to hear me tout a product, but you should have seen my manager that day in San Diego. I practically dragged him to Cliffside Software's booth to show him what I believe
to be the best software product in the entire disaster management arsenal. The presenters from Cliffside could not get a word in edgewise, while my manager could not believe his ears. He had never heard such
a sales story from one of his own employees before, and was flabbergasted that I would wax poetic about a product that I had only gained a cursory knowledge of earlier in the day. After dealing with disasters for
many years, I am generally not that excitable. Yet I was really over the moon about this application, the price was right, so I obtained the product when it first shipped in April of 1997.
I quickly realized that Plan AHEAD was the right software product to fulfill my needs. Although I have been performing and authoring emergency and disaster exercises for close to three decades, this product
really helped me to get organized. The embedded tutorial cured some fuzzy thinking I had about whether something was a need or an actual exercise objective. The product gave me a place to write and edit
events, messages, the narrative and other exercise elements. Through its MSEL (Master Sequence of Events Listing), I could ensure that the right messages and expected actions were correctly matched with
events. To have an event / message diagram automatically created so one can ensure proper order is very useful when designing a full scale exercise. Without it chaos is likely. This software is
definitely a far more powerful exercise design tool than any word processor could ever be.
The evaluation creation process alone is worth the price tag. The software automatically takes each exercise objective
and creates a blank evaluation form. Measurement criteria can be presented as scalar, time related, multiple choice, or even a grid of questions. Its ability to automatically format a printed and electronic
evaluation form is exceptional. However, to have these evaluations tied into corrected actions with an audit trail capability is absolutely outstanding. Plan AHEAD is so flexible that I have even tailored it
to help perform business impact analysis!
I was faced with putting together a full scale exercise before the end of Q2-97. The purpose of the exercise included these components:
- Test the ability of the company to respond during the Initial (Emergency) Response Phase.
- Test the ability of the company to transition into the Recovery Phase.
- Validate the new business recovery plan.
- Provide a base measurement for future improvement.
- Provide a "friendly" environment for participants to learn.
Nothing is funny or "cute" about a disaster, but it is one facet of our discipline that simulations can have a macabre humor about them. Establishing the scenario for an exercise is about the only "funny" part
of my job. So what was the disaster scenario for our exercise? Since the data center had all the eggs in one basket, I broke the basket by "destroying" it within thirty seconds of the exercise
beginning. It seems a terrorist bomb was "miss-sent" by the post office. Although a terrorist event may be improbable for a company such as ours (we are eyeglass insurers), it added drama and an
element of anxiety. The point was to pick a hazard that would take out the data center and management's reliance on it.
The exercise involved close to eighty participants over a day and a half, and since
we planned to feed everyone both breakfast and lunch, logistics became a primary challenge. Plan AHEAD helped facilitate this part of the exercise development process. It provides a place to document
equipment and supplies needed, at various locations, everything from passenger vans, to cell phones, modems and file servers.
I couldn't have asked for better or more informative results from the exercise. The
building evacuation, while meeting Fire Marshall specifications, pointed out deficiencies in our plans. The destruction of the data center dramatically exposed the difficulties our company would face even
obtaining a minimal amount of data processing capabilities from our remote site.
We performed the exercise in July 1997, and by November 1997 management had essentially carried out all of my recommendations. The
most important lesson learned centered on the decentralization of data. The exercise helped change management's perception of the data centralization problem. Also, the exercise lessons ensured that the data
processing growth plans now consider data management and recovery plans. Finally, many of our department's budget proposals were also justified thanks to the exercise.
A second lesson learned the inability
to declare a disaster properly, exposed the need to improve management's familiarity with this crucial procedure. Exercising creates a a safe "makebelieve" situation (free of embarrassment or
attribution). Now management can try their hand at various critical functions, and demonstrate the relative importance of these procedures, without suffering real costs or damage.
For me, the greatest lesson
learned was discovering how effective Plan AHEAD is when developing a thorough exercise, helping me capture the results and quickly creating management reports. I had thought the main use of the product was
to assist me in building exercise materials, but I soon realized that it had far greater impact on my entire exercising process. For example, I never would have believed the Plan AHEAD could have been so useful with my
Business Impact Analysis. By using the product I am able to get far more information out of a given exercise. By analyzing Plan AHEAD's Corrective Actions, I developed a graphic method of critical and
support team capabilities in a spreadsheet. This enabled me to graphically show the company's executives the status of the teams regarding planning, equipment, and training, thus justifying change.
The most
profound realization is that I now have a tool which helps me gradually integrate exercising into our standard planning process. Building an "exercise culture" is a longterm commitment on the part of the
entire organization, often taking three or more years. In my current company, I know that care must be taken to not move too quickly or overwhelm various business units with any one exercise. Instead I must
gradually introduce exercising as a familiar (if not frequent) business activity. More than just quickly building exercises, with Plan AHEAD I can manage a strategic long term approach to the process of
exercising. I can recall messages, objectives, and functional requirements for various types of exercises. In essence, I do not have to "reinvent the wheel" with every exercise.
It's true that
disasters (even simulated ones) don't happen often. But it is equally true that without testing your disaster preparedness plans, you'll only find out if they work on the day the big one hits. By using Plan
AHEAD, I was able to get a substantially greater return on the investment my company made in our exercise. Exercising takes away from production, yet because of the quality of the exercise, I was able to make
lasting and significant improvements to my company's disaster preparedness plans and processes.
All within the time line established by the company!!
