Course Overview

The Course Objectives include:
  • Develop a baseline knowledge of exercise fundamentals.
  • Identify the tasks necessary to complete each phase of the exercise process.
  • Define how exercises complete the preparedness process.
  • Identify the role of exercises in validating capabilities.
  • Identify phases of exercise evaluation and the improvement planning process.
The following topics are included in this course

HSEEP Fundamental Principles. You learned the fundamental HSEEP principles of; guidance by elected and appointed officials, the National Preparedness Goal (NPG), the Progressive Planning Approach, the importance of the whole community integration, risk management and a common methodology for exercises.

The reasons for conducting exercises. You learned the reasons for conducting exercises which included- National Preparedness Components, clarifying responsibilities and roles, improving interagency coordination and communication, identifying gaps in resources, developing individual performance, and identifying areas/opportunities for improvement.

Identification of how the participant has shareholder support. You learned how participants have shareholder support by engaging the whole community, and state and local jurisdictions.

Discussion-based exercises. You learned about discussion-based exercises as- a forum for discussing or developing plans and procedures, less complicated than operations-based exercises, focused on strategy and policy.

Operations-based exercises. You learned about operations-based exercises and their characteristics which included: drills, the involvement of deployment of resources and personnel, complexity compared to discussion based exercises, and their advantages including the improvement of individual and team performances.

Preparedness Resource Library

We encourage you to review and become familiar with the following documents that support this course:
  • Presidential Policy Directive 8
  • National Preparedness Goal
  • National Preparedness System
  • Overview of the National Planning Frameworks
  • Threat and Hazard Identification and Risk Assessment Guide: Comprehensive Preparedness Guide(CPG) 201
  • FEMA Preparedness Toolkit/HSEEP Policy and Guidance
  • Homeland Security Digital Library
  • FEMA Home Page

Course Overview

This course provides an introduction into Exercises, as a form of practice, to achieve National Preparedness Goals.

Objectives: After completing this course, you will be able to:

  • Develop a baseline knowledge of exercise fundamentals.
  • Identify the tasks necessary to complete each phase of the exercise process.
  • Define how exercises complete the preparedness process.
  • Identify the role of exercises in validating capabilities.
  • Identify phases of exercise evaluation and the improvement planning process.

Receiving Credit

Each lesson takes a variable amount of time to complete. If you are unable to complete the course in its entirety, you may close the window and reopen the course at any time. However, depending on the system used to take the course, it is possible you may have to repeat a portion of the last lesson you were studying.

Welcome to IS-120.c, An Introduction to Exercises

Exercises give communities, states, and regions a set of essential tools to prevent, prepare for, respond to, and recover from disasters.

The Homeland Security Exercise and Evaluation Program (HSEEP) provides a set of guiding principles for exercise programs and a common approach to exercise program management, design and development, conduct, evaluation, and improvement planning.

This course introduces the basics of emergency management exercises. It also builds a foundation for subsequent exercise courses, which provide the specifics of HSEEP.

An Introduction to Exercises is designed for personnel from all emergency management sectors to understand the basic fundamentals of exercise design, development, conduct, evaluation, and improvement planning.

Lesson Synopses

This course contains eight lessons.

The remainder of Lesson 1, Introduction and Program Fundamentals, covers HSEEP fundamentals, reasons to exercise, and the types of exercises.

  • Lesson 2, Exercise Management Program, delves deeper into the functionality, goals, and outcomes from the different types of discussion- and operations-based exercises, and exercise program management.
  • Lesson 3, Exercise Overview, discusses multi-year exercise planning priorities, the benefits of exercise foundation, and defines the exercise cycle.
  • Lesson 4, Establishing Exercise Foundation, steps through building the foundation from the prject management timeline to planning the activities based on the capabilities to be tested, and scheduling the appropriate planning meetings.
  • Lesson 5, Design and Development Phase, defines key elements of the design including,the objectives and scope of the exercise and the scenario to be used. This is where required documentation is determined and developed and needed resources are identified.
  • Lesson 6, Conduct, covers the actions taken for discussion- and operations-based exercises prior to, during conduct and wrap-up activities at the end of the exercise.
  • Lesson 7, Evaluation, addresses evaluator selection and required documentation and methods of observation and data collection.
  • Lesson 8, Improvement Planning, provides insight into corrective action and improvements following exercise evaluation analysis.

 

Lesson 1 Overview

This lesson provides the fundamental background of National Preparedness and the role exercises play in the process to protect our Nation.

Objectives: After completing this lesson, you will be able to:

  • Identify the benefits of using the HSEEP methodology to plan and carry out effective exercises.
  • Explain the National Preparedness System (NPS) components.
  • Explain the purpose of Core Capabilities.

What is HSEEP?

HSEEP is a consistent approach to capabilities-based exercise program management that uses a common methodology to measure progress toward building, sustaining, and delivering core capabilities. HSEEP provides a set of guiding principles for exercise programs and a common approach to exercise program management.

Exercise:

  • (n) Something performed or practiced in order to develop, improve, or display a specific power or skill.
  • (v) To practice in order to train, strengthen, or develop

Source: Merriam Webster’s Dictionary

Exercises improve readiness by:

  • Involving the whole community.
  • Providing a way to evaluate operations and validate plans and capabilities.
  • Reinforcing teamwork.
  • Identify both capability gaps and areas for improvement.
  • Demonstrating a community’s resolve to prepare for disastrous and catastrophic events.

Boston Marathon Bombing

Boston Marathon bombers didn’t know they were attacking a city that had relentlessly drilled for such a catastrophe—not just security personnel, but Whole Community participants usually sidelined in other cities’ disaster rehearsals.

“Planning, training, exercising, sometimes it gets a little boring, repetitious, until you see what happens in the real world.”

—James Hooley, Chief of Emergency Medical Services, Boston, MA

“Leaders should try to build cores of citizen responders. No city will be able to manage a crisis and rebuild…without the help of its people.”

—Thomas Menino, Former Mayor, Boston, MA

HSEEP Fundamental Principles

The fundamental principles of HSEEP are:

  • Guided by Elected and Appointed officials
  • Capability-based, Objective Driven
  • Progressive Planning Approach
  • Whole Community Integration
  • Informed by Risk
  • Common Methodology

Elected and Appointed Officials

It is important to engage elected and appointed officials early and as frequently as possible in the exercise process. They provide strategic direction with specific guidance for individual exercises. They can also advocate for budget and funding requirements, as well as community participation and support for the program.

Routine engagement with elected and appointed officials ensures that exercises have the support necessary for success.

Capability-based, Objective Driven

Capabilities can be delivered with any combination of properly planned, organized, equipped, trained, and exercised personnel to achieve an intended target.

Exercises are built around core capabilities as identified in the National Preparedness Goal (NPG). NPG identifies 32 core capabilities that are associated with the five mission areas: Prevention, Protection, Mitigation, Response, and Recovery. When capabilities are aligned with mission areas the outcome examines current capability levels and identifies gaps, or opportunities for improvement.

Using the HSEEP methodology, organizations can use exercises to focus on assessing performance against capability-based objectives.

Progressive Planning Approach

A progressive planning approach employs the use of various exercise types aligned to a common set of program objectives within a cycle of exercises. The complexity of the exercises increases over time.

Senior leadership positions in state, territory, tribal, and local government, and organizations of all sizes can provide direction on the principal objectives that guide each organization’s approach to planning and preparedness.

Whole Community Integration

The whole community approach focuses on enabling participation of a wider range of all levels of government, emergency, private, and nonprofit sectors to foster better coordination and working relationships.

Exercise planners are encouraged to engage the whole community, where appropriate, throughout exercise program management, design and development, conduct, evaluation, and improvement planning.

Whole Community Integration (cont’d)

Whole community integration may include the entities listed below:

  • Representatives from relevant disciplines, first responders or utility companies, for example
  • Senior leaders or those responsible for providing support and/or resources
  • Community member representation
  • Representatives with administrative responsibility
  • Representatives from volunteer, nongovernmental or nonprofit sectors, and social support organizations such as advocates for children, seniors, limited access and functional needs, limited language proficiency, and racially and/or ethnically diverse neighborhoods

Informed by Risk

Identifying and assessing risks and associated impacts helps organizations identify priorities, objectives, and core capabilities to be evaluated through exercises.

Planners should first identify pertinent risks, and then assess the potential impacts associated with each risk. For example, looking at a community’s Threat and Hazard Identification and Risk Assessment (THIRA) is a good starting place for exercise program priorities.

Common Methodology

HSEEP includes a common methodology for exercises that is applicable to all mission areas—prevention, protection, mitigation, response, and recovery, as defined in the National Preparedness Goal. This methodology enables organizations of divergent sizes, geographies, and capabilities to have a shared understanding of exercise program management, design and development, conduct, evaluation, and improvement planning; and fosters exercise-related interoperability and collaboration.

The Exercise Cycle is the common planning methodology provided by HSEEP for use with all exercise types. Regardless of scope or scale of an exercise, the exercise cycle will include exercise program management, design and development, conduct, evaluation, and improvement planning.

Presidential Policy Directive 8

The Presidential Policy Directive 8 (PPD-8) describes the Nation’s approach to national preparedness.

The National Preparedness Goal (NPG) is the cornerstone for the implementation of PPD-8. The Nation’s core capabilities are identified across five mission areas: Prevention, Protection, Mitigation, Response, and Recovery. The NPG identifies the desired achievements and the goals that have been set.  Mission areas and core capabilities will be covered in depth later in this lesson.

The National Preparedness System (NPS) is the mechanism the Nation uses to build, sustain, and deliver the core capabilities to meet the NPG and meet the goal of a secure and resilient Nation. The NPS helps define, the process communities can use to reach identified preparedness goals. The NPS includes a series of five national planning frameworks, one for each mission area. Each framework provides the following guidance:

  • Clarification and assignment of roles and responsibilities
  • Core capabilities for the specific mission area
  • Critical tasks for each of the core capabilities
  • Direction for delivering the core capabilities

Why Exercise?

Through exercises, whole community stakeholders evaluate and validate plans and capabilities, and identify capability gaps and opportunities for improvement. They bring together and strengthen the whole community in its’ efforts to prevent, protect against, mitigate, respond to, and recover from all types of hazards.

HSEEP provides a common approach and consistent methodology for exercise planning. It is flexible, scalable, and available to all organizations regardless of size or scope of the organization or planned activity.

Exercises help to:

  • Clarify roles and responsibilities
  • Improve interagency coordination
  • Find resource gaps
  • Develop individual performance
  • Identify opportunities for improvement

National Preparedness System

The National Preparedness System (NPS) outlines a process that can be used by communities in preparedness activities to achieve the National Preparedness Goal (NPG).

NPS consists of six components, including identify and assess risk, estimate capability requirements, build and sustain capabilities, plan to deliver capabilities, validate capabilities, and review and update risks, tools, and resources. Together, these components provide communities with a consistent and reliable approach for identifying high-priority capability targets.

Advance to the next screen for descriptions of each component.

National Preparedness System Component Descriptions

Component Component Description
Identify and Assess Risk Determine which capabilities should be evaluated, review historical and current data on existing, potential, and perceived threats for insight into the community’s current strengths and opportunities for improvement.
Estimate Capability Requirements Determine which resources are required to address identified risks and identify where the community currently stands in meeting those requirements.
Build and Sustain Capabilities Determine the best way to use or allocate resources to build capabilities.
Plan to Deliver Capabilities The foundation of the National Preparedness System (NPS) is the integration of the whole community in the planning process. Integration helps ensure that exercise planning accounts for core capability relationships and dependencies across mission areas.
Validate Capabilities Capability validation occurs during the commission of the planned exercise. Measuring progress toward achieving the National Preparedness Goal assists in resource allocation and preparedness priorities.
Review and Update Risks, Tools, and Resources Regular review and update of the tools and resources used to address these components.

National Preparedness Goal

The National Preparedness Goal presents an integrated, layered, and whole community approach to preparedness. The Goal, itself, is a result of contributions from the whole community. It recognizes that everyone can contribute to and benefit from national preparedness efforts.

National Preparedness Goal (cont’d)

The National Preparedness Goal sets the vision for preparedness nationwide and identifies the core capabilities necessary to achieve that vision across the following five mission areas:
  1. Prevention
  2. Protection
  3. Mitigation
  4. Response
  5. Recovery
Keep in mind, these five mission areas aid in organizing our National preparedness activities and enhance coordination of the core capabilities within each mission area. Successful achievement of the National Preparedness Goal will result in a secure and resilient Nation with the capabilities required across the whole community to prevent, protect against, mitigate, respond to, and recover from the threats and hazards that pose the greatest risk.

Visit the Preparedness Resource Library page to select the National Preparedness Goal

National Preparedness Goal: Capabilities and Mission Areas

The emphasis of the National Preparedness Goal is on building and sustaining core capabilities across the five mission areas.
National Preparedness Goal: A secure and resilient Nation with the capabilities required across the whole community. It establishes Core Capabilities for executing the Mission Areas of prevention, protection, mitigation, response, and recovery.
  • Prevention: The capabilities necessary to avoid, prevent, or stop a threatened or actual act of terrorism.
  • Protection: The capabilities necessary to secure the homeland against acts of terrorism and manmade or natural disasters.
  • Mitigation: The capabilities necessary to reduce loss of life and property by lessening the impact of disasters.
  • Response: The capabilities necessary to save lives, protect property and the environment, and meet basic human needs after an incident has occurred.
  • Recovery: The capabilities necessary to assist communities affected by an incident to recover effectively.

What are Core Capabilities

Preparedness is the responsibility of an entire Nation. The way to measure, describe, and implement our security and resilience techniques is through core capabilities.

Core capabilities are used to measure, describe, and implement our security and resilience techniques. There are 32 core capabilities identified in the National Preparedness Goal, each classified beneath one of the five mission areas.

The core capabilities:

  • Are distinct critical elements necessary to meet the National Preparedness Goal (NPG)
  • Are essential for the execution of each of the five mission areas
  • Provide a common language for preparedness across the whole community
  • Are not exclusive to any single level of government or organization

What are Core Capabilities? (cont’d)

Utilizing and implementing core capabilities is what we, as a Nation, require in order to deal with the risks we face.

As we begin to look at the core capabilities that fall under the five mission areas, notice that three capabilities span all mission areas: Planning, Public Information and Warning, and Operational Coordination. These help to unify mission areas and, in many ways, are necessary for the success of the remaining core capabilities. In addition, many core capabilities involve more than one mission area and are displayed in multiple mission areas, as appropriate.

For additional information on Core Capabilities, visit the Preparedness Resource Library and select the FEMA Home Page, then search for Core Capabilities.

For additional information on Core Capabilities, visit the Preparedness Resource Library and select the FEMA Home Page, then search for Core Capabilities.

Common Core Capabilities

There are three core capabilities that span all five mission areas: Planning, Public Information and Warning, and Operational Coordination. Within mission areas, there are capabilities that are unique to the specific mission area, but not unrelated.

Planning: Conduct a systematic process engaging the whole community as appropriate in the development of executable strategic, operational, and/or tactical-level approaches to meet defined objectives.

Public Information and Warning: Deliver coordinated, prompt, reliable, and actionable information to the whole community through the use of clear, consistent, accessible, and culturally and linguistically appropriate methods to effectively relay information regarding any threat or hazard, as well as the actions being taken and the assistance being made available, as appropriate.

Operational Coordination: Establish and maintain a unified and coordinated operational structure and process that appropriately integrates all critical stakeholders and supports the execution of core capabilities.

** Remember, all core capabilities under each specific mission area is intended to meet the NPG.

Core Capabilities by Mission Area

All mission areas include the common core capabilities, Planning, Public Information and Warning, and Operational Coordination. In addition, the Prevention mission area includes the following core capabilities:

Mission Area Mission Are Description Core Capabilities
Prevention Prevention comprises the capabilities necessary to avoid, prevent, or stop a threatened or actual act of terrorism.
  • Forensics and Attribution
  • Intelligence and Information Sharing
  • Interdiction and Disruption
  • Screening, Search, and Protection
Protection Protection includes the capabilities to safeguard the homeland against acts of terrorism and manmade or natural disasters.
  • Access Control and Identity Verification
  • Cybersecurity
  • Physical Protective Measures
  • Risk Management for Protection Programs and Activities
  • Supply Chain Integrity and Security

PLUS, Capabilities shared with the Prevention mission area:

  • Intelligence and Information Sharing
  • Interdiction and Disruption
  • Screening, Search, and Protection
Mitigation Mitigation includes capabilities necessary to reduce loss of life and property be lessening the impact of disasters.
  • Community Resilience
  • Long-Term Vulnerability Reduction
  • Risk and Disaster Resilience Assessment
  • Threats and Hazards Identification
Response Response includes capabilities necessary to save lives, protect property and the environment, and meet basic human needs after an incident has occurred.
  • Critical Transportation
  • Environmental Response/Health and Safety
  • Fatality Management Services
  • Fire Management and Suppression
  • Infrastructure Systems
  • Logistics and Supply Chain Management
  • Mass Care Services
  • Mass Search and Rescue Operations
  • On-scene Security, Protection, and Law Enforcement
  • Operational Communications
  • Public Health, Healthcare, and Emergency Medical Services
  • Situational Assessment
Recovery Recovery includes capabilities necessary to save lives, protect property and the environment, and meet basic human needs after an incident has occurred.
  • Infrastructure Systems (Shared capability with Response mission area)
  • Economic Recovery
  • Health and Social Services
  • Housing
  • Natural and Cultural Resources

Lesson 1 Summary

In this lesson you learned about the fundamentals of HSEEP, including whole community approach and the reasons for exercising. You also learned about the National Preparedness Goal, 5 mission areas, and 32 core capabilities and how they are used to support and measure jurisdictional, organizational, and National preparedness.

Objectives: Having completed this lesson, you are able to:

  • Identify the benefits of using the HSEEP methodology to plan and carry out effective exercises.
  • Explain the National Preparedness System (NPS) components.
  • Explain the purpose of Core Capabilities.
The next lesson presents information on the Exercise Management Program.

Lesson 2 Overview

This lesson provides an understanding of exercise program management approaches and exercise types with explanations and guidelines for usage.

Objectives: After completing this lesson, you will be able to:

  • Explain exercise management program.
  • Define the progressive approach to exercises.
  • Explain the purpose and use of discussions-based exercises, including goals, conduct, types and outcomes.
  • Explain the purpose and use of operations-based exercises, including goals, conduct, types and outcomes.

Exercise Management Program Overview

Exercise management program is the ongoing process of overseeing and integrating a variety of exercises over time.

An effective exercise program helps organizations maximize efficiency, resources, time, and funding by ensuring that exercises are part of a coordinated, integrated approach to building, sustaining, and delivering core capabilities. This approach—called multi-year planning—begins when elected and appointed officials, working with whole community stakeholders, identify and develop a set of multi-year exercise priorities informed by existing assessments, strategies, and plans. These long-term priorities help exercise planners design and develop a progressive program of individual exercises to build, sustain, and deliver core capabilities.

Effective exercise program management promotes a multi-year approach to:

  • Engaging elected and appointed officials.
  • Establishing multi-year exercise program priorities
  • Developing a multi-year TEP.
  • Maintaining a rolling summary of exercise outcomes.
  • Managing exercise program resources.

Multi-year Exercise Program Priorities

Exercising is a continual process where previous risk assessments, jurisdictional and threat changes are analyzed to establish new preparedness goals to be evaluated in the next series of exercises.

The multiyear approach is driven by these steps:

  • Engaging elected and appointed officials
  • Establishing multiyear exercise program priorities
  • Developing a multi-year Training and Exercise Plan (TEP)
  • Maintaining a rolling summary of exercise outcomes
  • Managing exercise program resources

Progressive Approach

A progressive exercise program is a series of exercises tied to a set of common program priorities. Each exercise builds on previous exercises using more sophisticated simulation techniques or requiring more preparation time, personnel, and planning.

Regardless of exercise type, each exercise within the progressive series is linked to a set of common program priorities and designed to evaluate associated capabilities.

Effective planning of exercises and integration of the necessary training will reduce the waste of limited exercise resources and serve to address known shortfalls prior to the conduct of the exercise.

The different types of exercises that may be included in a multi-year plan are described in the following sections.

Discussion-based Exercises

Discussion-based exercises provide a forum for discussing or developing plans, agreements, training and procedures. They are generally less complicated than operations-based types. Typically, discussion-based exercises focus on strategic, policy-oriented issues and they do not involve deployment of resources.

Discussion-based “action” comes from facilitated discussion with participants, either as a whole group or in break-out sessions. Facilitators are essential to keeping the discussions on track to meet exercise objectives.

Discussion-based goals Discussion-based conduct
Familiarize players with plans, policies, procedures, and agreements Site setup

  • Usually an indoor venue
  • Includes registration, refreshments, identification tags
  • Requires audio/visual equipment and participant tables
Develop new plans, policies, procedures, and agreements Guided presentation

  • A central tool for facilitating discussion
  • Usually involves multimedia (with video, sound, and graphics)
  • Often used to present scenario narratives (by module)
Facilitated discussion

  • Guided discussion aimed at meeting exercise objectives
  • Style varies by exercise type (i.e., formal vs. informal)
  • Often led by functional subject-matter experts

Discussion-based Exercises – Types, Goals, and Conduct

The four types include seminars, workshops, tabletop exercises (TTX’s), and games. A facilitator or a presenter usually leads the discussions in these exercises, helping to keep participants on track and ensuring that exercise objectives are met.

These types of exercises help to familiarize participants with current plans, policies, agreements, and procedures and are used to develop new ones.

Type of Exercise Participant Goals Conduct Characteristics Exercise Outcomes
Seminar – Seminars orient participants to or provide an overview into strategies, plans, policies, or procedures. Seminars can be valuable when an entity is developing new plans or making changes to existing plans or procedures.
  • Orient participants to new or existing plans, policies, or procedures
  • Research or assess interagency capabilities or inter-jurisdictional operations
  • Construct a common framework of understanding
  • Causal atmosphere
  • Minimal time constraints
  • Lecture-based
Workshop – Workshops are more structured than seminars. Participant attendance and collaboration from relevant stakeholders is essential to obtain consensus and produce effective plans, procedures, and agreements.
  • Develop a written product as a group, in coordinated activities
  • Obtain consensus
  • Collect or share information

 

  • Broad attendance by relevant stakeholders
  • Conducted based on clear objectives/goals
  • More participant discussion than lecture-based seminar
  • Frequently uses break-out sessions to explore parts of an issue with similar groups
  • Emergency Operations Plans (EOPs)
  • Mutual Aid Agreements
  • Standard Operations Procedures (SOPs)
Tabletop Exercise (TTX) – Tabletop exercises facilitate conceptual understanding, identify strengths, and areas for improvements, and/or achieving changes in perceptions. Participants are encouraged to problem-solve together through in-depth discussion. An effective TTX comes from active participants and their assessment of recommended revisions to current plans, policies, and procedures. It is important to have a facilitator that will keep the participants focused on the exercise objectives.
  • Enhance general awareness
  • Enhance roles and responsibility understanding
  • Validate plans and procedures
  • Rehearse concepts and/or assess types of systems in a defined incident
  • Requires an experienced facilitator
  • In-depth discussion
  • Low stress, problem-solving environment
Game – A simulation of operations that often involves two or more teams, usually in a competitive environment, using rules, data, and procedures designed to depict an actual or hypothetical situation. Identifying critical decision-making points is a major factor in the success of games.
  • Explore decision-making processes and consequences
  • Conduct “what-if” analyses of existing plans
  • Evaluate existing and potential strategies
  • No actual resources used
  • Often involves two or more teams
  • Includes models and simulations on increasing complexity as the game progresses
  • May include pre-scripted messages

Operations-based Exercises

Operations-based exercises involve deployment of resources and personnel. They are more complex than discussion-based exercises. Focusing on action-oriented activities, operations-based exercises are used to validate plans, policies, agreements, and procedures, clarify roles and responsibilities, and identify resource gaps and improvement opportunities.

These types of exercises are characterized by actual implementation in reaction to an exercise scenario.

Operations-based goals Operations-based conduct Exercise Outcomes
Validate plans, policies, agreements, and procedures. Site setup

  • Done by the planning team the day before conduct
  • Can be in an indoor or outdoor venue
Site setup may include:

  • Response Route – routes to the simulated incident
  • Response Area – location of exercise activities
  • Assembly Area – location of deployable resources participating in the exercise
  • Observer/Media Area – designated viewing area
  • Simulation Cell – location generating scenario injects
  • Registration – to ensure only authorized personnel are allowed on scene
  • Parking
Clarify roles and responsibilities, and identify resource gaps and improvement opportunities. Exercise briefings

  • Educate participants in their roles and responsibilities
  • Provide safety information to all personnel
  • Vary among roles (controllers, evaluators, players, and actors)
  • Explain exercise play rules (which vary for each exercise)

 

Exercise play
Coordinating logistics is a critical element in operations-based exercise conduct because of the large number of personnel and equipment involved.

Operations-based Exercises – Types, Goals, and Conduct

The three types of operations-based exercises include drills, functional exercises, and full-scale exercises.

Operations-based exercises are used to validate plans, policies, agreements, and procedures. These exercises are characterized by actual implementation of response activities in reaction to an exercise scenario.

Type of Exercise Participant Goals Conduct Characteristics Exercise Outcomes
Drill – A drill is a coordinated, supervised activity usually employed to validate a specific function or capability in a single agency organization. Drills are commonly used to provide training on tasks specific to new equipment or procedures, to introduce or validate procedures, or practice and maintain current skills.
  • Provide training on new equipment
  • Evaluate new procedures, policies, and/or equipment
  • Practice and maintain skills
  • Prepare for more complex exercises
  • Immediate feedback
  • Realistic but isolated environment
Functional Exercise (FE) – Functional exercises are designed to validate and evaluate capabilities, multiple functions and/or sub-functions, or interdependent groups of functions. FEs are typically focused on exercising plans, policies, procedures, and staff members involved in management, direction, command, and control functions.
  • Validate and evaluate capabilities
  • Focused on plans, policies, and procedures

 

  • Conducted in a realistic, real-time simulated environment
  • Simulated deployment of resources and personnel
  • Use of SimCell and Master Scenario Events List (MSEL)
  • Include controller and evaluators

 

 
Full-Scale Exercise (FSE)  Full-scale exercises (FSE) are high stress multi-agency, multi-jurisdictional activities designed to test coordinated responses and rapid problem solving skills. These are the most complex, resource-intensive, and possible expensive exercises.
  • Demonstrate roles and responsibilities as addressed in plans and procedures
  • Coordinate between multiple agencies, organizations and jurisdictions
  • High stress environment
  • Rapid problem solving
  • Critical thinking
  • Conducted in a realistic, real-time environment to mirror a real incident
  • Mobilization of units, personnel, and equipment

Lesson 2 Summary

In this lesson, you learned about exercise management program, the multi-year and progressive approach to developing exercises, discussion-based and operations-based exercises, the types of each, and when they might be used.

Objectives: Having completed this lesson, you are able to:

  • Explain exercise management program.
  • Define the progressive approach to exercises.
  • Explain the purpose and use of discussions-based exercises, including goals, conduct, types and outcomes.
  • Explain the purpose and use of operations-based exercises, including goals, conduct, types and outcomes.
The next lesson presents an Exercise Overview.

Lesson 3 Overview

This lesson provides an overview of the necessary components to build an exercise foundation.

Objectives: After completing this lesson, you will be able to:

  • Identify the components of developing and executing an exercise program.
  • Describe the purpose of the Training and Exercise Plan (TEP).
  • Explain the purpose of the Training and Exercise Planning Workshop (TEPW).

Exercise Foundation Key Factors

The exercise foundation is a compilation of factors that drive the exercise design and development process. Before starting the design, exercise program managers should review and consider the following items to set the foundation for an individual exercise:

  • Elected and appointed officials’ intent and guidance
  • Multi-year Training and Exercise Program (TEP)
  • Relevant AAR/IPs from real-world events and exercises
  • Threat and Hazard Identification and Risk Assessment (THIRA) or other risk, threat, and hazard assessments
  • Organizational plans and procedures
  • Grant or cooperative agreement requirements

By reviewing these elements, exercise program managers can ensure that the exercise both builds and sustains a jurisdiction’s capabilities while taking previous lessons learned into account in the exercise design process.

Exercise Cycle

The Exercise Cycle is provided by HSEEP for all exercise types and scopes. Exercise program management is the coordination of multiple exercises. Exercise project management is the coordination of a single exercise.

Project managers are responsible for:

  • Design and development (Design and Development)
  • Execution of a specific exercise (Conduct)
  • Evaluation (Evaluation)
  • Planning improvements (Improvement Planning)

Good project management involves:

  • Developing a project management timeline
  • Establishing project milestones
  • Identifying the exercise planning team
  • Scheduling planning meetings
These project management tasks are the foundation of every exercise. Without them, other tasks and stages of the exercise planning cycle could not happen.

Multi-year Training and Exercise Plan (TEP) Priorities

Multi-year exercise program priorities are the outcome of leadership’s assessment of hazards and risks related to specific core capability targets. At the program management level, the following priorities guide the development of exercise objectives, ensuring events that will form a progressive, comprehensive, integrated exercise program:

  • Identified by elected and appointed officials
  • Guide program planning and resource allocation
  • Determine type and range of training/exercises
  • Provide a roadmap for selection and prioritizing individual exercise development

 

Training and Exercise Planning Workshop (TEPW)

The Training and Exercise Planning Workshop (TEPW), lays the foundation for the strategy and structure of an exercise program. The TEPW purpose is to engage elected and appointed officials in identifying exercise program priorities and planning a schedule of training and exercise events to meet those priorities.

An essential component of the exercise management process, the TEPW is a collaborative environment where whole community stakeholders can engage in a forum to discuss and coordinate training and exercise activities across local organizations to maximize the use of available resources and prevent duplication of effort.

During the workshop, participants:

  • Review program accomplishments to date
  • Review each jurisdictions progress and accomplishments over the past year, or since the last TEPW
  • Identify needs and modifications required—including changes to the midyear training and exercise schedule and planning that might need updating
  • Translate needs into priorities and develop specific objectives to address through exercises
  • Compare Improvement Plan (IP) actions against current capabilities, training, and exercises
  • Identify and coordinate potential funding sources
  • Coordinate exercise activities and scheduling—a major part of the workshop should be spent on schedule coordination. The workshop is an excellent opportunity for all jurisdictions to coordinate their exercise schedules to avoid duplication of efforts and collaborate to maximize resources

TEPWs are convened on a periodic basis, usually annually or biannually, based on the program needs.

The outcome of the TEPW is the Multi-year Training and Exercise Plan.

Multi-year Training and Exercise Schedule

The multi-year training and exercise schedule should reflect a progressive program of increasingly complex exercises with each building on the capabilities evaluated and validated in previous exercises. Coordinating a program’s various exercises and exercise series is a crucial part of a Multi-year Training & Exercise Plan.

The schedule should include a graphic representation of the proposed activities scheduled in the multi-year exercise plan, it should emphasize coordination through jurisdictions, and allow adequate time for progressive exercises designed to build and validate capabilities.

Program managers use the Multi-year Training and Exercise Schedule to:

  • Avoid duplicating their efforts
  • Combine exercises and ensure the exercises don’t conflict
  • Combine training and ensure training does not conflict
  • Optimize and combine funding where possible
  • Prevent “over” training and exercising

Lesson 3 Summary

In this lesson, you learned about the components of an exercise foundation, including, the Multi-year Training and Exercise Plan and the Training and Exercise Planning Workshop.

Objectives: Having completed this lesson, you are able to:

  • Identify the components of developing and executing an exercise program.
  • Describe the purpose of the Training and Exercise Plan (TEP).
  • Explain the purpose of the Training and Exercise Planning Workshop.
The next lesson presents information on Establishing an Exercise Foundation

Lesson 4 Overview

This lesson provides an inside look at the Exercise Planning Team responsibilities and scheduled meetings based on exercise type.

Objectives: After completing this lesson, you will be able to:

  • Identify the roles and responsibilities of the exercise planning team.
  • Identify discussion points to be addressed during a concept and objectives meeting.
  • Identify the primary focus, discussion points, tools, and outcomes of the three planning meetings.
  • List the major components of a master scenario events list meeting.

Why the planning Team Structure is important.

The Exercise Planning Team manages and is responsible for these aspect of an exercise:

  • Design
  • Development
  • Conduct
  • Evaluation

The size of the team will vary depending on the type and scale of the exercise. Exercise planners may elect to use the Incident Command System (ICS) structure when establishing the structure and organization of the planning team (see organization chart).

A team consists of a Lead Exercise Planner and planning team members. The team lead has complete management responsibility, assigning tasks to team members and ensuring the successful execution of the exercise.

The team size should:

  • Be kept manageable but represent a full range of participating organizations and other relevant stakeholders.
  • Include whole community stakeholders such as:
    • First responders/support agencies
    • Advocacy groups
    • Those with limited English language proficiency

In addition, Subject Matter Experts (SMEs) bring functional knowledge from their area of expertise.

The Incident Command System (ICS) structure includes the Exercise Planning Team Leader, or Command level, Operations, Planning, Logistics, and Administrative functional areas. The Safety Officer ensure safety procedures and safe practices are followed.
Planning Team flowchart. At the top is the Exercise Planning Team Leader, who is responsible for all planning team functions. The team is broken into sub-teams on which to focus. The purpose of these sub-teams is to divide the exercise functions and are all on the same level. They are Operations, which includes Site Liaison and Resources, Planning, which includes Exercise Documentation and Evaluation, Logistics, which includes Props and Actors, and Admin/Finance, which includes Reporting and Budgeting. A safety officer or team is always needed for operations-based exercises and falls under the purview of the Exercise Planning Team Leader as well.

Subject Matter Experts (SMEs) and Trusted Agents

SMEs contribute to the exercise planning team by:

  • Adding expertise to the exercise planning team
  • Providing functional knowledge for player-specific tasks evaluated through objectives
  • Helping make the scenario realistic and plausible
  • Ensuring appropriate evaluation of capabilities

Trusted agents:

  • Are individuals who understand that confidentiality must be preserved to maintain the integrity of the exercise.
  • All members of the exercise planning team are trusted agents.
  • Team members must not reveal details or share insight in order to protect the integrity and confidentiality of the exercise and evaluation process.
  • This is especially important to those team members who may also serve as controllers or evaluators during exercise conduct.

Successful Planning Teams and Planning Team Roles and Responsibilities

Successful Planning Teams

Regardless of the size or scope of an exercise, an exercise planning team performs best with a clear organizational structure which clearly defines roles, responsibilities, and functional requirements for each role or position on the planning team.

Tips for creating a successful planning team:

  • Define roles, responsibilities, and functional requirements
  • Engage elected and appointed officials and whole community leadership in exercise planning
  • Use project management principles
  • Follow standardized process
  • Organize the team using NIMS Incident Command System (ICS) or other structure that defines support roles for each team member

Planning Team Roles and Responsibilities

Exercise planning team members are responsible for the exercise design and development, conduct of the exercise, and exercise evaluation. Based on elected and appointed officials’ guidance and program priorities, the team will establish exercise objectives and determine the core capabilities to be assessed during exercise play. Next, the team is responsible for creating a realistic scenario to assess the identified capabilities and objectives. As mentioned earlier, SMEs play a big role in creating plausible scenarios. The team develops supporting documentation for the exercise as well as process documentation, control and simulation documentation, and customized evaluation packets for participants. Team members also help with building and distributing pre-exercise materials and conducting exercise planning meetings, briefings, and training sessions. Keep in mind that being part of an exercise planning team is usually a collateral duty. It is very important to ensure candidates can make the commitment to actively participate throughout the process.

Planning Team Roles and Responsibilities

Exercise planning team members are responsible for the exercise design and development, conduct of the exercise, and exercise evaluation. Based on elected and appointed officials’ guidance and program priorities, the team will establish exercise objectives and determine the core capabilities to be assessed during exercise play.

Next, the team is responsible for creating a realistic scenario to assess the identified capabilities and objectives. As mentioned earlier, SMEs play a big role in creating plausible scenarios. The team develops supporting documentation for the exercise as well as process documentation, control and simulation documentation, and customized evaluation packets for participants.

Team members also help with building and distributing pre-exercise materials and conducting exercise planning meetings, briefings, and training sessions.

Keep in mind that being part of an exercise planning team is usually a collateral duty. It is very important to ensure candidates can make the commitment to actively participate throughout the process.

Planning Activities (Meetings)

HSEEP doctrine defines meetings as smaller events focused on a specific topic. The exercise planning team members decide the type and number of planning activities needed to successfully plan a given exercise.

Meetings are held to discuss, review, or develop exercise content. Typically, these are face-to-face meetings, which are critical in the initial and final stages of exercise development. This forum allows for coordination and collaboration among participants. Face-to-face meetings help organizations share knowledge and build relationships.

While not all development can be completed during face-to-face meetings, they are a great medium to assess team progress, assign new responsibilities, review completed work, and establish deadlines. Once the team is fully established, online collaborative meetings may be used to facilitate the planning process.

Planning Meeting Types

Five different types of planning meetings may occur during the planning process.

  • Concept and Objectives Meeting (C&O)
  • Initial Planning Meeting (IPM)
  • Midterm Planning Meeting (MPM)
  • Master Scenario Events List Meeting (MSEL)
  • Final Planning Meeting (FPM)

The remainder of this lesson will discuss each meeting type in further detail.

Concept and Objectives (C&O) Meeting

The Concept and Objectives (C&O) meeting marks the formal beginning of the exercise planning process. Elected and appointed officials, representatives from all supporting organizations, and the exercise planning team leader attend the C&O meeting. Based on guidance from elected/appointed officials, exercise program priorities are defined and objectives are determined and aligned to core capabilities. In addition, the remainder of the exercise planning team members is identified.

Meeting Focus Discussion Points Exercise Tools Exercise Outcomes Follow-up
  • Formal beginning of the planning process
  • Identify the scope and objectives of the exercise
  • Exercise scope
  • Proposed exercise objectives and their aligned core capabilities
  • Proposed exercise location, date, and duration
  • Participants and anticipated extent of play for exercise participants
  • Exercise planning team
  • Exercise assumptions and artificialities
  • Exercise control and evaluation concepts
  • Exercise security organizations and structure
  • Available exercise resources
  • Exercise logistics
  • Exercise planning timeline and milestones
  • Local issues, concerns, and sensitivities
  • Meeting agenda
  • Background briefing
  • Exercise concept
  • Exercise timeline (group consensus)
  • Extent of participant play
  • Identification of planning team members
  • Planning timeline, milestones, and meeting dates

 

  • Meeting minutes compiled and sent to participants within four (4) days

Initial Planning Meeting (IPM)

The Initial Planning Meeting (IPM) marks the formal beginning of the exercise development phase. Its purpose is to determine exercise scope by getting intent and direction from elected and appointed officials, and gathering input from the exercise planning team; and to identify exercise design requirements and conditions (e.g., assumptions and artificialities), exercise objectives, participant extent of play, and scenario variables (e.g., time, location, hazard selection). The IPM is also used to develop exercise documentation by obtaining the planning team’s input on exercise location, schedule, duration, and other relevant details.

During the IPM, exercise planning team members are assigned responsibility for activities associated with designing and developing exercise documents, such as the Exercise Plan (ExPlan) and the Situation Manual (SitMan), and coordinating exercise logistics.

Meeting Focus Discussion Points Exercise Tools Exercise Outcomes Follow-up
  • Formal beginning of the development phase
  • Identify the scope and objectives of the exercise
  • Clearly defined exercise objectives and aligned core capabilities
  • Evaluation requirements, including EEG capability targets and critical tasks
  • Relevant plans, policies, and procedures to be evaluated in the exercise
  • Exercise scenario
  • Modeling and simulation planning
  • Extent of play for each participating organization
  • Optimum duration of the exercise
  • Exercise planners’ roles and responsibilities
  • Decision to record exercise proceedings (audio or video)
  • Local issues, concerns, or sensitivities
  • Any discussion points typically covered during a C&O Meeting if a C&O Meeting was not conducted
  • Consensus regarding the date, time, and location for the next meeting.
These items should be provided at least five (5) days prior to the scheduled meeting:

  • Read-ahead packet
  • The meeting agenda
  • Core capabilities
  • Hazard and threat information  (where applicable to the exercise)
  • For discussion-based exercises: the proposed room layout
  • For operations-based exercises: a map of the proposed exercise venue, including a description of the local environment
  • A copy of the proposed project timeline and milestones for exercise design and development
  • Copies of the presentation briefing to be used at the meeting
  • Any outcomes from the C&O meeting
  • Clearly defined exercise objectives and aligned core capabilities
  • Initial capability targets and critical tasks, which will be reviewed and confirmed prior to the next meeting
  • Identified exercise scenario variables (e.g., threat scenario, scope of hazard, venue, conditions)
  • A list of participating exercise organizations and anticipated organizational extent of play
  • Draft SitMan or ExPlan
  • Identification and availability of all source documents (e.g., policies, plans, procedures) needed to draft exercise documents and presentations
  • A refined exercise planning timeline with milestones
  • Identification and availability of SMEs, as necessary, for scenario vetting and/or expert evaluation
  • Determination of preferred communication methods among the exercise planning team
  • Clearly identified and assigned responsibility for exercise logistical issues
  • A list of tasks to be accomplished by the next planning meeting with established dates for completion and responsible planning team members identified
  • An agreed-upon date, time, and location for the next planning meeting and the actual exercise
  • Compile and distribute the IPM meeting minutes, including the next meeting date, time, and location
  • Between meetings: Planning Team collaborates on assignments and prepares draft of exercise documentation
  • Distribute draft documentation prior to the next scheduled meeting

Midterm Planning Meeting (MPM)

Midterm Planning Meetings (MPMs) provide additional opportunities to engage elected and appointed officials and to settle logistical and organizational issues that may arise during exercise planning. A walkthrough of the exercise site or venue will be conducted during this meeting.

If only three planning meetings are held (i.e., IPM, MPM, and Final Planning Meeting [FPM]), a portion of the MPM should be devoted to developing the Master Scenario Exercise List (MSEL).

Meeting Focus Discussion Points Exercise Tools Exercise Outcomes Follow-up
  • Re-engage elected and appointed officials (prior to the meeting)
    • This allows the planning team leader to keep officials up to date with progress to date, answer any questions they may have, and ensure alignment with guidance and intent
  • Exercise organization
  • Scenario and timeline development
  • Logistics and administrative requirements
  • Comments on draft exercise documentation
  • Construction of the scenario timeline—usually the MSEL—if an additional MSEL Planning Meeting will not be held
  • Identification of exercise venue artificialities and/or limitations
  • Agreement on final logistical items
  • Assignment of additional responsibilities
As with all meetings, a read-ahead packet should be delivered to all participants at least five (5) days prior to the MPM.

Tools included in the packet include, but are not limited to:

  • A meeting agenda
  • IPM meeting minutes
  • Draft scenario timeline
  • Draft documentation (ExPlan, Controller/Evaluator [C/E] Handbook)
  • Other selected documentation needed to illustrate exercise concepts and provide planning guidance

 

  • Reviewed or final exercise documentation (as applicable)
  • Draft Facilitator Guide or C/E Handbook, including the EEGs
  • Well-developed scenario to include injects (if no MSEL Planning Meeting is scheduled)
  • Agreement on the exercise site
  • Identified logistics planning requirements
  • Finalization of date, time, and location on the MSEL Planning Meeting and/or Final Planning Meeting
  • Compile and distribute the MPM meeting minutes, including the next meeting date, time, and location
  • Between meetings: Planning Team collaborates on assignments and prepares draft of exercise documentation
  • Distribute draft documentation prior to the next scheduled meeting

Master Scenario Events List (MSEL)

Complex exercises may require one or more additional planning meetings, known as Master Scenario Events List (MSEL) meetings. MSEL meetings are used to review and edit the scenario timeline. The MSEL is a chronological listing that supplements the exercise scenario with event synopses, expected participant responses, objectives and core capabilities to be addressed, responsible personnel, and specific injects and methods used to provide them.

Meeting Focus Discussion Points Exercise Tools Exercise Outcomes Follow-up
The MSEL is a chronological listing that supplements the exercise scenario with:

  • Event synopses
  • Expected participant responses
  • Objectives and core capabilities to be addressed
  • Responsible personnel, and specific injects and methods used to provide them
  • Tasks, conditions, and standards required to meet exercise objectives
  • Key events and critical tasks
  • Event originator, target player, expected player actions, and timeframe
  • Contingency injects to prompt player action (if needed)
As with all meetings, a read-ahead packet should be delivered to all participants at least five (5) days prior to the MSEL meeting.

  • Tools in the packet include an agenda, previous planning meeting minutes, draft exercise documents, applicable plans, policies, and procedures
  • A template used to create the MSEL should be included in the read-ahead packet
  • All MSELs should identify the exercise name, date, and location, as well as the controller name and notes. An example is provided

 

  • Key event injects and delivery timeline
    • Assignment of responsibility for conducting remaining events
    • Draft documentation revision
    • Venue selection agreement
    • Identified logistics planning requirements
  • Timeline for completion
  • Compile and distribute the MSEL meeting minutes, including the next meeting date, time, and location
  • Between meetings: Planning Team collaborates on assignments and prepares draft of exercise documentation
  • Distribute draft documentation prior to the next scheduled meeting

Final Planning Meeting (FPM)

The Final Planning Meeting (FPM) is the final forum for reviewing exercise processes and procedures. Both before and after the FPM, the exercise team leader should engage elected and appointed officials to ensure that the exercise is aligning with their intent, address any questions, and receive any last-minute guidance. At the time of the FPM, there should be no major changes in the design of the exercise, scope, or supporting documentation.

An FPM should be conducted for all exercises to ensure that all elements of the exercise are ready for conduct.

Meeting Focus Discussion Points Exercise Tools Exercise Outcomes Follow-up
An FPM should be conducted for all exercises to ensure that all elements of the exercise are ready for conduct. Prior to the FPM, the exercise planning team receives final drafts of all exercise materials. No major changes to the exercise’s design, scope, or supporting documentation should take place at or following the FPM. The FPM ensures that all logistical requirements have been met, outstanding issues have been identified and resolved, and exercise products are ready for printing.
  • Conduct a comprehensive final review
  • Approve all remaining draft documents (i.e., SitMan, MSEL, Controller/Evaluator [C/E] Handbook, Exercise Evaluation Guides [EEGs]) and presentation materials
  • Resolve any open planning issues and identify last-minute concerns
  • Review all exercise logistical activities (i.e., schedule, registration, attire, special needs)
As with all meetings, a read-ahead packet should be delivered to all participants at least five (5) days prior to the MSEL meeting.

  • Meeting agenda
  • Minutes from each of the previous meetings, IPM, MPM, and MSEL, if needed
  • All drafted exercise documents/documentation
  • Previously finalized documents

 

  • Final approval of exercise documents and materials for production
  • Identified issues resolved
  • Attendees understand and approve exercise processes and procedures
  • Task assignments and logistical elements, including facilities, equipment, and schedules are confirmed
The exercise planning team:

  • Finalizes of all publications
  • Prepares of all supporting materials
  • Rehearses presentations and briefings
  • Prepares to conduct the exercise

Prior to the exercise, documentation and any additional instructions should be disseminated to all appropriate personnel.

Lesson 4 Summary

In this lesson, you learned about the following meetings and when each is needed: Concept and Objectives (C&O) Meeting, Initial Planning Meeting (IPM), Midterm Planning Meeting (MPM), Master Scenario Events List (MSEL) Meeting, and the Final Planning Meeting (FPM).

Objectives: Having completed this lesson, you are able to:

  • Identify the roles and responsibilities of the exercise planning team.
  • Identify discussion points to be addressed during a concept and objectives meeting.
  • Identify the primary focus, discussion points, tools, and outcomes of the three planning meetings.
  • List the major components of a master scenario events list meeting.
The next lesson presents information on the Design and Development Phase.

Lesson 5 Overview

This lesson provides an understanding of exercise design, developing an exercise scope and priorities, and an explanation of the documentation required for each type of exercise.

Objectives: After completing this lesson, you will be able to:

  • Identify the key elements used to determine the scope of the exercise design.
  • Define SMART objective.
  • Identify the components of exercise documentation.
  • Identify the documentation needed for an operations-based exercise.
  • Identify the logistical concerns when developing an exercise.

Exercise Design

Think of design as the framework of an exercise.

Exercise design includes:

  • Establish exercise scope
  • Set exercise objectives
  • Create exercise scenario
  • Develop exercise documentation
  • Determine media and public relations guidance

Defining the Scope of the Exercise

Defining the scope of an exercise enables planners to meet objectives while staying within the resource and personnel constraints of the exercising organization(s). By definition, the scope is an indicator of the extent of an exercise.

Key elements of Scope:

  • Type of exercise
  • Participation level
  • Duration
  • Location
  • Parameters

Some elements are discussed and/or determined through program management activities or grant requirements. However, the responsibility of the planning team is to finalize the scope based on exercise objectives.

What is an Exercise Objective?

Objectives are the distinct outcomes an organization wishes to achieve during an individual exercise. An objective is a description of the performance expected from participants. It conveys specifically how the exercise should achieve its purpose.

Objectives:

  • Driven by Exercise Program priorities
  • Establish the cornerstone of scenario design, development, exercise conduct, and evaluation
  • Follow SMART guidelines for development

Generally, the number of exercise objectives should be limited to enable timely execution, facilitate design of a reasonable scenario, and promote successful completion of the exercise purpose.

Aligning objectives to capabilities ensures:

  • Methodical tracking of progress throughout the exercise programs and/or cycles
  • Standardized collection of exercise data used to inform preparedness assessments
  • Fulfillment of grant or funding-specific reporting requirements

What are SMART Objectives?

SMART is an acronym used to identify the characteristics of good objectives. SMART objectives identify who should do what, under what conditions, according to which standards.

SMART objectives are specific, measurable, achievable, relevant, and time-bound. Use the following guide to develop good exercise objectives.

SMART Guidelines for Exercise Objectives
Specific Objectives should address the five Ws: who, what, when, where, and why. The objective specifies what needs to be done with a timeline for completion.
Measurable Objectives should include numeric or descriptive measures that define quantity, quality, cost, etc. Their focus should be on observable actions and outcomes.
Achievable Objectives should be within the control, influence, and resources of exercise play and participant actions.
Relevant Objectives should be instrumental to the mission of the organization and link to its goals or strategic intent.
Time-Bound A specified and reasonable timeframe should be incorporated into all objectives.

The Exercise Scenario

A scenario is an outline or model for a simulated sequence of events that take place during an exercise or discussion. Scenarios are usually written in a narrative format but also depicted in timeline events.

Three basic elements:

  • Context or comprehensive story of an incident event (narrative)
  • Conditions allowing players to exhibit proficiency in demonstrating core capabilities and satisfying objectives
  • Technical details needed to depict the conditions and events of the scenario (timelines, etc.)

Based on:

  • Realistic events
  • Plausible threat
  • Challenging, but not too complicated that it overwhelms players

Exercise Documentation

Comprehensive, organized exercise documentation is critical to ensure an accurate account of the exercise. Security and accessibility is important when developing documentation.

Exercise documentation ensures that an accurate account of the exercise is preserved, which allows organizations to leverage past documentation to support future exercises. Consideration should be given to the accessibility of presentations and documents, such as making information available in alternative formats (e.g., large print, compact disc, Braille, and closed captioning).

This table lists the key exercise design and development documents identified by exercise type and distribution audience.

Document Title Exercise Type Distribution Audience
Situation Manual (SitMan) Seminar (Optional), Workshop (Optional), TTX, Game All Participants
Facilitator’s Guide Seminar (Optional), Workshop (Optional), TTX, Game Facilitators
Multimedia Presentation Seminar (Optional), Workshop (Optional), TTX, Game All Participants
Exercise Plan (ExPlan) Drill, FE, FSE Players, Observers
Controller and Evaluator (C/E) Handbook Drill, FE, FSE Controllers, Evaluators
Master Scenario Events List (MSEL) Drill, FE, FSE, Complex TTX (Optional), Game (Optional) Controllers, Evaluators, Simulators
Extent of Play Agreement (XPA) FE, FSE Exercise Planning Team
Exercise Evaluation Guides (EEGs) TTX, Game, Drill, FE, FSE Evaluators
Participant Feedback Form All Participants All Participants

Discussion-based Exercise Documentation

Situation Manual (SitMan) – The primary reference material provided to all participants. It is a textual background for the facilitated exercise and discussion. The SitMan structure includes an overview (scope, objectives, core capabilities, rules, and exercise agenda), the scenario, broken into modules, and discussion questions at the end of each module.

Facilitator Guide – The purpose of the facilitator guide is to assist facilitators managing discussion-based exercises. It outlines facilitator instructions and key issues for discussion during the exercise. The facilitator guide also includes background information to prepare the facilitator to answer questions.

Multimedia Presentation – A multimedia presentation is frequently used to illustrate the scenario. It helps participants focus and makes the exercise more realistic. The presentation is shown at the start of the exercise (StartEx) and supports the SitMan. Presentation components include an introduction; exercise scope, objectives, and core capabilities; exercise play rules and administrative information and is segmented into modules that correlate with the SitMan.

Operations-based Exercise Documentation

Exercise Plan (ExPlan) – A summary of the planned exercise, including the scope, objectives, and core capabilities to be validated. The plan is published and distributed to all participating organizations. Exercise players and observers should review all elements of the ExPlan prior to exercise participation. Components of the ExPlan: Exercise scope, objectives, and core capabilities; participant roles and responsibilities; rules of conduct; safety and security policies; logistics; communication methods; maps and directions; and duration, date, and time of exercise and schedule of events.

Controller/Evaluator Handbook (C/E Handbook) – The C/E Handbook contains detailed scenario and logistics information and a communications plan which is meant for designated controllers and evaluators only.

  • The controller portion of the C/E handbook, also known as the Control Staff Instructions (COSIN), provides guidelines for control and simulation support and establishes a management structure for the activities.
  • The evaluator portion of the C/E handbook, sometimes known as the EvalPlan, provides evaluation staff with guidance and instruction on evaluation or observation methodology, as well as essential materials required to perform their specific functions.

Master Scenario Events List (MSEL) – A chronological listing of events, in spreadsheet format, that drive exercise play. The MSEL is used during operations-based exercises or complex discussion-based exercises. MSEL events include contextual injects, expected action events (milestones), and contingency injects.

Anatomy of an Inject

Injects are used to drive exercise play. MSEL injects serve three primary functions: linking simulation to action, enhancing exercise experiences for players, and reflecting an incident or activity that will prompt players to implement the policy or procedure currently being evaluated.

Components that build an inject:

  • Designated scenario time and actual execution time
  • Who delivers the inject (and by what means) and who is designated to receive the inject
  • Event description (what will happen)
  • Inject content (quoted text as it is to be delivered) plus any specific direction needed
  • Expected action (Based on the objective being tested, identify specific tasks, conditions, and standards to accomplish the objective.
  • Note section for controllers/simulators to record pertinent information

Considerations When Developing the MSEL

Considerations when developing the MSEL:

  • Review capabilities
  • Identify chronology of key events
  • Anticipate player actions
  • Identify information resources
  • Compile all MSEL events into a single list
  • Refine selected MSEL events; create a detailed long version

Once the MSEL is drafted, the exercise planning team should review it, arrange entries in sequential order, and resolve any scheduling conflicts between events for best results during the exercise.

 

Tip
    • Exercise type:
      Operations-based
      Complex discussion-based

      • Drill
      • Functional Exercise (FE)
      • Full-Scale Exercise (FSE)
      • Complex Tabletop Exercise (TTX) (optional)
      • Game (optional)
    • Audience: Controllers, evaluators, and simulators
  • Typical components and example

Tip
  • Exercise type:
    Discussion-based

    • Seminar (optional)
    • Workshop (optional)
    • Tabletop Exercise (TTX)
    • Game
  • Audience: Facilitators

Tip
    • Exercise type:
      Operations-based

      • Drill
      • Functional Exercise (FE)
      • Full-Scale Exercise (FSE)
    • Audience: Players and observers
  • Typical components

Tip
  • Exercise type:
    Discussion-based

    • Seminar (optional)
    • Workshop (optional)
    • Tabletop Exercise (TTX)
    • Game
  • Audience: All Participants

Tip
    • Exercise type:
      Operations-based

      • Drill
      • Functional Exercise (FE)
      • Full-Scale Exercise (FSE)
    • Audience: Controllers and evaluators
  • Typical components

Additional Exercise Documentation

Additional exercise documents include:

Extent of Play Agreements (XPAs) – XPAs are agreements formed between the exercise sponsor and participating organizations. Specific details of participant involvement are identified, such as, one fire company, with two ambulances for 5 hours. These agreements can be vital to exercise planning, evaluator recruitment, and development of support requirements.

Exercise Evaluation Guides (EEGs)  Exercise Evaluation Guides (EEGs) provide evaluators with a checklist of critical tasks to be completed by participants during an exercise. EEGs contain the information to be discussed by participants, space to record evaluator observations, and questions to consider after the exercise.

Participant feedback form – The participant feedback form should be given to participants during exercise wrap-up. At a minimum, ask participants for feedback on the strengths and opportunities for improvement based on the organization’s plans, policies, SOPs and their impressions on the exercise conduct and logistics.

Waiver forms  All actors should receive a waiver form prior to the exercise. Once signed, these forms waive liability for all exercise planners and participants. Volunteers under the age of 18 must be signed by parental or legal guardian.

Weapons and Safety Policy  All exercises, where applicable, should employ a written weapon and safety policy that is in accordance with all applicable State and/or local laws and regulations. Exercise sponsors should coordinate the application of this policy with the appropriate safety and/or legal department, as necessary.

Media and Public Affairs Guidelines

It is important to integrate media-related concerns into exercise planning sessions. Media sources can be leveraged to prepare the public before, during, and after an exercise.

  • Inform the public of impending community preparedness activities
  • Disseminate additional instruction/information to the public community, as needed
  • Report on exercise results and community state of preparedness following the exercise

Exercise Development

Exercise development encompasses planning for the three critical elements involved in exercise conduct. These elements include:

  • Logistics
  • Control Structure
  • Communications
  • Safety and Security

Exercise Development – Logistics

Planning for logistics is crucial to the success of the exercise. Well thought out logistical details are the difference between safe and smooth exercise and one that is confusing and unsafe.

Considerations should include:

  • Physical Environment: venue, facilities/rooms, supplies, food, refreshments, IDs.
  • Actors (Volunteers) are used to make exercises more realistic as players administer simulated victim care.
    • Prior to StartEx, actors should be provided instruction and logistical details, as well as symptomatology cards containing their signs and symptoms.
  • Parking, transportation and designated areas are to be clearly identified and staffed to direct vehicles.

Exercise Development – Control Structure

The framework that allows for coordination and communication between controllers at multiple sites is known as the Control Structure. In a discussion-based exercise, the control structure is usually minimal. Operations-based exercises, on the other hand, may require an extensive structure to support proper coordination between the sites, venues, and control cells.

Exercises that span multiple sites and/or venues can improve communication and coordination by establishing a master control cell to oversee and coordinate venue control cells, as identified in the graphic.

A SimCell is used to generate injects, receive player responses, and disseminate information in place of an organization that may not participate in the exercise but would be on the scene of the incident, if it were real.

Exercise Development – Communications Plan

The communications plan is designed to keep all participants, control cells, and sites on the same page. The communications section in the C/E handbook (or COSIN) serves as the communication plan explaining how, what, and with whom controllers should communicate. Components that may be included:

  • Controller communications
  • Timing and content of communications
  • Communications methodology

Exercise Development – Safety and Security

Safety and Security are important considerations, especially in exercises that involve dangerous play or classified operations. The controller’s role is critical to safeguarding the integrity and security of the exercise scene.

Safety is paramount when planning any exercise. For operations-based exercises, consider appointing safety controller(s), outline safety requirements and policies, and plan for dedicated ambulance/EMT coverage for real-world emergencies.

Security of the exercise site should also be considered. Local law enforcement can be employed where appropriate. If classified information is involved, follow appropriate security protocols to ensure information is not compromised.

Lesson 5 Summary

In this lesson, you learned about SMART objectives and how to create and align them to capabilities, the documentation used in discussion- and operations-based exercises, and logistical and safety/security concerns to consider when developing exercises.

Objectives: Having completed this lesson, you are able to:

  • Identify the key elements used to determine the scope of the exercise design.
  • Define SMART objective.
  • List the three basic elements of the exercise scenario.
  • Identify the components of exercise documentation.
  • Identify the documentation needed for an operations-based exercise.
  • Identify the logistical concerns when developing an exercise.
The next lesson will present information on Conduct.

Lesson 6 Overview

This lesson provides an understanding of what takes place prior to, during, and after an exercise.

Objectives: After completing this lesson, you will be able to:

  • Identify the activities that take place prior to exercise conduct.
  • Explain the purpose of orientation briefings for the different participant groups.

Exercise Conduct

Discussion-based Exercises Operations-based Exercises
Preparation/Setup One day prior to conduct, the exercise planning team should tour the exercise venue to confirm room arrangement, test A/V equipment, and review any outstanding logistical concerns.

Prior to exercise conduct, the following items/equipment should be delivered, moved into place, tested, and made ready for use:

  • SitMans or other written materials for exercise participants
  • Multimedia presentation
  • Appropriate A/V equipment including televisions, projectors, projection screens, microphones, and speakers
  • Table and name tents
  • Badges identifying the role of each exercise participant
  • Sign-in sheets
  • Participant Feedback Forms
Depending on exercise complexity, the exercise planning team should provide ample setup time.

Setup includes room arrangement, configuring and testing all A/V equipment, marking exercise areas and their perimeters, setting up all props/effects, and reviewing all areas from a safety perspective.

Prior to Conduct:

  • Print exercise written materials
  • Distribute Media and Public Service Announcements
  • Arrange briefing areas, configure and test A/V equipment
  • Perform a safety check walk throughout exercise premises
Conduct The four main areas of discussion-based exercises are:

Multimedia presentation – Used to convey opening remarks, introduce jurisdictional officials and organizations, and exercise facilitators, controllers and/or evaluators, as needed. The brief describes the scenario and relevant background information.

Facilitated discussion – The facilitator(s) keep participant discussions focused on the exercise objectives using:

  • Breakout groups (a facilitator is required for each group)
  • Contiguous group

Moderated discussion – Usually follows breakout sessions. Speakers for each group relays their group’s discussion with all participants. Speakers present key findings and concerns and unresolved issues or questions. Following all group discussions, the facilitator(s) open the floor to questions.

Exercise data collection – During discussion-based exercises, facilitators help collect useful data by keeping discussions focused on exercise objectives, core capabilities, capability targets, and critical tasks.

On the day of conduct, all planning team members should report several hours prior to StartEx to address any logistical or administrative concerns before players arrive. A final check of the communications system should be evaluated before the exercise begins, to include:

  • Mark exercise boundaries, post signage, and set up props
  • Establish a registration area
  • Perform a final communications check
  • Conduct participant briefings

Exercise Participants

Participants are selected based on exercise scope, type, objectives, and specifics of the scenario being played out. Discussion-based exercises usually requires senior-level decision making personnel. Alternatively, Operations-based exercises usually select players from junior and senior staff. Selections are based on rank and experience. Participant roles include:

  • Exercise Director
  • Evaluator(s)
  • Lead Evaluator
  • Facilitator(s)
  • Controller(s)
  • Simulators
  • Senior Controller
  • Safety Controllers
  • Exercise Assembly Area Controller
  • Players
  • Actors
  • Observers

 

Exercise Orientation Briefings are used to educate each participant group about their particular roles and responsibilities during exercise play. Participant groups should only attend briefings designed for their specific groups. These briefings are mandatory for all participants. Specialized briefings should be presented for Elected/Appointed Officials, Controller/Evaluators, Actors, Players, and Observers.

Elected/Appointed Officials should be reengaged before exercise conduct to verify that the exercise, as designed, meets leadership expectations and intent.

Controllers should receive an overview of the exercise, specifics on their role and assigned location, the schedule of events, and direction on MSEL injects they will be delivering during the exercise.

Evaluators should receive an overview of the evaluation plan, methodology, objectives, and evaluation materials. Evaluators should receive instruction on the use of exercise materials, including the Exercise Handbook and Exercise Evaluation Guide (EEG). Evaluators should understand the exercise methodology and objectives, and know the schedule or agenda.

Orientation Briefings (cont’d)

Controller/Evaluators are briefed on both roles with the understanding that consistency in the dual role will contribute to the accuracy of the evaluation process.

Controllers are assigned participant groups to oversee. These oversight roles include Actor Controller, Exercise Assembly Area Controller, and Observer/Media Controller. Shortly before StartEx, controllers conduct a briefing for their particular groups.

Actor Controllers deliver the actor briefing. The briefing gives actors an overview of the exercise and their roles as “victims”, adding realism to the exercise.

Exercise Assembly Area Controllers deliver the player briefing, which relays information on player roles and responsibilities, safety considerations, exercise parameters, and responses to questions and concerns from players. If security credentials are needed, proper usage should be covered in the briefing.

Observer/Media Controllers deliver the observer briefing, which provides a background on the exercise program, specifics for the exercise about to commence, observer restrictions and/or limitations, and the exercise schedule of events. This briefing is also attended by VIPs and media representatives.

Operations-based Conduct

An operations-based exercise is developed to be a realistic representation of the capabilities to be examined.

During conduct of operations-based exercises, the Senior Controller or Exercise Director role is usually filled by the exercise planning team leader. Controllers and evaluators report their findings and concerns to the senior controller. The senior controller is responsible for officially beginning (StartEx) and ending (EndEx) exercise play.

Operations-based exercises can be staged in an actual environment or simulated if the environment is too dangerous to stage. For example, responding to an infectious virus spillage would be simulated to protect all involved.

Operations-based Conduct (cont’d)

Exercise areas for operations-based exercises need to be clearly defined, with exercises taking place within the designated areas. For instance, Functional exercises take place in a command or control center. Any additional activity is simulated by staff in SimCells. Clearly marked area help to eliminate confusion between the exercise and real-world operations.

Exercise data collection is recorded in Exercise Evaluation Guides (EEGs) by all evaluators. Positioning evaluators strategically to observe play will accommodate better data collection.

Lastly, an effective contingency plan is crucial to assure the exercise can be stopped, rescheduled, or canceled if a real-world event were to take place. All participants and stakeholders should be briefed on this plan.

Control

Operations-based control structure identifies how controllers communicate and coordinate within themselves and how they track exercise information. Along with roles and responsibilities, communication procedures are contained in the C/E Handbook.

Control is guidance for:

  • Communication and coordination
  • Roles and responsibilities
  • Safe and effective play
  • Simulation

During functional exercises (FEs), SimCell control is especially important. FEs use a lot of simulated activity, Thus, requiring a robust, detailed MSEL and close communication between site controllers and SimCells.

During full-scale exercises (FSEs) and drills, the exercise assembly area controller plays an integral role. Thus, the exercise assembly area controller must be in close communication with controllers and personnel to ensure safe and realistic deployment of resources.

Exercise Wrap-Up

Complete and thorough wrap-up ensures that relevant data is collected so effective evaluation and improvement planning can occur.

Debriefings immediately follow exercise conduct. They are short meetings conducted with the exercise planning team to discuss levels of satisfaction with the exercise, air any issues or concerns, and propose improvements. Debriefing notes, attendance lists, and participant feedback forms should be included in the meeting outcome.

The Player Hot Wash provides an opportunity for players to discuss strengths and opportunities for improvement. Hot washes are facilitator-led. They are usually brief and constructive. Comments can be used to develop the After Action Report/Improvement Plan (AAR/IP).

The Controller/Evaluator Debriefing is a forum for controllers and evaluators to review the exercise. The exercise planning team leader facilitates this meeting. Captured discussion comments an lessons learned are captured and incorporated into the AAR/IP.

Lesson 6 Summary

In this lesson, you learned about exercise setup, conduct, and wrap-up for discussion-based and operations-based exercises and the tasks specific to each exercise type. Discussion-based exercise Day of Conduct briefings are geared to all participants taking part in the exercise. Operations-based exercises are more complex and have individualized briefings based on the role of participants. Good communication and coordination are essential in operations-based exercise to keep all participants and venue sites on the same schedule.

Objectives: Having completed this lesson, you are able to:

  • Identify the activities that take place prior to discussion-based exercise conduct.
  • Explain the purpose of orientation briefings for the different participant groups.
The next lesson presents information on Evaluation.

Lesson 7 Overview

This lesson provides an understanding of the evaluation process and evaluation team.

Objectives: After completing this lesson, you will be able to:

  • Identify the information contained in an Exercise Evaluation Guide (EEG).
  • Identify the information contained in an exercise evaluation plan.
  • Identify how exercise data is collected and analyzed.
  • Identify the information contained in an After-Action Report (AAR).

Evaluation Planning

Evaluation is the link between exercise and improvement planning. Through evaluation, capabilities necessary to accomplish a mission or function or meet an objective are assessed, based on performance.

Effective exercise evaluation involves:

  • Planning for exercise evaluation
  • Observing the exercise and collecting exercise data during conduct
  • Analyzing collected data to identify strengths and areas for improvement
  • Reporting exercise outcomes in a draft After-Action Report (AAR)

 

Evaluation Team Responsibilities

In the early stages of the exercise planning process, the exercise planning team leader should appoint a lead evaluator to manage the evaluation process. The lead evaluator participates as a member of the planning team.

Together, the planning team and lead evaluator should determine which tools and documentation are needed to support the evaluation team. Data collection methods are determined by the lead evaluator.

Lead Evaluator
An effective lead evaluator will present the following skills:

  • Management skills
  • Knowledge and analytical skills
  • Effective communication skills
  • Mission area/core capability familiarity
Responsibilities:

  • Oversees evaluation process and planning
  • Evaluation Plan development using:
    • Exercise-specific information
    • Plans, policies, and procedures
    • Determining evaluator assignments
    • Developing instructions and Evaluation tools (EEGs)

Evaluation Team Responsibilities

Evaluation Team Members should:

  • Be familiar with the mission areas, core capabilities, plans, policies, and procedures to be examined during the exercise
  • Determine the structure of the evaluation team
  • Determine the supporting tools and documentation
  • Conduct a pre-exercise C/E Briefing
  • Recruit, train, and assign additional evaluators

Evaluation Plan Development Tasks

The Evaluation Plan (EvalPlan), developed under the direction of the lead evaluator, is the comprehensive plan exercise evaluators will use to observe, collect data, and evaluate player performance. The EvalPlan is the Evaluator portion of the Controller/Evaluator (C/E) Handbook.

Evaluation Plan development tasks:

  • Define evaluation requirements
  • Prepare a plan for evaluating the exercise
  • Select or develop evaluation forms
  • Finalize the evaluation plan

The Evaluation Plan (EvalPlan)

The evaluation plan shall include:
Exercise-specific information

  • Scenario or summary of the scenario
  • Functional groups for the exercise
  • Schedule of events (and evaluation schedule)

 Plans, policies, procedures, and agreements

  • Jurisdiction’s applicable plans, policies, procedures, and agreements that one would expect to discuss during discussion-based exercises and implemented/utilized during an operations-based exercise

Evaluation tools

  • Data collection instruments
  • Jurisdiction-specific Exercise Evaluation Guides (EEGs)
Evaluator requirements and assignment

  • Number of evaluators needed
  • Subject matter expertise or background required
  • Functional group or discipline that each evaluator will observe

 Evaluator instructions

  • What evaluators should do prior to arrival
    • Review exercise material
    • Review jurisdictional plans and procedures
    • Review the evaluation plan and process
  • Roles and responsibilities during conduct
  • Required deliverables

Exercise Evaluation Guide (EEG) Development

Exercise Evaluation Guides (EEGs) provide a consistent tool to guide exercise observation and data collection. Aligned to exercise objectives and core capabilities, EEGs list capability targets and critical tasks.

EEGs accomplish several goals:

  • Streamline data collection
  • Enable thorough assessments of the participant organizations’ capability targets
  • Support development of the After-Action Report (AAR)
  • Provide a consistent process for assessing preparedness through exercises
  • Help organizations map exercise results to exercise objectives, core capabilities, capability targets, and critical tasks for further analysis an assessment

EEG Format

EEGs are designed to document who, what, when, where, and how tasks were completed.

They document the following evaluation requirements for exercise evaluators:

  • Core capabilities: Critical elements needed to achieve a specific mission area
  • Capability target(s): Performance thresholds for each core capability. Typically written as quantitative or qualitative statements.
  • Critical Tasks: Elements required to perform a core capability and describe how the target will be met. Generally, they include activities, resources, and responsibilities required to fulfill capability targets.
  • Performance ratings: Description of performance against target levels. Includes target ratings and core capability ratings.

Pre-exercise Evaluator Briefing

Before StartEx, the lead evaluator should meet with all evaluators to confirm roles, responsibilities, and assignments.

Evaluation Documentation and Tools include:

  • Evaluation Plan
    • SitMan or scenario
    • C/E Handbook
  • Evaluator Team Organization
    • Assignments
    • Locations
  • Evaluation Instructions
    • Instruction on use of tools, logs, and forms
  • Evaluation Tools
    • Jurisdiction-specific or organization-specific SOPs
    • EEGs
    • MSEL

Evaluation Documentation and Tools

The C/E handbook or Evaluation Plan is the primary evaluation documentation for the exercise and typically contains the following information:

  • Exercise-specific details: Exercise scenario, schedule of events, and evaluation schedule
  • Evaluator team organization, assignments, and locations: A list of evaluator locations, shift assignments, a map of the exercise site(s), evaluation team organizational chart, and team contact information
  • Evaluator instructions: Step-by-step instructions for evaluators for activities before, during, and after the exercise
  • Evaluation tools: EEGs, MSEL, or a list of venue-specific injects, electronic or manual evaluation logs or data collection forms, relevant plans and procedures, Participant Feedback Forms, and Hot Wash Templates

Observation and Data Collection

Observation and data collection can differ between discussion-based an operations-based exercises. Data collection forms the analytic basis for determining if critical tasks were successfully demonstrated and capability targets were met.

Discussion-based exercises focus on issues involving plans, policies, and procedures. Observations may consist of an evaluator recording data from participant discussions on the EEGs.

Operations-based exercises focus on issues affecting the operational execution of critical tasks and capabilities. Observations are recorded from participant actions. These observations form the basis used to determine if critical tasks were successfully demonstrated and capability targets met.

Observation and Data Collection (cont’d)

Observation Data Collection
Evaluators should observe the following topics related to execution of capabilities and tasks examined during the exercise:

  • Utilization of plans, policies, and procedures related to capabilities
  • Implementation of legal authorities
  • Understanding and assignment of roles and responsibilities of participating organization and players
  • Decision-making processes used
  • Activation and implementation of processes and procedures
  • How and what information is shared among participating agencies/organizations and the public
Data collected by evaluators is compiled to create the after-action report. On an as needed basis, the lead evaluator may assign evaluators to collect supplemental data during an exercise.

Capturing the following data support AAR development:

  • Decisions and recommendations
  • Roles and responsibilities
  • Coordination and cooperation
  • Supplemental data/written records

NOTE: Evaluators should NOT be a distraction or interfere with exercise play.

EEG Observations

Observations come from many different sources. All attempts should be made by the evaluator to collect as much information as possible.

Some additional sources include:

  • Event logs
  • Video or audio recordings
  • Evaluator notes
  • Photographs

Recording Observations

Observation notes include if and how quantitative or qualitative targets were met. For example, a capability target might state, “Within 4 hours of the incident…” Observation notes on that target should include the actual time required for exercise players to complete the critical task(s).

Additionally, observations should include:

  • Actual time required for exercise players to complete the critical task(s)
  • How target was or was not met
  • Decisions made and information gathered to make decision
  • Requests made and how requests were handled
  • Resources utilized
  • Plans, policies, procedures, or legislative authorities used or implemented
  • Any other factors contributed to the outcomes.

Based on their observations, evaluators assign a target rating for each capability target listed on the EEG. Evaluators then consider all target ratings for the core capability and assign an overall core capability rating.

The rating scale includes four ratings:

  • Performed without Challenge (P)
  • Performed with Some Challenges (S)
  • Performed with Major Challenges (M)
  • Unable to be performed (U)

Root-Cause Analysis

After evaluators identify discrepancies between what happened and what was supposed to happen (the issues), they explore the source of these issues. This step is called root-cause analysis. When conducting a root-cause analysis, evaluators ask why each thing happened or did not happen.

Root-cause analysis may also require the review and evaluation of an organization’s plans, policies, and procedures. When completing the analysis, evaluators should consider the following questions:  Root Cause Analysis

Ask this First… Then Ask…
Were the objectives for each critical task met? If not, what factors contributed to this result?
Did exercise discussions or activities suggest the critical tasks performed sufficiently to meet the capability targets? If not, what were the resulting impacts or consequences?
What improvements are required? Are other resources needed? If so, how will they be obtained?
What strengths are required to carry out those tasks? What decisions would need to be made, and who would make them?
Do current plans, policies, and procedures support the performance of the critical tasks? Are participants familiar with these documents?
Are personnel trained to perform the critical tasks? If not, what personnel may require additional training?
Do personnel from multiple agencies or jurisdictions need to work together to perform the tasks? If so, are agreements or relationships in place to support the performance of the tasks?
What should be learned from this exercise?
What improvements are recommended? Who (position or agency) is responsible for implementing the improvements?  What is the expected timeframe for completion of the improvement?

After-Action Report (AAR)

What is an AAR?

  • The document that summarizes key information related to the evaluation
  • Overview of performance related to each exercise objective and associated core capabilities
  • Length, format, and development timeframe of the AAR depend on the exercise type and scope

Elements of the AAR include:

  • Exercise overview
  • Analysis of core capabilities (The main focus of the AAR)
  • Appropriate appendices (Improvement plan and participant list)

AARs also include basic exercise information such as the name and type of the exercise, dates, location, participating organizations, mission areas, sponsor, and POC.

AAR Review Meeting

Exercised sponsor distributes the AAR draft to participating organizations and elected and appointed officials to:

  • Review and determine areas for improvement
  • Determine organization with responsibility for corrective actions

Lesson 7 Summary

In this lesson, you learned about the responsibilities of an evaluation team, the EEG format and how it is used to observe, collect data, and the role the data plays in root-cause analysis.

Objectives: Having completed this lesson, you are able to:

  • Identify the information contained in an Exercise Evaluation Guide (EEG).
  • Identify the information contained in an exercise evaluation plan.
  • Identify how exercise data is collected and analyzed.
  • Identify the information contained in an After-Action Report (AAR).
The next lesson presents information on Improvement Planning.

Lesson 8 Overview

This lesson provides an overview on the Improvement Planning process.

Objectives: After completing this lesson, you will be able to:

  • Identify the process of improvement planning, including its inputs and outputs.

Corrective Actions

Corrective actions are concrete, actionable steps intended to resolve capability shortfalls identified in exercises or real-world events. Elected/appointed officials should review and revise the draft AAR, as needed, prior to the After-Action Meeting (AAM) to confirm the validity of evaluator-identified issues.

Corrective actions may include:

  • Changes to plans and procedures, organizational structures, and/or management processes
  • Additional training, equipment, or resources

Organizational reviewers determine which issues fall within their purview and take ownership of those issues for action. At this point, a draft improvement plan (IP) is developed.

After-Action Meeting

The AAM is a forum to review the AAR and the draft IP. Prior to the meeting, the exercise sponsor is responsible for distributing the revised AAR, which includes feedback on strengths and opportunities for improvement, as well as the draft IP.

Meeting Outcomes:

  • Final consensus on draft corrective actions
  • Develop deadlines for implementation of corrective actions
  • Identify corrective action owners and assignees

Finalizing the AAR/IP

Finalizing the After-Action Report/Improvement Plan (AAR/IP) – Corrective Action Tracking and Implementation

  • Distribute to exercise planners, participants, and other preparedness stakeholders, as appropriate
  • Track corrective actions to completion
  • Ensure a system is in place to validate previous corrective actions have been successfully implemented (Tracked continuously until implementation)

Using IPs to Support Continuous Improvement

Identifying strengths, areas of improvement, and corrective actions resulting from exercises help organizations build capabilities as part of a larger continuous improvement process.

The principles of continuous improvement are:

  • Important to the National Preparedness System
  • Consistent approach to strengthening Whole Community preparedness
  • Builds capabilities as part of a larger continuous improvement process
  • Proven method of issue resolution and information sharing
  • Applicable to all operational phases

Lesson 8 Summary

In this lesson, you learned about corrective actions, the after-action meeting, finalizing the AAR/IP, and how continuous improvement is supported.

Objectives: Having completed this lesson, you are able to:

  • Identify the process of improvement planning, including its inputs and outputs.

Course Summary

The Course Objectives include:
  • Develop a baseline knowledge of exercise fundamentals.
  • Identify the tasks necessary to complete each phase of the exercise process.
  • Define how exercises complete the preparedness process.
  • Identify the role of exercises in validating capabilities.
  • Identify phases of exercise evaluation and the improvement planning process.
The following topics are included in this course:

HSEEP Fundamental Principles. You learned the fundamental HSEEP principles of; guidance by elected and appointed officials, the National Preparedness Goal (NPG), the Progressive Planning Approach, the importance of the whole community integration, risk management and a common methodology for exercises.

The reasons for conducting exercises. You learned the reasons for conducting exercises which included- National Preparedness Components, clarifying responsibilities and roles, improving interagency coordination and communication, identifying gaps in resources, developing individual performance, and identifying areas/opportunities for improvement.

Identification of how the participant has shareholder support. You learned how participants have shareholder support by engaging the whole community, and state and local jurisdictions.

Discussion-based exercises. You learned about discussion-based exercises as- a forum for discussing or developing plans and procedures, less complicated than operations-based exercises, focused on strategy and policy.

Operations-based exercises. You learned about operations-based exercises and their characteristics which included: drills, the involvement of deployment of resources and personnel, complexity compared to discussion based exercises, and their advantages including the improvement of individual and team performances.