p. 1

[close]

p. 2

homeland security exercise and evaluation program volume iii exercise evaluation and improvement planning revised february 2007

[close]

p. 3

this page is intentionally blank.

[close]

p. 4

hseep volume iii preface homeland security exercise and evaluation program hseep volume i was initially published in 2002 and provided an overview of the exercise design development conduct evaluation and improvement planning process as well as doctrine for u.s department of homeland security dhs exercises subsequent volumes ii­iv provided more detailed descriptions of the planning and evaluation process as well as sample exercise materials since the initial versions of the hseep volumes were published the homeland security community has experienced numerous changes including the building of a new and cohesive federal agency and the release and adoption of the national response plan nrp national incident management system nims national preparedness goal universal task list utl and target capabilities list tcl this 2007 release of the hseep volumes represents an exercise policy and program reflective of these changes the following changes have been made · the volumes have been made more user-friendly and concise · new policies have been incorporated e.g nims nrp national preparedness goal utl tcl · references to dhs-specific doctrinal or grant-related requirements such as the need for terrorism-related scenarios have been eliminated · comments from the federal interagency as well as several state and local stakeholders have been incorporated so the hseep policy and guidance is more applicable to all exercises regardless of scope scale scenario or sponsoring agency · the order of volumes ii and iii has been reversed to follow the natural progression of exercise design development conduct evaluation and improvement planning it is important to note that the fundamentals of the exercise design development planning evaluation and improvement planning methodologies have not changed with these volume revisions developing and implementing comprehensive exercise policies is a continually evolving process as strategies policies and plans evolve future revisions will be issued preface i

[close]

p. 5

this page is intentionally blank.

[close]

p. 6

hseep volume iii contents preface i introduction v homeland security exercise and evaluation program v purpose v organization v security guidance vi chapter 1 evaluation and improvement planning overview 1 evaluation improvement planning and capabilities-based planning 1 exercise evaluation methodology 1 task-level analysis 1 activity-level analysis 2 capability-level analysis 2 exercise evaluation guides 3 after action report improvement plan 5 chapter 2 exercise evaluation data collection and analysis steps 1­4 6 step 1 plan and organize the evaluation 7 appoint lead evaluator 7 develop evaluation requirements 7 exercise scope 7 exercise objectives 7 exercise evaluation team organization and structure 8 define evaluation requirements 9 recruit assign and train evaluators 9 recruiting evaluators 9 assigning evaluators 9 training evaluators 10 finalize evaluation plan 10 conduct controller and evaluator briefing 11 step 2 observe the exercise and collect data 11 discussion-based exercises 11 operations-based exercises 12 conducting a player hot wash 14 collecting supplemental data 14 step 3 analyze data 14 identifying root cause and developing recommendations 15 discussion-based exercises 15 controller and evaluator debrief 15 draft aar content 15 operations-based exercises 16 controller and evaluator debrief 16 draft aar content 16 exercise event timeline 16 contents iii

[close]

p. 7

hseep volume iii step 4 develop the draft after action report improvement plan 17 chapter 3 improvement planning steps 5­8 18 step 5 conduct after action conference and refine draft aar 19 step 6 identify corrective actions to be implemented 20 step 7 finalize aar/ip 21 step 8 track implementation 21 event points of contact 21 participating entity points of contact 22 action officers 22 continual improvement 22 appendix a after action report improvement plan guidelines a-1 appendix b after action quick look report guidelines b-1 appendix c exercise evaluation guide sample c-1 appendix d development of discussion-based evaluation materials d-1 appendix e acronyms e-1 figures and tables figure 1-1 levels of criteria for analysis 3 figure 1-2 eeg relationships 4 figure 2-1 data collection and analysis steps 6 figure 2-2 example exercise evaluation team organization 8 figure 3-1 improvement planning steps 19 figure 3-2 flow of observations 23 table a-1 example ip a-7 table a-2 example exercise events summary table format a-9 table a-3 descriptions of performance ratings a-9 figure d-1 developing scenario-specific discussion-based materials d-2 contents iv

[close]

p. 8

hseep volume iii introduction homeland security exercise and evaluation program following the domestic terrorist attacks in 1993 1995 and 2001 and the establishment of the u.s department of homeland security dhs in 2002 officials at all levels of government and in all types of communities have worked to prepare for prevent respond to and recover from a variety of threats to public safety exercises play a crucial role in preparedness providing opportunities for emergency responders and officials to practice and assess their collective capabilities purpose the purpose of the homeland security exercise and evaluation program hseep is to provide common exercise policy and program guidance that constitutes a national standard for exercises hseep includes consistent terminology that can be used by all exercise planners regardless of the nature and composition of their sponsoring agency or organization the volumes also provide tools to help exercise managers plan conduct and evaluate exercises to improve overall preparedness hseep reflects lessons learned and best practices from existing exercise programs and can be adapted to the full spectrum of hazardous scenarios and incidents e.g natural disasters terrorism technological disasters the hseep reference volumes integrate language and concepts from the national response plan nrp the national incident management system nims the national preparedness goal the universal task list utl the target capabilities list tcl existing exercise programs and prevention and response protocols from all levels of government in accordance with nims all efforts should be made to ensure consistent use of the terminology and processes described in hseep organization this document is the third of five hseep volumes all of which are available at the hseep website http hseep.dhs.gov the volumes are organized as follows hseep volume i hseep overview and exercise program management provides guidance for building and maintaining an effective exercise program and summarizes the planning and evaluation process described in further detail in volumes ii through v hseep volume ii exercise planning and conduct helps planners outline a standardized foundation design development and conduct process adaptable to any type of exercise hseep volume iii exercise evaluation and improvement planning offers proven methodology for evaluating and documenting exercises and implementing an improvement plan ip hseep volume iv sample exercise documents and formats provides sample exercise materials referenced in hseep volumes i ii iii and v readers with internet connectivity may click on exercise materials referenced in this volume to link to hseep volume iv hseep volume v prevention exercises draft contains guidance consistent with the hseep model to assist entities in designing and evaluating exercises that validate pre incident capabilities such as intelligence analysis and information sharing this volume hseep volume iii exercise evaluation and improvement planning which provides guidance for exercise evaluation and improvement planning is organized as follows chapter 1 introduction evaluation and improvement planning overview v

[close]

p. 9

hseep volume iii chapter 2 chapter 3 exercise evaluation data collection and analysis steps 1­4 improvement planning steps 5­8 appendix a after action report improvement plan guidelines appendix b after action report quick look report guidelines appendix c exercise evaluation guides appendix d discussion-based eeg development guidance appendix e acronyms security guidance while most of the content found in hseep is not sensitive or classified some hseep materials e.g scenario examples particularly those in volume iv may necessitate restrictions on distribution exercise materials that are produced in accordance with hseep guidance and are deemed sensitive should be designated as for official use only fouo fouo identifies unclassified information of a sensitive nature not otherwise categorized by statute or regulations of which the unauthorized disclosure could adversely impact a person s privacy or welfare the conduct of federal programs or programs or operations essential to national interest examples of materials that may require fouo designation include scenario information the master scenario events list msel and the after action report improvement plan aar/ip access to fouo information is on a need-to-know basis fouo information may be shared with other agencies federal state local or tribal government appropriate private sector representatives and law enforcement officials provided a specific need-to-know has been established and the information is shared in furtherance of a coordinated and official governmental activity certain exercise-related information from private sector partners may require or be eligible for additional protections under the protective critical infrastructure information pcii program established pursuant to the critical infrastructure information cii act of 2002 the pcii program is an information-protection tool that enables members of the private sector to submit proprietary confidential or sensitive infrastructure information to dhs with the assurance that the information will be protected from public disclosure under the pcii program information that satisfies the requirements of the cii act of 2002 is protected from public disclosure under the freedom of information act foia state and local disclosure laws and use in civil litigation dhs and other federal state and local analysts use pcii in pursuit of a more secure homeland focusing primarily on analyzing and securing critical infrastructure and protected systems identifying vulnerabilities and developing risk assessments and enhancing recovery preparedness measures introduction vi

[close]

p. 10

this page is intentionally blank.

[close]

p. 11

hseep volume iii chapter 1 evaluation and improvement planning overview exercise evaluation maintains a fundamental link to improvement planning because it assesses an entity s performance in an exercise and identifies strengths and areas for improvement following exercise conduct improvement planning leverages the outputs of the evaluation process by developing improvement plans ips which assign responsibility for correcting deficiencies or shortcomings observed during a given exercise through this process evaluation identifies improvement opportunities and improvement planning provides a disciplined process for implementing corrective actions evaluation improvement planning and capabilities-based planning in accordance with homeland security presidential directive 8 hspd-8 the homeland security exercise and evaluation program hseep has adopted a capabilities-based planning approach which is a process intended to build capabilities suitable for responding to a wide range of threats and hazards capabilities-based planning emphasizes the need to analyze a diverse array of realistic scenarios and identify corresponding capabilities necessary for effective prevention protection response and recovery efforts capabilities-based planning is the basis for guidance such as the national preparedness goal the target capabilities list tcl and the universal task list utl the tcl and utl drive the application of capabilities-based planning by identifying 37 capabilities that will prepare the nation for terrorism natural disasters and other emergencies exercise evaluation and improvement planning play an important role in the capabilities-based planning process by assessing an entity s capabilities based on exercise objectives and developing ips that enhance those capabilities exercise evaluation guides eegs provide standards for assessing objectives through the execution of tasks and activities linked to each target capability based on areas for improvement identified using the eegs after action reports improvement plans aars/ips provide concrete steps that an entity can take to remedy deficiencies or shortcomings observed during exercises exercises are also an opportunity to identify lessons learned and best practices that can be shared with other jurisdictions and organizations to help build the nation s overall preparedness exercise evaluation methodology the hseep evaluation methodology is an analytical process used to assess the demonstration of capabilities during exercises according to this methodology exercise evaluation incorporates three distinct levels of analysis task-level analysis activity-level analysis and capability-level analysis task-level analysis tasks are specific discrete actions that individuals or groups must successfully perform or address during chapter 1 evaluation and improvement planning overview 1

[close]

p. 12

hseep volume iii operations-based and discussion-based exercises task-level analysis assists representatives of exercising entities in analyzing shortcomings or strengths related to these individual actions this analysis can also help entities target plans equipment and training resources to improve specific task performance each task is accompanied by performance measures designed to assist an exercise evaluator in assessing relevant individual or group performance pertaining to the task for example the wmd/hazmat response and decontamination capability eeg contains the task implement mass decon [decontamination operations which is accompanied by performance measure check boxes marked fully partially not and not applicable to designate the degree to which the task was demonstrated during the evaluation certain tasks may also be accompanied by a target timeframe for initiating these operations which in the case of the example decontamination task is less than 15 minutes after arrival these performance measures are designed to prompt evaluators to capture multiple aspects of individual or group performance related to each specific task activity-level analysis activities are groups of similar tasks that when carried out according to plans and procedures allow an entity to demonstrate an associated capability from the tcl/utl for example the task implement mass decon operations is part of the activity decontamination and clean-up/recovery operations other related decontamination tasks also fall under this same activity when conducting activity-level analysis exercise evaluators seek to determine whether all activities have been performed successfully and in accordance with plans policies procedures and agreements through this analysis exercise evaluators gain valuable insight into broad thematic successes or challenges in performing related tasks awareness of such themes is key to improving the performance of individual tasks and thus demonstrating the associated capability such analysis is also vital in assessing the effectiveness with which individuals worked together at the discipline or organizational level and how well team members communicated across organizational boundaries during an exercise capability-level analysis capabilities are combinations of elements e.g personnel planning organization and leadership equipment and systems training exercises assessments and corrective actions that provide the means to achieve a measurable outcome capability-level analysis involves assessing an entity s ability to demonstrate its priority capabilities necessary to successfully prevent protect against respond to or recover from the threat or hazard simulated in the exercise scenario when conducting capability-level analysis exercise evaluators examine whether an entity s performance of specific tasks and activities was sufficient to demonstrate the desired capability outcome for example an evaluator of the wmd/hazmat response and decontamination capability would evaluate how well exercise players identified and mitigated a hazmat release adequately performed rescue decontamination and treatment of exposed victims limited the impact of the release and effectively protected responders and at-risk populations capability-level analysis is designed to assist managers and executives in developing operating plans and budgets communicating with political officials setting long-range training and planning goals and developing interagency and/or inter-jurisdictional agreements figure 1-1 depicts the structure of these levels of analysis all capabilities link to the prevention protection response or recovery hseep mission areas chapter 1 evaluation and improvement planning overview 2

[close]

p. 13

hseep volume iii figure 1-1 levels of criteria for analysis exercise evaluation guides eegs assist exercise evaluators by providing them with consistent standards and guidelines for observation data collection analysis and report writing eegs have been developed for capabilities in the tcl and are linked to a capability s activities tasks and performance measures if necessary the eeg template format also allows the exercise planning team to add tasks specific to the exercising entity that are not found in the tcl/utl to the eegs for evaluation eegs accomplish several goals they streamline data collection enable a thorough assessment of the exercising entity s target capabilities and objectives support development of the aar/ip and provide a consistent and replicable process for assessing preparedness through exercises during the exercise planning stage the eegs assist the exercise planning team in developing objectives they are also used to map exercise results to exercise objectives and elements of the tcl/utl for further analysis and assessment figure 1-2 illustrates the scope of an eeg by showing the relationships between the capabilities activities tasks and performance measures chapter 1 evaluation and improvement planning overview 3

[close]

p. 14

hseep volume iii figure 1-2 eeg relationships evaluators use eegs before and during exercise observation because they provide evaluators with the activities tasks and performance measures associated with a target capability information in the eeg is sequenced according to the typical flow of activities and tasks to be accomplished for each capability the template is designed to allow evaluators to record the degree to which a prescribed task or performance measure was completed or met during the exercise however within this section of the eeg exercise evaluators do not rate the entity s performance because the eeg is neither a grading tool nor a scorecard rather evaluators are asked to objectively record the full partial or non-completion of each task the eeg is a reference for exercise evaluators giving a sense of when activities can be expected to occur and how those activities relate to capability completion in addition to information on the activities tasks and performance measures associated with a target capability the eeg includes analysis sheets which are designed for use after the exercise is complete these sheets are broken down as follows · an observations summary sheet allows exercise evaluators to record a general chronological narrative of exercise player actions based on the evaluator s observations on this sheet evaluators record when exercise events specific actions deserving special recognition particular challenges or concerns and areas needing improvement occurred the content recorded on this form will be used to develop the aar/ip · in the evaluator observations section evaluators record and analyze at least three observed strengths and three observed areas for improvement demonstrated by the entity during the exercise for each strength and area for improvement evaluators should record specific observations regarding what occurred a root cause analysis examining why events occurred and if necessary specific recommendations for corrective action the recommendations and observations that evaluators record in the evaluator observations section are used to develop the final observations and recommendations that are captured in the entity s aar/ip from the aar/ip s observations and recommendations proposed corrective actions are generated at the after action conference consistent eegs facilitate the creation of effective aar/ips by relating capabilities to activities tasks and performance measures eegs establish the foundation for ips that can strategically target personnel chapter 1 evaluation and improvement planning overview 4

[close]

p. 15

hseep volume iii planning organization and leadership equipment and systems training and exercises and assessments and corrective actions pertaining to identified shortcomings in priority capabilities see appendix c for an example of the wmd/hazmat response and decontamination eeg after action report improvement plan while the eegs are important observation tools and contribute to the improvement planning process by collecting initial observations and recommendations for improvement they are only a reference point from which to produce the main product of the evaluation and improvement planning process the aar/ip an aar captures observations of an exercise and makes recommendations for post-exercise improvements and an ip identifies specific corrective actions assigns these actions to responsible parties and establishes target dates for action completion because the aar and the ip are developed through different processes and perform distinct functions they are referred to separately at many points in this volume however in practice the aar and the ip should be printed and distributed jointly as a single aar/ip following an exercise chapter 1 evaluation and improvement planning overview 5

[close]