This is the accessible text file for GAO report number GAO-09-849 
entitled 'Homeland Defense: U.S. Northern Command Has a Strong Exercise 
Program, but Involvement of Interagency Partners and States Can Be 
Improved' which was released on September 9, 2009. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to Congressional Requesters: 

United States Government Accountability Office: 
GAO: 

September 2009: 

Homeland Defense: 

U.S. Northern Command Has a Strong Exercise Program, but Involvement of 
Interagency Partners and States Can Be Improved: 

GAO-09-849: 

GAO Highlights: 

Highlights of GAO-09-849, a report to Congressional Requesters. 

Why GAO Did This Study: 

U.S. Northern Command (NORTHCOM) exercises to test preparedness to 
perform its homeland defense and civil support missions. GAO was asked 
to assess the extent to which NORTHCOM is (1) consistent with 
Department of Defense (DOD) training and exercise requirements, (2) 
involving interagency partners and states in its exercises, (3) using 
lessons learned and corrective actions to improve preparedness, and (4) 
integrating its exercises with the National Exercise Program (NEP). To 
do this, GAO reviewed NORTHCOM and NEP guidance and postexercise 
documentation, assessed NORTHCOM compliance, and compared DOD and NEP 
exercise requirements. 

What GAO Found: 

NORTHCOM’s exercise program is generally consistent with the 
requirements of DOD’s Joint Training System, but its exercise reporting 
is inconsistent. Since the command was established in 2002, NORTHCOM 
has conducted 13 large-scale exercises and generally completed exercise 
summary reports within the required time frame. However, those reports 
did not consistently include certain information, such as areas needing 
improvement, because NORTHCOM lacks guidance that specifies exercise 
reports’ content and format, potentially impacting its ability to meet 
internal standards for planning and execution of joint exercises, and 
to compare and share exercise results over time with interagency 
partners and states. 

Nineteen federal agencies and organizations and 17 states and the 
District of Columbia have participated in one or more of the seven 
large-scale exercises that NORTHCOM has conducted since September 2005. 
However, NORTHCOM faces challenges in involving states in the planning, 
conduct, and assessment of its exercises, such as adapting its exercise 
system and practices to involve other federal, state, local, and tribal 
agencies that do not have the same practices or level of planning 
resources. Inconsistencies with how NORTHCOM involves states in 
exercises are occurring in part because NORTHCOM officials lack 
experience dealing with states and do not have a consistent process for 
including states in exercises. Without such a process, NORTHCOM 
increases the risk that its exercises will not provide benefits for all 
participants, impact the seamless exercise of all levels of government, 
and potentially affect NORTHCOM’s ability to provide civil support 
capabilities. 

NORTHCOM has a systematic lessons learned and corrective action program 
to improve preparedness, but gaps remain with collecting and sharing 
lessons with agency and state partners and managing corrective actions. 
Access to the system NORTHCOM uses for managing exercise observations 
is limited for non-DOD participants, and DOD believes that the 
Department of Homeland Security’s system is not adequately protected 
from unauthorized users. NORTHCOM’s mitigation steps have not resolved 
the issues. In addition, about 20 percent of the corrective actions 
tracked by NORTHCOM were being closed prematurely due to gaps in 
oversight. Closing issues prematurely increases the risk that issues 
will reoccur and limits the knowledge gained and value of the 
exercise.  

NORTHCOM has taken steps to integrate its exercises with the NEP, but 
guidance is not consistently applied. NORTHCOM has participated in 
several NEP exercises and is leading its first major NEP exercise in 
the fall of 2009. However, NORTHCOM has used DOD’s Joint Training 
System planning and documentation requirements rather than DHS’s 
requirements, because NEP guidance is not clear on what exercise 
planning standard should be used and DOD guidance does not address the 
issue. The states we visited use NEP guidance. Differences between NEP 
and DOD guidance could affect the ability of all participants to 
develop effective working relationships. 

What GAO Recommends: 

GAO is making recommendations to DOD to direct NORTHCOM to consistently 
involve the states in planning, executing, and assessing exercises and 
improve oversight of corrective actions. GAO is also recommending that 
DOD define when NORTHCOM should use NEP planning and documentation 
requirements. DOD agreed or partially agreed with the recommendations 
and cited ongoing and future efforts to satisfy the recommendations’ 
intent. DOD did not fully address a recommendation on training to 
NORTHCOM staff on specific state emergency management structures. GAO 
believes such training would benefit NORTHCOM personnel in advance of a 
crisis and for exercise planning. 

View [hyperlink, http://www.gao.gov/products/GAO-09-849] or key 
components. For more information, contact Davi M. D'Agostino, (202) 512-
5431, dagostinod@gao.gov. 

[End of section] 

Contents: 

Letter: 

Results in Brief: 

Background: 

NORTHCOM’s Exercise Program Is Consistent with DOD’s Joint Training 
System, but Exercise Reporting is Inconsistent: 

Interagency Partners and States Have Participated In NORTHCOM’s 
Exercises, but Challenges Remain with Including States in Planning, 
Conducting, and Assessing Exercises: 

NORTHCOM has a Systematic Lessons Learned and Corrective Action 
Program, but Gaps Remain: 

NORTHCOM Has Taken Steps to Integrate Its Exercises with the National 
Exercise Program, but Guidance Is Not Consistently Applied: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Scope and Methodology: 

Appendix II: Comparison of DOD’s Joint Training System with DHS’s 
Homeland Security Exercise and Evaluation Program: 

Appendix III: Comments from the Department of Defense: 

Appendix IV: GAO Contact and Staff Acknowledgments: 

Related GAO Products: 

Tables: 

Table 1: Major Documents Related to the National Exercise Program and 
NORTHCOM: 

Table 2: Large-Scale NORTHCOM Exercises Since 2002: 

Table 3: Interagency Partner and State Participation in Large-scale 
NORTHCOM Exercises Since 2006: 

Table 4: Emergency Management Structures of the States We Visited: 

Table 5: Number of Observations Input into Lessons Learned Management 
System for Large-Scale Exercises Since 2006: 

Table 6: NORTHCOM Participation in National Level and Tier II Exercises 
Since 2007: 

Table 7: Comparison of Areas of Application between DOD’s Joint 
Training System and DHS’s Homeland Security Exercise Evaluation 
Program: 

Table 8: Comparison of DOD Joint Training System and Homeland Security 
Exercise and Evaluation Program: 

Figures: 

Figure 1: NORTHCOM’s Area of Responsibility: 

Figure 2: Pathways for Requesting Military Assistance for Incident 
Response: 

Figure 3: DOD’s Joint Training System: 

Figure 4: National Exercise Program and U.S. Northern Command Exercise 
Program Time line: Major Events and Milestones (2001-2009): 

Figure 5: Description of NEP Tiers: 

Figure 6: Homeland Security Exercise and Evaluation Program Tool Kit: 

Figure 7: NORTHCOM Lessons Learned and Corrective Action Process: 

Abbreviations: 

DOD: Department of Defense: 

DHS: Department of Homeland Security: 

FEMA: Federal Emergency Management Agency: 

JLLIS: Joint Lessons Learned Information System: 

NEP: National Exercise Program: 

NORTHCOM: U.S. Northern Command: 

[End of section] 

United States Government Accountability Office: 
Washington, DC 20548: 

September 9, 2009: 

Congressional Requesters: 

The U.S. homeland continues to face complex and dynamic threats and 
vulnerabilities from terrorism and other catastrophic incidents, 
including natural disasters, industrial accidents, infrastructure 
failures, and infectious diseases.[Footnote 1] These incidents can 
produce devastating consequences as observed during the September 11, 
2001, terrorist attacks and in the aftermath of Hurricane Katrina, 
which required an effective and coordinated national effort, with 
shared goals and responsibilities for protecting and defending the 
homeland. The complexity of national-level coordination makes preparing 
for an incident all the more crucial.[Footnote 2] According the 
Quadrennial Defense Review, unified interagency efforts among the 
Department of Defense (DOD), Department of Homeland Security (DHS), and 
other federal, state, and local agencies are required to address 
threats to the homeland.[Footnote 3] Exercises play an instrumental 
role in preparing these agencies to respond to an incident by providing 
opportunities to test plans, improve proficiency, assess capabilities 
and readiness, and clarify roles and responsibilities.[Footnote 4] 
Short of performance in actual operations, exercises provide the best 
means to assess the effectiveness of organizations in achieving mission 
preparedness. Exercises also provide an ideal opportunity to enhance 
preparedness by collecting, developing, implementing, and disseminating 
lessons learned and verifying corrective action taken to resolve 
previously identified issues.[Footnote 5] 

Following the September 11 terrorist attacks, DOD realized the need for 
a more integrated civilian and military response capability for any 
future attack on the homeland. In response, DOD established the U.S. 
Northern Command (NORTHCOM) to provide and manage homeland defense and 
civil support. These two activities are among DOD's contribution to 
homeland security. DHS is the lead federal agency for homeland 
security, which is a national effort to prevent terrorist attacks 
within the United States, reduce America's vulnerability to terrorism, 
and minimize the damage and recover from attacks that do 
occur.[Footnote 6] DOD is the lead federal agency for homeland defense-
-defined as the protection of U.S. sovereignty, territory, domestic 
population, and critical defense infrastructure against external 
threats and aggression against the United States.[Footnote 7] Civil 
support is defense support of civil authorities--such as DHS or another 
federal agency or state--for domestic emergencies and for designated 
law enforcement and other activities.[Footnote 8] NORTHCOM differs from 
other combatant commands in that, in addition to Canada and Mexico, its 
area of responsibility includes the 48 contiguous states, Alaska, the 
District of Columbia, Puerto Rico, and the U.S. Virgin Islands. 

Numerous local, state, tribal, and federal agencies and organizations--
including DOD, DHS, and the Federal Emergency Management Agency (FEMA)--
have jurisdiction over or can coordinate resources within the homeland 
and, therefore, may be involved in the response to an incident. 
According to the National Defense Strategy, a whole-of-government 
approach is only possible when every government department and agency 
understands the core competencies, roles, missions, and capabilities of 
its partners and works together to achieve common goals.[Footnote 9] 
Ensuring an effective response to an incident will require that federal 
departments and agencies, states, and local governments conduct 
integrated disaster response planning and test these plans by 
exercising together.[Footnote 10] To achieve its goal of being ready to 
execute joint operations and ensure a seamless operating environment, 
NORTHCOM's training goals include maximizing exercise participation 
with federal, state, and local agencies and National Guard units. 
[Footnote 11] NORTHCOM uses long-established DOD practices for training 
and exercises to test and enhance the preparedness of its personnel and 
forces to perform its homeland defense and civil support missions. As a 
result, DOD has more experience than other federal agencies in terms of 
the scope and range of its exercises and the robustness of the process 
for conducting them. DOD's goal is to improve the homeland defense and 
consequence management capabilities of its interagency partners by 
leveraging this comparative advantage through shared training and 
exercises.[Footnote 12] 

The National Exercise Program (NEP) was established in April 2007 under 
the leadership of the Secretary of Homeland Security to prioritize and 
coordinate the various federal, regional, and state exercise activities 
and serves as the principal mechanism for examining the preparation and 
efficiency of the federal government to respond to an incident. 
[Footnote 13] The NEP includes a series of national exercises projected 
on a 5-year exercise schedule and has established a four-tier system to 
determine the relative priority of interagency participation in each 
exercise. DOD and other federal agencies, states, and local 
organizations participate in these national-level exercises. FEMA 
administers the NEP and has developed Homeland Security Exercise and 
Evaluation Program guidance to provide standardized policy, 
methodology, and terminology for exercise design, development, conduct, 
evaluation, and improvement planning. 

In our previous work following Hurricanes Katrina and Rita, we 
identified a need to improve, among other things, the nation's disaster 
response capabilities. Overall, capabilities are built upon the 
appropriate combination of people, skills, processes, and assets. 
Ensuring that needed capabilities are available requires effective 
planning and coordination in conjunction with training and exercises in 
which the capabilities are realistically tested and problems identified 
and subsequently addressed in partnership with other federal, state, 
and local stakeholders.[Footnote 14] Specific to DOD, we reported that 
prior to Hurricane Katrina, disaster plans and training exercises 
involving DOD were insufficient, and did not incorporate lessons 
learned from past catastrophes to fully delineate the military 
capabilities that could be needed to respond to a catastrophic natural 
disaster. Moreover, disaster plans had not been tested and refined with 
a robust exercise program. As a result of the inadequate plans--and the 
lack of realistic exercises to test those plans--a lack of 
understanding existed within DOD and among federal, state, and local 
responders as to the types of assistance and capabilities that DOD 
might provide, the timing of this assistance, and the respective 
contributions of the active-duty and National Guard components. We 
recommended that DOD establish milestones and expedite the development 
of detailed plans and exercises to fully account for the unique 
capabilities and support that the military is likely to provide to 
civil authorities in response to an incident.[Footnote 15] In addition, 
we recently found that although FEMA has developed plans, guidance, and 
systems to design exercises, gather lessons learned, and track 
corrective actions to implement the NEP, it faces challenges in 
ensuring that the exercises are carried out in accordance with Homeland 
Security Exercise and Evaluation Program guidance. We recommended, 
among other things, that FEMA establish a program management plan and 
better ensure that exercises follow Homeland Security Exercise 
Evaluation Program guidance, for example, by revising its grant 
monitoring guidance to include a checklist of specific requirements for 
states that receive grant funds.[Footnote 16] 

In order to better understand NORTHCOM's efforts to effectively prepare 
for its homeland defense and civil support missions, we were asked to 
review NORTHCOM's exercise program. This review is the final part of a 
broader congressional request to review NORTHCOM's efforts to plan and 
coordinate its homeland defense and civil support missions. We have 
issued two previous reports in support of that request.[Footnote 17] 
For this report, our objectives were to determine the extent to which 
(1) NORTHCOM's exercise program is consistent with DOD training and 
exercise requirements; (2) NORTHCOM involves relevant interagency 
partners and states in planning, conducting, and assessing exercises; 
(3) NORTHCOM is using lessons learned and corrective actions during 
exercises to improve mission preparedness; and (4) NORTHCOM is 
integrating its exercises with the NEP. 

In conducting this review, we generally focused our scope on NORTHCOM's 
large-scale exercises[Footnote 18] conducted since Hurricane Katrina 
made landfall in August 2005. To determine the extent to which 
NORTHCOM's exercise program is consistent with DOD exercise 
requirements and includes relevant exercise partners, we reviewed 
exercise documentation for all 13 large-scale exercises the command 
performed since it was established in 2002; evaluated NORTHCOM's 
compliance with requirements established in DOD guidance; and assessed 
the level of interagency and state participation in NORTHCOM's large- 
scale exercises. We also interviewed Office of the Secretary of 
Defense, Joint Staff, NORTHCOM, FEMA, FEMA regional offices, and state 
officials with knowledge of and experience with NORTHCOM's exercise 
program. We selected a nongeneralizable sampling of states based on the 
extent to which they have participated in major NORTHCOM exercises 
since Hurricane Katrina and the varying scenarios of the exercises. The 
states we selected played a major role in NORTHCOM exercises by having 
a portion of the exercise conducted in their state and having various 
state agencies and officials participate. States we selected include 
Arizona, California, Michigan, Oregon, Rhode Island, and Washington. We 
also met with Nevada officials who participated in a NORTHCOM exercise 
prior to Hurricane Katrina--Determined Promise 03--to provide context 
to the extent that changes may have been made to NORTHCOM's exercise 
program and help develop our state selection methodology. To determine 
the extent to which NORTHCOM is using lessons learned during exercises 
to improve mission preparedness, we reviewed DOD, NORTHCOM, and DHS's 
NEP guidance for lessons learned. Based on this guidance, we assessed 
the management of all unclassified exercise observations and issues 
identified from its last 6 large-scale exercises by determining each 
record's status (open or closed), type (issue or lesson learned), and 
disposition after NORTHCOM staff have acted on these records. Finally, 
to determine the extent to which NORTHCOM is integrating its exercises 
with the NEP we reviewed DOD and DHS guidance regarding the NEP and how 
DOD and its subordinate commands are required to or should participate. 
We conducted our review between June 2008 and September 2009 in 
accordance with generally accepted government auditing standards. Those 
standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe that 
the evidence obtained provides a reasonable basis for our findings and 
conclusions based on our audit objectives. 

Results in Brief: 

NORTHCOM's exercise program is generally consistent with the overall 
policies and practices of DOD's Joint Training System, but the way it 
reports on exercises is inconsistent. DOD and NORTHCOM guidance require 
that the command establish an exercise program consistent with the 
Joint Training System and complete postexercise documentation within an 
established time frame.[Footnote 19] NORTHCOM has developed a 
comprehensive exercise program consistent with DOD's Joint Training 
System, conducted 13 large-scale exercises since it was created in 
2002, and generally completed exercise summary reports within the 
required time frame. However, those reports did not consistently 
include certain information; for example, only 5 of the 11 exercise 
summary reports included an identified section on lessons learned. DHS 
currently has a template for exercise documentation that includes 
guidance on content and format; however, NORTHCOM does not follow this 
template and neither DOD nor NORTHCOM has similar guidance that 
specifies content and format. Without a consistent record of what 
occurred during an exercise, NORTHCOM cannot ensure that it has met 
internal standards for planning and execution of joint exercises, 
compare exercise results over time, and share lessons learned with 
interagency partners and states. We are recommending that NORTHCOM's 
Commander establish criteria for the format and content of postexercise 
documentation. 

Interagency partners and states have participated in NORTHCOM 
exercises, but NORTHCOM faces challenges involving interagency partners 
and states in planning, conducting, and assessing exercises. NORTHCOM 
guidance requires that NORTHCOM maximize exercise participation with 
interagency partners and states to achieve preparedness goals and 
ensure a seamless operating environment.[Footnote 20] Seventeen federal 
agencies and organizations and 17 states and the District of Columbia 
have participated in one or more of the seven large-scale exercises 
that NORTHCOM has conducted since September 2005. We found that the 
states we visited derived benefits from their participation in these 
exercises. For example, officials from 3 states told us this was their 
first opportunity to interact with federal military forces. However, we 
found that challenges remain which have resulted in inconsistencies in 
the way that NORTHCOM involves the states in its exercises. One of 
DOD's challenges is adapting its exercise system and practices to 
accommodate the coordination and involvement of other federal, state, 
local, and tribal agencies that do not have the same kinds of practices 
or level of planning effort. NORTHCOM also faces the challenge of 
balancing its training objectives with those of state agencies and 
organizations, particularly given the limited resources and funding 
states have available to exercise. While state and local governments 
seek to exercise their first responder capabilities before having their 
resources overwhelmed and needing to seek federal assistance, 
NORTHCOM's goal is to exercise its capability to provide support to 
civil authorities when local, state, and other federal resources are 
overwhelmed. As a result of this challenge, officials from 5 states 
told us that all of their needs were not fully met during the 
exercises, for example, due to large-scale, unrealistic scenarios that 
overwhelmed the states' resources before they had the opportunity to 
exercise their training objectives. Inconsistencies with how NORTHCOM 
involves states in planning, conducting, and assessing exercises are 
occurring in part because NORTHCOM officials lack experience in dealing 
with the differing emergency management structures, capabilities, and 
needs of the states. Inconsistencies are also occurring because 
NORTHCOM has not established a process for including states in 
exercises, such as consistent procedures for requesting state 
involvement in exercises through DHS/FEMA or the National Guard Bureau. 
Without an informed and consistent process, NORTHCOM increases the risk 
that its exercises will not provide benefits for all participants, 
impacting the seamless exercise of all levels of government and 
potentially affecting NORTHCOM's ability to provide support to civil 
authorities. We are recommending that (1) DOD work with DHS to 
establish guidance and procedures for requesting state participation 
and involving states in planning, executing, and assessing exercises 
and (2) NORTHCOM's Commander develop a training plan for NORTHCOM staff 
on state emergency management structures and relevant issues related to 
working with civilian state and local emergency management officials. 

NORTHCOM has a systematic lessons learned and corrective action program 
to improve preparedness, but gaps remain with collecting and sharing 
lessons with agency and state partners and managing corrective actions. 
DOD and NORTHCOM guidance requires that NORTHCOM identify lessons 
learned during the course of normal operations, exercises, and real- 
world events; share valid observations and findings as widely as 
possible; and track them until the corrective actions are verified 
during subsequent events or exercises.[Footnote 21] NORTHCOM uses DOD's 
Joint Lessons Learned Information System (JLLIS), an automated tool for 
managing and tracking observations, and a corrective action process to 
manage issues requiring action for resolution. Access to this system is 
limited for non-DOD participants, and DOD believes that DHS's system is 
not adequately protected from potential unauthorized users. NORTHCOM 
has taken steps to address these issues, such as placing a template on 
NORTHCOM's restricted access portal to provide a venue for non-DOD 
officials to record exercise observations, but these efforts have not 
been successful. Because NORTHCOM is not fully involving other federal 
agencies and states in its lessons learned process, it is missing 
opportunities to learn lessons from an exercise. For example, officials 
from two states did not provide NORTHCOM with observations from 
exercises because they did not attend the command's postexercise 
reviews for varying reasons, such as limited resources. As a result, 
NORTHCOM risks the reoccurrence of these issues. We also found that 
about 20 percent of the corrective actions tracked by NORTHCOM were 
being closed prematurely because, under existing procedures, NORTHCOM 
exercise directorate officials do not have oversight over issues that 
are resolved within other directorates or are not giving long-standing 
issues the sustained management attention to ensure resolution. A 
NORTHCOM official told us that they do not have the staff necessary to 
oversee the actions on records handled within the other directorates. 
Closing issues requiring corrective actions prematurely could increase 
the risk that they could reoccur and limits the knowledge gained and 
value of the exercise. We are recommending that (1) NORTHCOM's 
Commander establish and publicize procedures for non-DOD exercise 
participants to submit observations, (2) the Chairman of the Joint 
Chiefs of Staff direct the Joint Staff to work with DHS to either 
resolve information assurance issues so that combatant commands can 
post Exercise Summary Reports with lessons learned on DHS's Lessons 
Learned Information Sharing system or establish an alternative method 
to systematically collect and share lessons learned, and (3) the 
Chairman of the Joints Chiefs of Staff direct the Joint Staff to revise 
the joint lessons learned operating instruction to include procedures 
to ensure that corrective actions are implemented and verified in a 
subsequent exercise or operation before being closed. 

NORTHCOM has taken steps to integrate its exercises with the NEP, such 
as participating in several NEP exercises, including combining two of 
its large-scale exercises with NEP exercises. However, NORTHCOM has not 
consistently applied NEP planning and documentation requirements, 
because NEP guidance is unclear on the extent to which federal agencies 
should follow NEP requirements and DOD guidance does not specifically 
address this issue. NEP guidance provides policies and tools for the 
design, planning, conduct, and evaluation of exercises--known as the 
Homeland Security Exercise and Evaluation Program.[Footnote 22] FEMA 
requires that entities, such as states, receiving homeland security 
grant funding for their exercises use Homeland Security Exercise and 
Evaluation Program tools, such as reporting templates. We reviewed NEP 
guidance, such as the Implementation Plan, and found it does not 
clearly state the extent to which federal agencies are required to 
follow the Homeland Security Exercise and Evaluation Program planning 
and documentation guidance. As a result of this unclear guidance, we 
found that agency officials have varying interpretations of the 
requirements. A DOD and a Joint Staff official told us that NEP 
guidance does not require agencies to use the Homeland Security 
Exercise Evaluation Program even for NEP exercises; therefore, NORTHCOM 
uses the Joint Training System as the basis for planning, conducting, 
and assessing exercises. FEMA officials told us that federal agencies 
should use the Homeland Security Exercise Evaluation Program for 
exercises that require some level of interagency participation, so that 
consistent approaches are used by the various participants.[Footnote 
23] Neither DOD nor NORTHCOM guidance specifically addresses this 
issue.[Footnote 24] We recognize that DOD and NORTHCOM must meet their 
own mission and exercise requirements and the Joint Training System may 
be best suited for NORTHCOM's exercises; however, all of the states we 
visited use Homeland Security Exercise and Evaluation Program guidance. 
We found that having differing sets of guidance makes exercise 
development more difficult and potentially limits the benefits for 
participating states. Further, inconsistent approaches to the 
development and content of postexercise documentation may affect the 
ability of organizations to fully learn lessons identified in 
exercises.[Footnote 25] We believe that achieving national preparedness 
requires a whole-of-government approach and is a shared responsibility 
among federal, state, and local governments and organizations and an 
integration of their various standards, policies, and procedures into 
the national system.[Footnote 26] In the absence of clear NEP guidance 
on this issue, DOD should ensure that its components clearly understand 
when use of NEP planning and reporting guidelines is appropriate. We 
are recommending that the Secretary of Defense and the Joint Staff 
provide specific guidance that defines the conditions under which the 
combatant commands should follow the Homeland Security Exercise and 
Evaluation Program planning and documentation requirements or the DOD's 
Joint Training System should be modified for those civil support 
exercises. 

DOD provided written comments on a draft of this report and provided 
technical comments which we incorporated into the final report as 
appropriate. DOD agreed or partially agreed with all our 
recommendations and described actions it is taking or plans to take to 
implement them. DOD generally agreed with the recommendation to work 
with DHS to establish guidance and procedures for requesting state 
participation and involving states in planning, executing, and 
assessing exercises. DOD agreed that better coordination for 
interfacing with state officials can be achieved. However, DOD noted 
that NORTHCOM is the primary training audience for its exercises and 
that the recommendation should also be addressed to an interagency 
exercise planning committee. We noted NORTHCOM's considerable efforts 
to involve state and interagency exercise partners. However, we believe 
that the effective involvement of and interaction with state and other 
federal partners is a critical component of improving and maintaining 
NORTHCOM's preparedness for providing support to civil authorities 
regardless of whether it is a strictly NORTHCOM-sponsored exercise or 
an event conducted under the National Exercise Program. Therefore, 
developing procedures to improve coordination with the states can only 
assist DOD and all its interagency and state partners. We also noted 
that the interagency exercise planning committee is one venue at which 
DOD can effectively coordinate with its interagency partners. Although 
DOD agreed with the recommendation on a training plan for NORTHCOM 
staff and discussed current and future NORTHCOM training efforts, its 
response did not address the need for NORTHCOM staff to be provided 
more thorough training on specific emergency management organizations 
and structures of each state. We continue to believe that such training 
is needed since it is an important element of preparedness for 
exercises and for actually providing civil support in the event of a 
major incident. A summary of DOD's comments and a summary of our 
response to these comments follow the Recommendations for Executive 
Action section of this report. DOD's written comments are reprinted in 
appendix III. DHS also reviewed a draft of this report and provided 
technical comments, which we incorporated into the final report as 
appropriate. 

Background: 

The U.S. homeland continues to face an uncertain, complex security 
environment with the potential for terrorist incidents and natural 
disasters which can produce devastating consequences. Ensuring an 
effective response will require that federal departments and agencies, 
states, and local governments conduct integrated disaster response 
planning and test these plans by exercising together. 

Training and Exercises: 

Exercises play an instrumental role in preparing the nation to respond 
to an incident by providing opportunities to test emergency response 
plans, evaluate response capabilities, assess the clarity of 
established roles and responsibilities, and improve proficiency in a 
simulated, risk-free environment.[Footnote 27] Short of performance in 
actual operations, exercises provide the best means to assess the 
effectiveness of organizations in achieving mission preparedness. 
Exercises provide an ideal opportunity to collect, develop, implement, 
and disseminate lessons learned and to verify corrective action taken 
to resolve previously identified issues.[Footnote 28] Sharing positive 
experiences reinforces positive behaviors, doctrine, and tactics, 
techniques, and procedures, while disseminating negative experiences 
highlights potential challenges in unique situations or environments or 
identifies issues that need to be resolved.[Footnote 29] According to 
the National Response Framework, well-designed exercises improve 
interagency coordination and communications, highlight capability gaps, 
and identify opportunities for improvement.[Footnote 30] There are 
various types of exercises ranging from tabletop exercises that involve 
key personnel discussing simulated scenarios in informal settings to a 
full-scale exercise, including many agencies, jurisdictions, and 
disciplines and a "boots on the ground" response, such as firefighters 
decontaminating mock victims. 

Military Mission and Organization: 

DOD established the Office of the Assistant Secretary of Defense for 
Homeland Defense and Americas' Security Affairs to oversee homeland 
defense activities for DOD, under the authority of the Under Secretary 
of Defense for Policy, and, as appropriate, in coordination with the 
Chairman of the Joint Chiefs of Staff. This office develops policies, 
conducts analysis, provides advice, and makes recommendations on 
homeland defense, defense support of civil authorities, emergency 
preparedness, and domestic crises management matters within the 
department. The assistant secretary assists the Secretary of Defense in 
providing policy directions to NORTHCOM and other applicable combatant 
commands to guide the development and execution of homeland defense 
plans and activities. This direction is provided through the Chairman 
of the Joint Chiefs of Staff. 

The Chairman of the Joint Chiefs of Staff, as principal military 
advisor to the President and Secretary of Defense, has numerous 
responsibilities relating to homeland defense and civil support, 
including providing advice on operational policies, responsibilities, 
and programs. Furthermore, the Chairman of the Joint Chiefs of Staff 
and the Joint Staff are responsible for formulating joint training 
policy and doctrine.[Footnote 31] The Joint Staff assists the Chairman 
by facilitating implementation of the Chairman's joint training 
programs, including the Joint Training System, Chairman's sponsored 
exercise program, and joint exercise program. 

NORTHCOM: 

NORTHCOM is the military command responsible for planning, organizing, 
and executing DOD's homeland defense and civil support missions within 
its area of responsibility--the continental United States (including 
Alaska) and territorial waters (see figure 1).[Footnote 32] Homeland 
defense is the protection of U.S territory, sovereignty, domestic 
population, and critical defense infrastructure against external 
threats and aggression. DOD is the primary federal agency responsible 
for homeland defense operations, such as air defense, and NORTHCOM is 
the combatant command responsible for commanding and coordinating a 
response to a homeland defense incident.[Footnote 33] To carry out its 
homeland defense mission, NORTHCOM is to conduct operations to deter, 
prevent, and defeat threats and aggression aimed at the United States. 

Figure 1: NORTHCOM's Area of Responsibility: 

[Refer to PDF for image: map] 

NORTHCOM's Area of Responsibility includes: 

United States; 
Canada; 
Mexico. 

Alaska falls within NORTHCOM’s area of responsibility, but military 
forces in the state are assigned to U.S. Pacific Command. 

Sources: GAO analysis; Map Resources (map). 

[End of figure] 

NORTHCOM's second mission is civil support or defense support of civil 
authorities. Civil support is DOD support to U.S. civilian authorities, 
such as DHS, for domestic emergencies, both natural and man-made, and 
includes the use of DOD personnel--federal military forces and DOD's 
career civilian and contractor personnel--and DOD agency and component 
resources. Because these missions are complex and interrelated, they 
require significant interagency coordination. Civil support missions 
include domestic disaster relief operations for incidents such as 
fires, hurricanes, floods, and earthquakes. Such support also includes 
counterdrug operations and management of the consequences of a 
terrorist incident employing a weapon of mass destruction. DOD is not 
the primary federal agency for such missions (unless so designated by 
the President) and thus provides defense support of civil authorities 
only when (1) state, local, and other federal resources are overwhelmed 
or unique military capabilities are required; (2) assistance is 
requested by the primary federal agency; and (3) NORTHCOM is directed 
to do so by the President or the Secretary of Defense. [Footnote 34] 
See figure 2 for the pathway for requesting DOD and NORTHCOM assistance 
during an incident. 

Figure 2: Pathways for Requesting Military Assistance for Incident 
Response: 

[Refer to PDF for image: illustration] 

Incident Occurs: 

Incident commander and local first responders: 
Request for federal assistance to Local elected and emergency 
management officials; 

Local elected and emergency management officials: 
Request for federal assistance to Governor and state officials, 
including emergency management officials; 

Governor and state officials, including emergency management officials: 
Request for federal assistance to FEMA Regional Administrator; 
Request for federal assistance after Stafford declaration: State 
Coordinating Officer; 

FEMA Regional Administrator: 
Request for federal assistance to Secretary of Homeland Security/
FEMA Administrator; 

Secretary of Homeland Security/FEMA Administrator: 
Request for federal assistance to President of the United States; 
Declares incident as either a major disaster or emergency under the 
Stafford Act; 
Direct assistance (when assigned) to Secretary of Defense, who Approves 
FEMA requests for assistance. 

President of the United States: 
Appoints Federal Coordinating Officer; 

Joint Field Office (Temporary): 
Established to coordinate federal support for response, but does not
manage on-scene operations; 
State Coordinating Officer: Request for federal assistance after 
Stafford declaration to Federal Coordinating Officer; 

Federal Coordinating Officer: Request for federal assistance after 
Stafford declaration to Defense Coordinating Officer; 
Directs assistance (when assigned) to other federal departments; 

Defense Coordinating Officer: Request for federal assistance after 
Stafford declaration to NORTHCOM; assistance approved by NORTHCOM after 
approval by Secretary of Defense; 

NORTHCOM: 
Approves assistance to Defense Coordinating Officer; 
Directs assistance (when assigned) to Joint Task Force Commander
(Temporary); 

Joint Task Force Commander (along with Defense Coordinating Officer): 
Directs assistance (when assigned) to designated forces support as 
directed; 

Other federal departments and designated forces support as directed: 
Deliver assistance when incident occurs. 

Source: GAO analysis of DOD and DHS documentation. 

[End of figure] 

NORTHCOM conducts or participates in exercises to improve readiness to 
perform its assigned missions. The command annually conducts 2 large- 
scale exercises[Footnote 35]--Ardent Sentry and Vigilant Shield--and 
participates in over 30 smaller command, regional, state, and local 
exercises. Each Ardent Sentry and Vigilant Shield training event 
emphasizes one of the key missions while at the same time including 
elements of the other. Ardent Sentry emphasizes the civil support 
missions; Vigilant Shield the homeland defense missions. The basis for 
NORTHCOM's exercises is DOD's Joint Training System. NORTHCOM's 
Training and Exercise Directorate is responsible for planning and 
executing joint training, exercises, and education programs to ensure 
NORTHCOM is prepared to accomplish its assigned missions. 

DOD's Joint Training System: 

Due to the need to prepare for and conduct military operations to 
defend the United States and fight the nation's wars, DOD has developed 
an established, authoritative, time-tested process for planning, 
conducting, and evaluating exercises in order to test and improve 
preparedness to meet its wide range of critical missions.[Footnote 36] 
NORTHCOM uses DOD's Joint Training System as the basis to design, 
develop, and conduct exercises.[Footnote 37] The Joint Training System 
provides an integrated, requirements-based method for aligning training 
programs with assigned missions consistent with command priorities, 
capabilities, and available resources. The joint system consists of 
four phases beginning with the identification of critical capabilities 
required based on assigned missions, proceeding through the planning 
and scheduling of training events, the execution and evaluation of 
required training, and assessing training proficiency against required 
capability (see figure 3). This process is designed to ensure that an 
organization's training program is linked to the Joint Mission 
Essential Task List,[Footnote 38] the personnel executing the tasks are 
properly trained, and shortfalls in training are identified and 
corrected in order to improve readiness. The Joint Training Information 
Management System is an automated system specifically designed to 
assist users in managing elements of each of the four phases of the 
Joint Training System.[Footnote 39] During the execution phase, 
commanders and directors focus on executing and evaluating planned 
training events, which can be accomplished through academic training, 
exercises, or a combination of these activities.[Footnote 40] During 
the execution stage of the Joint Training System, the Joint Event Life 
Cycle provides a five-stage methodology for joint-event development 
design, planning, preparation, execution, and evaluation. For example, 
DOD components prepare for the execution of an exercise by conducting 
five conferences, such as the Concept Development Conference where 
exercise and training objectives are discussed and scenarios developed. 
Activities for the Joint Event Life Cycle are managed through the Joint 
Training Information Management System. 

Figure 3: DOD's Joint Training System: 

[Refer to PDF for image: illustration] 

Joint Training System: 

Identify capability requirements; 
Identify training requirements & plan required training; 
Execute and evaluate planned training: 
* Joint event life cycle: 
- Design; planning; preparation; execution; evaluation; 
Determine training resources. 

Source: DOD. 

[End of figure] 

Evaluating lessons learned and identifying issues for corrective 
actions are fundamental components of DOD's training and exercise 
process. The Chairman of the Joint Chiefs of Staff provides policy, 
direction, and guidance for DOD's Joint Lessons Learned Program. 
[Footnote 41] The objectives of this program are to collect and analyze 
observations from exercises and real world events; disseminate 
validated observations and findings to appropriate officials; identify 
and implement corrective actions; and track corrective actions until 
reobserved in a subsequent exercise or event to ensure that the issue 
has been successfully resolved. Combatant commands, including NORTHCOM, 
execute lessons discovery, knowledge development, and implementation 
activities scaled to meet the command's requirements while supporting 
and feeding into the Chairman's Joint Lessons Learned Program by 
identifying lessons applicable across combatant commands and the 
services. 

The National Exercise Program: 

The NEP was established in April 2007 under the leadership of the 
Secretary of Homeland Security to prioritize and coordinate federal, 
state, and local exercise activities and serves as the principal 
mechanism for examining the preparation of the federal government to 
respond to an incident and adopting policy changes to improve such 
preparation.[Footnote 42] The day-to-day staff-level coordination of 
the NEP is managed by the NEP Executive Steering Committee--a working 
group of the White House's Domestic Readiness Group Exercise and 
Evaluation Sub-Policy Coordination Committee--and is chaired and 
facilitated by FEMA's National Exercise Division. The steering 
committee is also responsible for framing issues and recommendations 
for the full coordination committee on exercise themes, goals, 
objectives, scheduling, and corrective actions.[Footnote 43] Figure 4 
illustrates the major events and milestones of the NEP and NORTHCOM's 
exercise program and table 1 provides information on related major 
documents. 

Figure 4: National Exercise Program and U.S. Northern Command Exercise 
Program Time line: Major Events and Milestones (2001-2009): 

[Refer to PDF for image: illustration] 

9/11/01: Terrorist attacks; 

2002: 
NORTHCOM Established; 
DHS Established; 
FEMA merges with DHS. 

2003: 
NORTHCOM: 
* United Defense 03 Exercise; 
* Determined Promise 03 Exercise. 
Homeland Security Presidential Directive 8 Issued. 

2004: 
NORTHCOM: 
* United Defense 04 Exercise; 
* Determined Promise 04 Exercise; 
DHS: 
* National Planning Scenarios Issued. 

2005: 
DHS: 
* National Response Plan Published. 

2006: 
Hurricane Katrina; 
NORTHCOM: 
* Vigilant Shield 06 Exercise; 
* Ardent Sentry 06 Exercise; 
Post Katrina Act Enacted. 

2007: 
National Exercise Program (NEP) Charter signed; 
NORTHCOM: 
* Vigilant Shield 07; 
* Ardent Sentry 07; 
DHS: 
* Revised Homeland Security Exercise and Evaluation Program Guidance 
published; 
FEMA: 
* FEMA reorganizes. 
National Exercise Program Implementation Plan approved; 
NORTHCOM: 
* Ardent Sentry 07;
DHS: 
* National Preparedness Guidelines published; 
National Level Exercise 1-08 (Top Officials 4 + Vigilant Shield 08). 
DHS: 
* National Response Framework published. 

2008: 
Joint Chiefs of Staff Lessons Learned Instructions issued; 
National Level Exercise 2-08 (Ardent Sentry 08). 
NORTHCOM: 
* NORTHCOM exercise guidance issued. 

2009: 
DOD NEP Instruction issued; 
NORTHCOM: 
* Vigilant Shield 09. 

Source: GAO Analysis of DOD and DHS documentation. 

[End of figure] 

Table 1: Major Documents Related to the National Exercise Program and 
NORTHCOM: 

Title: Homeland Security Presidential Directive-8; 
Description: Requires the development of the national preparedness 
goal; preparedness priorities; an assessment system; a training and 
exercise program; a system to collect, analyze, and disseminate 
information from exercises, training events, and actual incidents; 
equipment standards; and a federal response capability inventory. 

Title: National Response Framework; 
Description: Establishes a comprehensive, national, all-hazards 
approach to domestic incident response. Replaced the National Response 
Plan. 

Title: National Preparedness Guidelines; 
Description: Establishes a vision for national preparedness and provide 
a systematic approach for prioritizing preparedness efforts across the 
United States. 

Title: National Planning Scenarios; 
Description: Provides planning tools that represent a minimum number of 
credible scenarios depicting the range of potential terrorist attacks 
and natural disasters and related impacts facing the United States They 
form a basis for coordinated federal planning, training, and exercises. 

Title: U.S. North American Aerospace Defense Command (NORAD)-NORTHCOM 
Instruction 10-156; 
Description: Establishes the NORAD and NORTHCOM exercise program, 
provides an overview of the exercise program, describes the principal 
NORAD and NORTHCOM-sponsored exercises, and outlines the process for 
scheduling, planning, execution, and after action review of exercises. 

Title: Chairman of the Joint Chiefs of Staff Instruction 3150.25D; 
Description: Codifies the Joint Lessons Learned Information System as 
the DOD system of record for the Joint Lessons Learned Program. 

Title: DOD Instruction 3020.47; 
Description: Establishes policy, assigns responsibilities, and provides 
instructions for DOD participation in the National Exercise Program. 

Source: DOD and DHS. 

[End of table] 

The NEP includes a series of national exercises projected on a 5-year 
exercise schedule. These exercises are organized into four tiers with 
each tier reflecting different requirements for interagency 
participation (see figure 5). 

Figure 5: Description of NEP Tiers: 

[Refer to PDF for image: illustration] 

The illustration is a pyramid with Tier I and the top, and each 
subsequent tier below: 

Tier I: 
White House directed, U.S. Government-wide strategy and policy focus, 
full federal agency participation. 5 Tier I exercises conducted 
annually (1 national-level exercise and 4 principal-level exercises). 

Tier II: 
Federal Department/Agency directed, U.S. government-wide strategy and 
policy focus. Full Federal Agency participation in person or through
simulation. No more than 3 Tier II exercises conducted annually. 

Tier III: 
Other federal exercises, regional operational, tactical or 
organizational-specific focus. Full federal agency participation is 
optional. The number of exercises conducted annually depends upon 
individual entities’ needs and requirements. 

Tier IV: 
Non-Federal Exercises, State, Territorial, Local and or tribal 
governments and/or private sector focus. Federal Agency participation 
is optional. The number of exercises conducted annually depends upon 
individual entities’ needs and requirements. 

Source: FEMA. 

[End of figure] 

FEMA administers the NEP and maintains the Homeland Security Exercise 
and Evaluation Program--a capabilities and performance-based exercise 
program--to provide standardized policy, methodology, and terminology 
for exercise design, development, conduct, evaluation, and improvement 
planning. DHS maintains policy and guidance for this program.[Footnote 
44] 

Similar to DOD's Joint Training System, the Homeland Security Exercise 
Evaluation Program uses an exercise life cycle with five phases: 
foundation, design and development, conduct, evaluation, and 
improvement planning. This program also provides document templates for 
exercise planning and evaluation and a collection of interactive, on- 
line systems for exercise scheduling, design, development, conduct, 
evaluation, and improvement planning, referred to as the Homeland 
Security Exercise Evaluation Program Tool Kit (see figure 6).[Footnote 
45] FEMA also has additional resources to support exercises. For 
example, exercise stakeholders can access FEMA's Lessons Learned 
Information Sharing system, an interagency Web site for posting lessons 
learned and sharing best practices, to learn about promising practices 
that could facilitate exercise activities.[Footnote 46] 

Figure 6: Homeland Security Exercise and Evaluation Program Tool Kit: 

[Refer to PDF for image: illustration] 

Foundation: 
National Exercise Schedule System: 
A national online comprehensive tool that facilitates scheduling and
synchronization of national-level, federal, state, and local exercises.
Exercise program management. 

Exercise program management: 

Design, development, and conduct: 

Design and Development System: 
A project management tool and comprehensive tutorial for the design,
development, conduct, and evaluation of exercises. 

Exercise Evaluation Guide Builder (Beta): 
An online application that enables users to customize exercise 
evaluation guides and templates. 

Master Scenario Events List Builder (Beta): 
A tool that enables users to create customized master scenario events 
list formats by selecting from a list of data fields. 

Evaluation and Improvement Planning: 
Corrective Action Program System: 
An online application that enables users to prioritize, track, and 
analyze improvement plans developed from exercises and real-world 
events. 

Source: GAO analysis of FEMA documentation. 

[End of figure] 

NORTHCOM's Exercise Program Is Consistent with DOD's Joint Training 
System, but Exercise Reporting is Inconsistent: 

NORTHCOM's Exercise Program is Consistent with DOD's Joint Training 
System: 

NORTHCOM's Commander's Training Guidance requires that NORTHCOM 
establish a training and exercise program consistent with the Joint 
Training System and establishes that training efforts and resources 
will be focused on two large-scale exercises annually.[Footnote 47] The 
Joint Training System requires, among other things, that an 
organization's training objectives be linked to its Joint Mission 
Essential Task List and include the use of the Joint Events Life Cycle 
for planning, conducting, and assessing exercises.[Footnote 48] We 
found that NORTHCOM has developed a comprehensive exercise program 
consistent with DOD's Joint Training System. For example, NORTHCOM uses 
the Joint Training Information Management System to link training 
objectives with its Joint Mission Essential Task List. NORTHCOM 
officials enter information on task performance of exercise 
participants into the Joint Training Information Management System to 
evaluate the extent to which the command is trained based on 
performance requirements in the Joint Mission Essential Task List. 
[Footnote 49] NORTHCOM also uses the Joint Training Information 
Management System to manage the Joint Events Life Cycle for its large- 
scale exercises, including planning exercise milestones and developing 
a time line that allows exercise planners to see where they are in the 
event life-cycle process. For example, NORTHCOM holds five planning 
conferences for each exercise, including a concept development 
conference, where exercise and training objectives are discussed and 
scenarios developed.[Footnote 50] We also found that NORTHCOM has 
conducted 13 large-scale exercises since it was created in 2002, 
generally including 2 exercises each year (see table 2). Vigilant 
Shield is held in the fall and focuses primarily on NORTHCOM's homeland 
defense mission, and Ardent Sentry is generally conducted in the spring 
and focuses on defense support of civil authorities. 

Table 2: Large-Scale NORTHCOM Exercises Since 2002: 

Exercise: Ardent Sentry 09; 
Dates: June 18-24, 2009; 
Scenarios: 
* Flooding along Mississippi River; 
* Domestic terrorist organization releases foot and mouth disease at 4-
H expo; 
* Anthrax released into the food supply; 
* Train derailments in Iowa and Kansas resulting in chemical releases; 
* Domestic terrorist organization launches attack against a Wyoming Air 
Force Base; 
* U.S. and Canadian maritime incidents. 

Exercise: Vigilant Shield 09; 
Dates: Nov. 12-18, 2008; 
Scenarios: 
* Earthquake along San Andreas Fault; 
* Homeland defense vignettes associated with mission areas. 

Exercise: Ardent Sentry 08; 
Dates: May 1-8, 2008; Scenarios: 
* Category 4 hurricane strike and terrorist threat affecting National 
Capital Region; 
* Accidental VX nerve gas release in Oregon; 
* Multiple terrorist attacks in Washington to exercise both response to 
chemical weapons attack and specific defense support of civil 
authorities capabilities. 

Exercise: Vigilant Shield 08; Dates: Oct. 15-20, 2007; Scenarios: * 
Series of exploded radiological dispersal devices; * Aircraft 
accidents; * Reverse no-fly list; * Deployment of a command assessment 
element. 

Exercise: Ardent Sentry 07; 
Dates: April 30-May 17,2007; 
Scenarios: 
* Critical infrastructure protection; 
* Maritime operations; 
* Detonation of a 10-kiloton improvised nuclear device in Indiana; 
* Category III hurricane impacting the New England area. 

Exercise: Vigilant Shield 07; 
Dates: Oct. 16-27, 2006 and Dec. 4-15, 2006; 
Scenarios: 
* Crash and explosion of plane carrying nuclear weapons; 
* Maritime attacks/events; 
* Rogue aircraft and North American Air Defense Command Air and 
Strategic Threat events; 
* Strategic Ballistic Missile Defense Force Protection; 
* Defense against cyber attacks. 

Exercise: Ardent Sentry 06; 
Dates: May 8-18, 2006; 
Scenarios: 
* Hurricane strike on New Orleans, Louisiana; 
* Explosion of a railcar carrying phosgene; 
* U.S. and Canadian Infrastructure attack; 
* Chemical attack; 
* Radiological incident; 
* Avian flu pandemic outbreak scenario. 

Exercise: Vigilant Shield 06; 
Dates: Nov. 1-10, 2005; Scenarios: 
* Chemical scenario; 
* Critical infrastructure protection of a nuclear power plant; 
* Biological scenario; 
* Ballistic missile defense threat; 
* Aerospace defense. 

Exercise: Ardent Sentry 05; 
Dates: Apr. 4-9, 2005; 
Scenarios: 
* Exercise deployment and employment of Joint Task Force-Civil Support 
in support of FEMA Region I mission assignment requests in a chemical 
response; 
* Response to a biological event; 
* Protection of critical infrastructure (nuclear power plant). 

Exercise: Determined Promise 04; 
Dates: Aug. 5-10, 2004; 
Scenarios: 
* Simultaneous terrorist air events (hijacked aircraft, aircraft with 
terrorists on watch list, low-altitude defense and swarm attack by 
business jet aircraft); 
* Explosion of container with radiological dispersal device in Long 
Beach, California harbor; 
* Train derailment and chemical spill; 
* Multiple sarin, mustard, and chlorine gas attacks in Virginia; 
* Maritime intercept operations; 
* Attack by unmanned aerial vehicles with high-explosive warheads fired 
from oil platform; 
* Cyber security event; 
* Ballistic missile defense. 

Exercise: Unified Defense 04; 
Dates: Feb. 19-25, 2004; 
Scenarios: 
* Air threats to Alaska, Canada, and the United States; 
* Category IV+ hurricane in Corpus Christi, Texas; 
* Nuclear detonation in Texas; 
* Quick reaction force deployments to Texas and Alaska; 
* Transportation security and U.S. Coast Guard incidents in Alaska; 
* Maritime intercept operations events; 
* Ballistic missile defense threat. 

Exercise: Determined Promise 03; 
Dates: Aug. 18-28, 2003; 
Scenarios: 
* Terrorist attack of a biological weapon of mass destruction in Clark 
County, Nevada; 
* Wildfires in the western United States; 
* 2003 World Gymnastics Championship in California; 
* Ongoing operational plan load-out operations; 
* Sustained, diverse maritime events; 
* Air threat to Alaska; 
* High-interest vessel threat; 
* Strategic infrastructure threats in the northwest United States; 
* Category III hurricane in the southeast United States; 
* Train derailment in Kentucky; 
* Shipping and security issues in Alaska. 

Exercise: Unified Defense 03; 
Dates: Feb. 6-13, 2003[A]; 
Scenarios: 
* National special security event - Super Bowl; 
* Joint Task Force-Civil Support held in strategic reserve for State of 
the Union Address and Super Bowl; 
* Nuclear threat along southwest U.S. border; 
* Border surveillance; 
* Threat to port; 
* Consequence management in northeast United States; 
* Natural disaster in Alaska. 

Source: GAO analysis of NORTHCOM documents. 

[A] Unified Defense 03 started late due to the Columbia Shuttle 
disaster and ended early due to a level Orange alert in the Homeland 
Security Alert System. 

[End of table] 

NORTHCOM's Documentation of Exercises Is Generally Timely but 
Inconsistent: 

NORTHCOM guidance outlines the postexercise documentation required to 
be completed for each exercise, including quick look, after-action, and 
exercise summary reports; provides a time line for the completion of 
these documents; and includes general direction that these documents 
follow the same focus areas as the collection management plan--the 
source document from which exercise analysts identify, examine, and 
recommend emerging issues and trends.[Footnote 51] We found that 
NORTHCOM has generally completed exercise summary reports for its 
exercises; however, neither NORTHCOM nor Joint Forces Command officials 
could locate an exercise summary report for Unified Defense 03. 
[Footnote 52] In addition, postexercise documentation is not 
consistently included on NORTHCOM's portal or the Joint Training 
Information Management System. NORTHCOM guidance issued in June 2008 
provides a time line for the completion of postexercise documents and 
has been applicable to 2 subsequent exercises-Ardent Sentry 08 and 
Vigilant Shield 09. According to the 2008 guidance, the exercise 
summary report is to be submitted to the NORTHCOM Commander within 90 
days of completing an exercise.[Footnote 53] The Ardent Sentry 08 and 
Vigilant Shield 09 exercise summary reports were issued 99 days and 92 
days, respectively, after the completion of each exercise. Overall, we 
reviewed exercise summary reports for 11 of NORTHCOM's large-scale 
exercises that have taken place since 2003. Seven of the 11 exercise 
summary reports were issued within 100 days. Four of the reports were 
issued later than 100 days, and 1 of NORTHCOM's earlier reports was 
issued in less than 30 days. 

NORTHCOM guidance states that exercise summary reports should provide 
the official description of the exercise, identify significant lessons 
learned, and be targeted toward a national audience.[Footnote 54] 
Guidance also requires that exercise summary reports follow the same 
focus areas as the collection management plan--the source document from 
which exercise analysts identify, examine, and recommend emerging 
issues and trends.[Footnote 55] We found that NORTHCOM's exercise 
summary reports generally included an executive summary, training 
objectives, and the exercise's major scenarios and events, but did not 
consistently include lessons learned, exercise strengths and 
weaknesses, or clear recommendations. The exercise summary reports that 
included a section on lessons learned lacked details. For example, 6 of 
the 11 exercise summary reports we reviewed included an identified 
lessons learned section, and just 1 of these 6 reports--Unified Defense 
04--provided additional information on lessons learned beyond 
identifying the title of each observation and the status of the 
observation in the lessons learned management system. As discussed 
later in this report, access to this system is required in order to 
obtain any additional information on the lesson learned.[Footnote 56] 

We also found that NORTHCOM exercise summary reports have not followed 
the same focus areas as collection management plans.[Footnote 57] For 
example, none of the seven exercise summary reports for NORTHCOM 
exercises conducted since Hurricane Katrina in 2005 reported on the 
information identified in the collection management plans' focus areas. 
Inconsistencies in exercise documentation may be occurring because DOD 
and NORTHCOM guidance do not require a standard format or specific 
content for postexercise documentation. Although NORTHCOM uses other 
methods to document exercises, such as the Joint Training Information 
Management System, this system does not include a complete record of 
each exercise. For example, the Joint Training Information Management 
System does not include the lessons learned from an exercise. In 
addition, access to this system is generally limited to DOD officials. 
Recognizing the need for a complete and consistent record of each 
exercise, DHS's Homeland Security Exercise Evaluation Program provides 
a template for exercise documentation, including format and content. 
[Footnote 58] NORTHCOM used this template for National Level Exercise 2-
08, but does not use the template for its own exercises.[Footnote 59] 
Despite differences in the requirements and complexities of NORTHCOM's 
and DHS' exercise programs, the lack of a complete and consistent 
record of each exercise lessens the extent to which NORTHCOM can ensure 
it has trained to key focus areas. Further, it deprives the command of 
a key source of historical information upon which to base current and 
future assessments of exercises and a consistent venue for sharing 
lessons learned with interagency partners and states. 

Interagency Partners and States Have Participated In NORTHCOM's 
Exercises, but Challenges Remain with Including States in Planning, 
Conducting, and Assessing Exercises: 

Federal Agencies and States Have Participated in NORTHCOM Exercises: 

NORTHCOM recognizes the importance of exercising with key partners in 
all its missions and that, in order to achieve its goal of being 
trained and ready to execute joint operations and ensure a seamless 
operating environment, NORTHCOM should maximize exercise participation 
with federal, state, and local agencies and National Guard units. 
[Footnote 60] NORTHCOM has included interagency partners, such as DHS, 
FEMA, and the U.S. Coast Guard, and several states in its large-scale 
exercises (see table 3). We found that 17 civilian federal agencies and 
organizations have participated to varying degrees in one or more of 
the seven large-scale NORTHCOM exercises that have occurred since 
Hurricane Katrina made landfall in August 2005. Seventeen states have 
participated in NORTHCOM exercises since that time, and 8 of these 
states--Arizona, California, Connecticut, Indiana, Michigan, Oregon, 
Rhode Island, and Washington--played a major role by having a portion 
of the exercise conducted in the state and having various state 
agencies and officials participate.[Footnote 61] For example, Indiana 
and Rhode Island played major roles in Ardent Sentry 07 for the 
detonation of a 10-kiloton improvised nuclear device and category III 
hurricane impacting the New England region, respectively. Both states 
established emergency operating centers and exercised large numbers of 
state emergency management personnel. 

Table 3: Interagency Partner and State Participation in Large-scale 
NORTHCOM Exercises Since 2006: 

Exercise: Vigilant Shield 06; 
Other federal agencies and organizations: 
* Department of Homeland Security; 
* Other federal organizations in the areas of homeland defense and 
defense support of civil authorities[B]; 
States: 
* Connecticut; 
* New Jersey. 

Exercise: Ardent Sentry 06; 
Other federal agencies and organizations: 
* Coast Guard Sector Detroit District 9; 
* Department of Energy; 
* Department of Health and Human Services; 
* Department of Homeland Security; 
* Department of Justice; 
* Department of State; 
* Department of Transportation; 
* Environmental Protection Agency; 
* Federal Aviation Administration; 
* Federal Bureau of Investigation; 
* FEMA; 
* Transportation Security Agency; 
* U.S. Coast Guard/Atlantic Area; 
* U.S. Department of Agriculture; 
States: 
* Arizona; 
- Division of Emergency; Management; 
- Joint Terrorist Task Force; 
- Tactical Intelligence Center; 
* Maine; 
* Michigan; 
- Detroit Metropolitan Airport; 
- State Emergency Operation Center; 
- State Police; 
- Service Emergency Preparedness Liaison Officer; 
- St. Clair County Regional Response Team; 
- Wayne County. 

Exercise: Vigilant Shield 07; 
Other federal agencies and organizations: 
* Department of Energy; 
* DHS National Interagency Simulation Cell; 
* Department of Justice; 
* Department of Transportation; 
* FEMA; 
* DHS Principal Federal Official; 
* U.S. Coast Guard; 
* Department of Agriculture; 
* Environmental Protection Agency; 
* Federal Bureau of Investigation; 
* Department of Health and Human Services; 
* DHS Immigration and Customs Enforcement; 
* National Geospatial Intelligence Agency; 
* DHS Transportation Security Administration; 
* National Security Agency; 
* Veterans Administration; 
States: 
* Arizona; 
- Pima County Emergency Management; 
- Tucson Police and Fire; 
* Washington, DC; 
* Alaska. 

Exercise: Ardent Sentry 07; 
Other federal agencies and organizations: 
* DHS-National Operations Center; 
* FEMA; 
* FEMA Region I; 
* FEMA Region II; 
* FEMA Region X; 
* Federal Bureau of Investigation; 
* U.S. Coast Guard Pacific Area; 
States: 
* Indiana; 
- Department of Homeland Security; 
- Indianapolis, Indiana; 
- Marion County, Indiana; 
* Rhode Island; 
* Connecticut; 
* Massachusetts; 
* Vermont; 
* New Hampshire; 
* Maine; 
* New York; 
* Ohio; 
* Illinois; 
* Alaska. 

Exercise: National Level Exercise 1-08/Vigilant Shield 08; 
Other federal agencies and organizations: 
* FEMA; 
* DHS-National Operations Center; 
* FEMA Regions IX and X; 
* U.S. Coast Guard Pacific Area; 
States: 
* Oregon; 
- City of Portland; Arizona; 
- City of Phoenix; 
* Guam[A]. 

Exercise: National Level Exercise 2-08/Ardent Sentry 08; 
Other federal agencies and organizations: 
* Department of Homeland Security; 
* FEMA National Exercise Division; 
* Other federal organizations in the areas of homeland defense and 
defense support of civil authorities[B]; 
States: 
* Washington, DC; 
* Oregon; 
* Washington. 

Exercise: Vigilant Shield 09; 
Other federal agencies and organizations: 
* FEMA Region IX; 
* Other federal organizations in the areas of homeland defense and 
defense support of civil authorities[B]; 
States: 
* California. 

Exercise: Ardent Sentry 09; 
Other federal agencies and organizations: 
* Dept of Energy; 
* Dept of Health and Human Services; 
* Dept of Homeland Security; 
* Environmental Protection Agency; 
* Federal Aviation Administration; 
* Federal Bureau of Investigation; 
* FEMA Region VII; 
* FEMA Region VIII; 
* U.S. Coast Guard Operational Command; 
* FEMA National Response Coordination Center; 
States: 
* Iowa; 
- Dept of Agriculture; 
- Dept of Health; 
- Dept of Public Safety; 
- Homeland Security & Emergency Management Division; 
- Iowa State Emergency Operations Center; 
- City of Des Moines Fire Dept.; 
- City of Des Moines Police Dept.; 
* Kansas; 
- Dept of Emergency Management; 
- Dept of Health/Environment; 
- Dept of Agriculture; 
- Dept of Animal Health; 
- Dept of Transportation; 
- Highway Patrol; 
* Nebraska; 
* Missouri; 
* Wyoming. 

Source: GAO analysis of NORTHCOM exercise documents. 

[A] The US territory of Guam is in U.S. Pacific Command's area of 
responsibility. 

[B] Exercise summary reports for Ardent Sentry 08 and Vigilant Shield 
06 and 09 indicate that other federal organizations in the areas of 
homeland defense and defense support of civil authorities participated 
in the exercises, but the reports do not provide specific names of 
organizations and we were unable to identify these organizations in the 
Joint Training Information Management System. 

[End of table] 

State emergency management and National Guard officials told us that 
they participated in NORTHCOM exercises because they wanted to better 
understand the (1) capabilities that NORTHCOM could bring to the 
response to an incident and (2) command and control issues of the 
troops in a state when NORTHCOM is involved. We previously reported 
that states' participation in NORTHCOM exercises helps to build 
relationships and improve coordination.[Footnote 62] Officials from all 
of the states we met with told us that they derived benefits from their 
participation in these exercises. For example, state emergency 
management officials from three states told us that first-hand 
interaction with federal military forces and the opportunity to observe 
the federal response to an incident was beneficial. In addition, two 
state emergency management and National Guard officials told us that 
NORTHCOM officials were professional, well-trained, and helpful. 
Further, officials from five states told us that NORTHCOM provided 
beneficial resources, such as funds for travel to attend exercise 
planning conferences and contractor staff to help state officials 
prepare exercise scripts and injects.[Footnote 63] Finally, officials 
from two states told us that the benefits of working with NORTHCOM 
included gaining an understanding of the resources and capabilities 
that NORTHCOM can provide, as well as understanding how NORTHCOM 
coordinates its response through FEMA. 

NORTHCOM is also attempting to include states in exercises through the 
Vigilant Guard Program. The goal of the Vigilant Guard Program is to 
enhance National Guard and State emergency management agency 
preparedness to perform their homeland defense and Defense Support to 
Civil Authorities roles and responsibilities. It focuses on State Guard 
Joint Force Headquarters coordination with the state emergency 
management agency and Joint Task Force-State operations and involves 
multiple states and agencies. The program began in September 2004 and 
included one exercise in fiscal year 2005. Now the plan is to conduct 
four exercises annually. NORTHCOM was given management responsibility 
for the Vigilant Guard exercises in 2007, although the National Guard 
Bureau retains responsibility for budgeting for these events. Two of 
the four annual Vigilant Guard exercises are to be linked to major 
combatant command exercises, usually NORTHCOM's Ardent Sentry and 
Vigilant Shield. States hosting a Vigilant Guard exercise determine the 
objectives for these events, and NORTHCOM provides support. Separate 
planning begins for these Vigilant Guard exercises prior to the related 
planning meetings for any linked NORTHCOM exercise. NORTHCOM's Ardent 
Sentry 09 is linked with a Vigilant Guard exercise in Iowa with 
scenarios including a train derailment and a chemical spill, an 
epidemic outbreak, and terrorism incident. 

NORTHCOM Faces Challenges Involving Interagency Partners and States in 
Planning, Conducting, and Assessing Exercises: 

A key element to developing effective working relationships with all 
states is a well-thought out and consistent process for including the 
states in planning, conducting, and assessing exercises. Without such a 
process, states may be unwilling to participate in future NORTHCOM 
exercises, impacting the seamless exercise of all levels of government 
and potentially affecting NORTHCOM's ability to provide support to 
civil authorities. We found that challenges remain which have resulted 
in inconsistencies in the way that NORTHCOM involves the states in its 
exercises. One of DOD's challenges is adapting its exercise system and 
practices to accommodate the coordination and involvement of other 
federal, state, local, and tribal agencies that do not have the same 
kinds of practices or level of planning effort. Differences in exercise 
culture stem from differences in missions, experience, authority, 
scope, and resources available to DOD, interagency partners, and 
states. DOD has an established, authoritative, time-tested process for 
planning, conducting, and evaluating exercises in order to test and 
improve preparedness to meet its wide range of critical missions. 
Within DOD, training and exercises are considered a vital component of 
its overall mission of defending the national interests and significant 
resources are devoted to these activities. In contrast, DHS, as the 
lead for interagency homeland security efforts, is a new agency and has 
faced challenges since it was created due to frequent reorganization 
and not being fully staffed.[Footnote 64] DHS and other civilian 
agencies and state and local governments have day-to-day missions and 
responsibilities that may take priority over exercises and often do not 
have the resources or experience to participate in or conduct 
exercises. For example, DOD exercises often are conducted 24 hours a 
day, 7 days a week and may last a week or more to enhance the realism 
of the exercise, while civilian agencies generally participate 8 hours 
per day, usually--according to NORTHCOM officials--during normal 
business hours, and do not exercise longer than a few days. Therefore, 
DOD exercises are generally longer in duration, more resource 
intensive, and involve more participants than other federal and state 
exercises. Furthermore, DOD views itself as the last line of defense 
and often exercises until resources are exhausted to fully assess 
capabilities and identify areas needing improvement. Civilian agencies 
and states may prefer not to exhaust resources during an exercise in 
order to avoid appearing unprepared for an incident and the associated 
political controversy. 

Another challenge that NORTHCOM faces is exercising with the various 
states and territories within its area of responsibility considering 
the legal and historical limits of the constitutional federal-state 
structure. The states have a wide range and type of civilian state 
agencies responsible for emergency management, some of which are headed 
by the Adjutant General of the state, who also heads the military 
department or National Guard, and others are completely separate 
entities (see table 4). 

Table 4: Emergency Management Structures of the States We Visited: 

State: Arizona; 
Exercise(s): Ardent Sentry 06; Vigilant Shield 07; 
Emergency management structure: 
* The Adjutant General is the director of the Department of Emergency 
and Military Affairs, which is a state agency combining Arizona's Army 
and Air National Guard with Joint Military Programs and Emergency 
Management. 

State: California; 
Exercise(s): Vigilant Shield 09; Determined Promise 4; 
Emergency management structure: 
* The California Emergency Management Agency is a Governor's cabinet-
level agency that is led by a Secretary who reports to the Governor; 
* The Adjutant General is the head of the California State Military 
Department under which the National Guard operates. 

State: Michigan; 
Exercise(s): Ardent Sentry 06; 
Emergency management structure: 
* The Civilian Emergency Management and Homeland Security Division is 
located in the State Police Department; 
* The Adjutant General is the director of the Department of Military 
and Veteran Affairs under which the National Guard operates. 

State: Nevada; 
Exercise(s): Determined Promise 03; 
Emergency management structure: 
* The Division of Emergency Management is part of the Department of 
Public Safety (which also includes the Highway Patrol and State Fire 
Marshall); 
* The Adjutant General is the head of the Nevada Office of the Military 
which is also known as the National Guard. 

State: Oregon; 
Exercise(s): Vigilant Shield 08; National Level Exercise 1-08; 
Emergency management structure: 
* The Oregon Office of Emergency Management operates administratively 
under the Adjutant General and the Oregon Military Department. The 
Adjutant General also directs, manages, and supervises the Oregon 
National Guard. The Office of Emergency Management's operations, 
however, are kept separate from those of the Oregon National Guard. 

State: Rhode Island; 
Exercise(s): Ardent Sentry 07; 
Emergency management structure: 
* The Adjutant General heads the Emergency Management Agency and the 
National Guard. These offices are co-located with the National Guard 
playing a supporting role to the Emergency Management Agency. 

State: Washington; 
Exercise(s): Ardent Sentry 08; 
National Level Exercise 2-08; Emergency management structure: 
* The Adjutant General heads the Military Department which houses the 
Washington Emergency Management Division, National Guard, Air National 
Guard, and State Guard. 

Source: GAO analysis of NORTHCOM documents and information provided by 
states. 

[End of table] 

Working with states has been the responsibility of the National Guard 
and is relatively new for a federal military command like NORTHCOM. 
NORTHCOM officials face challenges in dealing with the various civilian 
agencies, differing emergency management structures, capabilities, and 
needs of the states. For example, for Ardent Sentry 08 (linked with 
National Level Exercise 2-08), NORTHCOM planned a scenario involving a 
chemical bomb attack in Seattle, Washington without consulting the 
state health department or civil support team--the agencies responsible 
for responding to a chemical or biological attack. State officials told 
us that NORTHCOM invited the health department to participate once 
state officials informed them that they should be involved, but that 
the scenario was already locked in without the input of this key 
participant. DOD officials told us that they rely on FEMA regional 
offices to provide information on state agencies. However, we believe 
that NORTHCOM officials should have determined if all relevant agencies 
were included in the exercise when directly interacting with state 
officials during the scenario development and other planning 
conferences, before the scenarios were locked in. Washington emergency 
management officials told us that this affected the realism of the 
exercise. 

NORTHCOM also faces challenges in balancing its training objectives 
with those of state agencies and organizations. State and local 
governments seek to exercise their first responder capabilities before 
having their resources overwhelmed and needing to seek federal 
assistance. On the other hand, NORTHCOM seeks to exercise its 
capability to provide support to civil authorities when local, state, 
and other federal resources are exhausted. This necessarily requires 
scenarios that exceed the states' capabilities and that stress DOD 
capabilities. Officials from four of the seven states we interviewed 
told us that NORTHCOM's exercise scenarios appeared unrealistic, 
overwhelmed their states too soon during the exercise, or did not allow 
states to fully exercise their own training objectives.[Footnote 65] 
For example, the scenario for Ardent Sentry 06 included multiple 
improvised explosive devices detonating over a 4-day period in various 
sites, such as the City of Detroit, St. Clair and Wayne Counties, 
Michigan, and Windsor, Ontario, Canada with over 14,000 fatalities and 
a simultaneous pandemic flu outbreak in Michigan.[Footnote 66] State 
emergency management officials told us that such a large number of 
casualties would overwhelm state resources almost immediately, and 
therefore precluded fully exercising training objectives for state and 
local responders.[Footnote 67] Officials from these states told us that 
because they did not have the opportunity to exercise their own 
training objectives, they believed NORTHCOM was using them as a 
training tool.[Footnote 68] A NORTHCOM official told us that NORTHCOM 
needs the states to participate in exercises and, therefore, will be 
flexible to accommodate other organizations' training objectives; 
however, NORTHCOM ultimately has its own objectives to exercise. 
Officials from five of seven states noted that, for example, they face 
budget and staffing limitations, and playing a major role in a NORTHCOM 
exercise often requires establishing a state emergency operations 
center with numerous staff and agencies involved. Given the expansive 
scenarios NORTHCOM uses to guide its exercises and the perception of 
half of the states we visited that this limits the benefits to them, we 
believe that the states may be less likely to expend scarce resources 
to participate in future NORTHCOM exercises. 

Inconsistencies with how NORTHCOM involves states in planning, 
conducting, and assessing exercises is occurring in part because 
NORTHCOM officials lack experience dealing with the various state 
agencies and emergency management structures. Inconsistencies are also 
occurring because NORTHCOM has not established an informed, consistent 
process for including states in its exercises. One aspect of this 
process is the way that NORTHCOM requests state participation in its 
exercises. Currently, NORTHCOM has various processes for requesting 
that other federal departments and agencies participate in its 
exercises, such as making the request through the Joint Staff. FEMA 
officials told us that requests for state participation in NORTHCOM 
exercises should be made through FEMA's regional offices. However, 
because NORTHCOM does not have an established process for requesting 
state participation, officials from the states we visited told us that 
NORTHCOM officials made requests informally and in a variety of ways, 
including through the National Guard Bureau, the state's National 
Guard, or FEMA's regional offices. [Footnote 69] In some cases, such as 
when the state emergency management agency and state National Guard 
have a close working relationship, this method has been effective for 
NORTHCOM. However, in other cases, this method has led to more limited 
exercises. For example, emergency management officials from one state 
told us that NORTHCOM does not have full state representation if it 
only exercises with the state National Guard. In that case, NORTHCOM 
therefore misses out on interaction with other key state emergency 
management officials and responders and affects the realism of the 
exercise. 

Another aspect of the lack of a consistent process for requesting state 
participation is potentially missing the opportunity to leverage the 
existing expertise of the National Guard Bureau and defense 
coordinating officers located in each of the 10 FEMA regional offices. 
As we previously reported, the National Guard Bureau and defense 
coordinating officers have knowledge and experience in dealing with 
states in their region and may be a valuable resource for NORTHCOM 
officials during the planning and conduct of exercises.[Footnote 70] 
The three defense coordinating officers with whom we met told us that 
they participate in NORTHCOM exercises, but currently their role does 
not involve requesting state participation on behalf of NORTHCOM or 
providing state-specific information to NORTHCOM exercise officials. 
Without an informed and consistent process for including the states in 
planning, conducting, and assessing its exercises, NORTHCOM increases 
the risk that its exercises will not provide benefits for all 
participants, impacting the seamless exercise of all levels of 
government and potentially affecting NORTHCOM's ability to provide 
support to civil authorities. 

NORTHCOM has a Systematic Lessons Learned and Corrective Action 
Program, but Gaps Remain: 

NORTHCOM Has a Systematic Lessons Learned and Corrective Action 
Process: 

DOD and NORTHCOM guidance requires that NORTHCOM identify observations 
during the course of normal operations, exercises, and real-world 
events; capture the detail required to fully understand the problem; 
and share valid lessons learned and issues as widely as possible. 
[Footnote 71] NORTHCOM has been identifying observations, lessons 
learned, and needed corrective actions from its exercises and 
operations since the command was created in 2002. NORTHCOM collects and 
tracks observations through the Joint Lessons Learned Information 
System (JLLIS)--the automated official DOD system for managing and 
tracking exercise observations and recording lessons learned.[Footnote 
72] As of April 2009, DOD exercise participants input 94 observations 
into JLLIS during NORTHCOM's most recent large-scale exercise, Vigilant 
Shield 09. Table 5 shows the observations entered into JLLIS or its 
predecessor for NORTHCOM's major exercises since 2006.[Footnote 73] 

Table 5: Number of Observations Input into Lessons Learned Management 
System for Large-Scale Exercises Since 2006: 

NORTHCOM exercise: Vigilant Shield 09; 
Unclassified observations[A]: 75; 
Classified observations: 19; 
Total observations per exercise: 94. 

NORTHCOM exercise: Ardent Sentry 08 (National Level Exercise 2-08); 
Unclassified observations[A]: 46; 
Classified observations: 0; 
Total observations per exercise: 46. 

NORTHCOM exercise: Vigilant Shield 08 (National Level Exercise 1-08); 
Unclassified observations[A]: 34; 
Classified observations: 0; 
Total observations per exercise: 34. 

NORTHCOM exercise: Ardent Sentry 07; 
Unclassified observations[A]: 113; 
Classified observations: 4; 
Total observations per exercise: 117. 

NORTHCOM exercise: Vigilant Shield 07; 
Unclassified observations[A]: 95; 
Classified observations: 12; 
Total observations per exercise: 107. 

NORTHCOM exercise: Ardent Sentry 06; 
Unclassified observations[A]: 54; 
Classified observations: 2; 
Total observations per exercise: 56. 

NORTHCOM exercise: Vigilant Shield 06; 
Unclassified observations[A]: 12; 
Classified observations: 1; 
Total observations per exercise: 13. 

Source: Joint Lessons Learned Information System. 

Note: This includes active records on the Non-Secure Internet Protocol 
Router Network as of April 9, 2009; and on the Secure Internet Protocol 
Router Network (classified and unclassified) as of May 4, 2009: 

[A] Includes unclassified active records from the Secure Internet 
Protocol Router Network. 

[End of table] 

The philosophy and approach of NORTHCOM's Lessons Learned Program have 
been largely the same since NORTHCOM published its first instruction 
for the program in 2003,[Footnote 74] although the requirement to re- 
observe corrective actions in a subsequent exercise or operation before 
closing them was not established until 2005.[Footnote 75] We found that 
NORTHCOM generally has a systematic lessons learned and corrective 
action program, based on clear procedures and a regular process. 
Observations are assigned to an office of primary responsibility within 
NORTHCOM and categorized as either a lesson learned--a positive 
finding---or an issue which requires corrective action. NORTHCOM's 
intent is to manage and resolve issues requiring corrective action at 
the lowest organizational level possible. This responsibility is 
generally within NORTHCOM's various directorates, component commands, 
or a Joint Task Force.[Footnote 76] Issues may be closed at the 
directorate level without external approval or oversight. Broader scope 
or more sensitive issues requiring the involvement of more than one 
directorate or subcommand go into the formal corrective action board 
process for review, tracking, and approval as necessary.[Footnote 77] 
This formal process includes two boards--the Corrective Action Board 
and the Executive Corrective Action Board--to review and resolve 
issues.[Footnote 78] Figure 7 illustrates the flow of NORTHCOM's 
lessons learned and corrective action process. 

Figure 7: NORTHCOM Lessons Learned and Corrective Action Process: 

[Refer to PDF for image: illustrated flow-chart] 

Observation recorded: 

Responsible office assigned; 

Valid finding: 
If no: 
* Responsible office does not concur; 
* Record closed; 
* End. 
If yes: continue. 

Lesson learned or issue: 
Lesson learned: 
* Record activated; 
* End. 
Issue: 
* Record activated; 
* Corrective action developed; 
* Corrective action implemented; 
* Verify corrective action in a subsequent exercise or operation; 
* Corrective action verified; 
If no, repeat sequence starting at Corrective action developed; 
If yes, Issue becomes lesson learned; 
End. 

Source: GAO analysis of NORTHCOM guidance. 

[End of figure] 

Gaps Remain with Collecting Observations From and Sharing Lessons 
Learned with Interagency Partners and States: 

JLLIS was intended to make lessons learned more widely available; 
however, we found that non-DOD exercise participants have limited 
access to this system, which presents challenges for NORTHCOM in 
collecting observations from and sharing lessons learned with 
interagency partners and states.[Footnote 79] Joint Staff and NORTHCOM 
officials told us that only a small number of interagency staff and no 
state representatives currently have access to JLLIS and therefore are 
not able to directly input exercise observations or review NORTHCOM's 
lessons learned through this system.[Footnote 80] DOD officials told us 
that access to JLLIS is restricted in order to safeguard sensitive 
information because of concerns that adversaries may be able to exploit 
any weaknesses contained in those records. On the other hand, NORTHCOM 
officials told us that most of the observations collected from 
exercises are unclassified, and we found this to be true for more than 
90 percent of the observations from NORTHCOM's six most recent 
exercises. The vast majority of the observations we reviewed are listed 
as unclassified and few records indicate they have a "For Official Use 
Only" restriction. For users to access the unclassified portion of the 
JLLIS they must have a DOD common access card, a card reader, and 
appropriate computer software prior to requesting access. Special 
security arrangements must be made for granting JLLIS access to 
civilian federal officials who do not possess common access cards but 
have other federally issued personal identification verification cards. 
These officials cannot access JLLIS because they also lack the 
appropriate software to operate the personal identification 
verification cards within the system. In an attempt to address this gap 
and gather more input from non-DOD exercise participants, in April 2008 
NORTHCOM officials placed a Word document template on its internet 
portal to provide a venue for non-DOD exercise participants to record 
exercise observations. Any observations provided in this format would 
be input by NORTHCOM Joint Training and Exercise Directorate officials 
into JLLIS. However, NORTHCOM's portal requires the same DOD-issued 
card to gain entry.[Footnote 81] In addition, NORTHCOM's lessons 
learned manager told us no one has submitted observations using the 
template since it was put on the portal. This may be because the 
command has not actively publicized how to access the template and 
underscored the value to the command of obtaining observations from 
interagency partners and states. In response to our inquiries in May 
2009, NORTHCOM's lessons learned manager told us that the command is in 
the process of adding a link to DHS's Homeland Security Information 
Network so that interagency partners and states will be able to submit 
lessons learned which can subsequently be transferred to JLLIS by 
NORTHCOM officials.[Footnote 82] 

In addition to collecting observations using JLLIS, NORTHCOM can obtain 
lessons learned from interagency partners and states during 
postexercise meetings. NORTHCOM conducts a review called a Hotwash 
within hours of completing the exercise so that exercise participants 
can discuss observations that significantly impacted their mission and 
recommend emergent themes for discussion during a subsequent review 
known as the facilitated after-action review. This review, generally 
held 7 days after the exercise is completed, provides an opportunity to 
present major issues to senior leaders and obtain the Commander's 
guidance for resolution. However, the extent to which interagency and 
state officials are attending and participating in NORTHCOM's 
postexercise meetings is unclear. Based on NORTHCOM's documentation, 
only two states (out of the last six major exercises) participated in a 
Facilitated After Action Review--California in Vigilant Shield 09 and 
Alaska in Ardent Sentry 07. Officials from three states in addition to 
California told us that they participated in the after-action meeting 
for the exercises they participated in, but they may have participated 
in the regional or national-level meeting rather than NORTHCOM's. 
[Footnote 83] Officials from two of the seven states we met with told 
us that they did not attend NORTHCOM's postexercise reviews for the 
exercises in which they participated at least partly due to staffing 
and budget limitations. 

NORTHCOM has also attempted to share lessons learned with other federal 
agencies and states by using FEMA's lessons learned sharing system. For 
example, NORTHCOM has posted six reports onto FEMA's lessons learned 
system, including four recent exercise reports and two reports from 
operations in 2008.[Footnote 84] However, with one exception, the 
documents that NORTHCOM has made available on this system (1) include 
only lists of observations and, in some cases, record-tracking numbers 
from JLLIS and previous lessons learned systems, and (2) lack detailed 
information on individual lessons learned and corrective actions. 
[Footnote 85] Joint Staff and NORTHCOM officials told us that they do 
not post detailed information on the unclassified Lessons Learned 
Information Sharing system Web site, because it is not adequately 
protected from the potential for unauthorized access to records. As a 
result, the security of the information cannot be assured. According to 
these officials, if an adversary nation or terrorist group gained 
access to this information, it may be possible for them to identify 
weaknesses in NORTHCOM's operations that can be exploited. In a recent 
exercise summary report NORTHCOM stated that it will post lessons 
learned, best practices, and reports that may benefit their non-DOD 
mission partners in FEMA's Lessons Learned Information Sharing system, 
which the report describes as a secure, restricted-access information 
system.[Footnote 86] Because security concerns are preventing NORTHCOM 
from openly sharing all its unclassified lessons learned with its 
interagency partners and the states, the information NORTHCOM does 
provide may be of limited value for helping its partners improve the 
nation's disaster responsiveness. Because NORTHCOM is not fully 
involving other federal agencies and states in its lessons learned 
process, it is missing opportunities to learn lessons from an exercise. 
For example, officials from two states did not provide NORTHCOM with 
lessons learned from exercises because they did not attend the 
command's post exercise reviews. As a result, NORTHCOM risks the 
reoccurrence of potential problems that were not identified in its 
process. 

NORTHCOM Faces Challenges Managing Corrective Actions: 

DOD and NORTHCOM guidance requires that issues requiring corrective 
actions be tracked and remain open until the solutions are completed 
and verified as effective--through training, operations, or exercises. 
[Footnote 87] We found that NORTHCOM directorates and subcommands are 
closing some issues prematurely, without confirming that corrective 
actions were made or verifying in a subsequent exercise or operation 
that the corrective action is effective. We reviewed unclassified 
records in JLLIS from NORTHCOM's previous six large-scale exercises and 
found at least 77 of the 375 records or about 20 percent required 
corrective actions but were either closed prior to completing the 
corrective action or closed without verifying the effectiveness of the 
corrective action.[Footnote 88] For example, an observation was made 
during Ardent Sentry 07 that NORTHCOM did not have a process for 
addressing a foreign nation's offer of military-to-military assistance 
in a major disaster. The issue was validated and the corrective action 
developed, but the issue was closed by the originating organization 
before the corrective action could be verified or reobserved in a 
subsequent exercise.[Footnote 89] The record was closed even though the 
Executive Corrective Action Board directed that it remain open until an 
exercise of suitable scope to require significant military support was 
developed.[Footnote 90] Another example of a record being closed 
without verification or re-observation is an observation made during 
Ardent Sentry 07 raising concerns that NORTHCOM personnel could arrive 
to assess a disaster site without alerting state officials they would 
be coming. As a result, NORTHCOM developed a new Command Assessment 
Element Concept of Execution[Footnote 91] in July 2007 to promote 
better command and control and situational awareness; however, the 
issue was closed before the procedure could be observed in a subsequent 
exercise or operation to verify effective resolution. 

These issues are likely being closed without verification or re- 
observation, because NORTHCOM Training and Exercise Directorate 
officials do not have oversight over the disposition of open issues 
that are resolved within directorates or are unable to give long- 
standing issues the sustained management attention needed to ensure 
resolution. NORTHCOM's lessons learned manager told us that the command 
does not have the staff necessary to oversee the actions on records 
handled within the other directorates. In addition, while the checkbox 
format in JLLIS makes it easy to see whether an issue is open, awaiting 
verification, or closed, entries made in JLLIS regarding corrective 
actions required, implementation date, and plan for verification are 
primarily in a narrative format, which may make the review and 
oversight process more time consuming. Without sufficient oversight, 
NORTHCOM cannot ensure that corrective actions are verified and 
reobserved in a subsequent exercise or operation before the issue is 
closed, so that the command knows the solution is effective. We 
recognize that such oversight should be addressed without significantly 
stressing NORTHCOM's staff. However, if NORTHCOM does not ensure that 
corrective actions are fully resolved, it increases the risk that these 
issues may occur again, possibly during crucial, real-world situations. 
This lack of oversight, coupled with the lack of a well-thought out and 
consistent process for including the states in assessing exercises as 
discussed earlier in this report, further limits the knowledge gained 
and value of the exercise for all participants. 

NORTHCOM Has Taken Steps to Integrate Its Exercises with the National 
Exercise Program, but Guidance Is Not Consistently Applied: 

NORTHCOM Has Participated in National Exercises and Taken Steps to 
Integrate with the NEP: 

Since the NEP Charter was approved in January 2007, NORTHCOM has 
participated in the major national exercise held under the NEP and 
taken steps to integrate its exercises into the national program. 
[Footnote 92] NEP guidance requires that heads of departments and 
agencies actively participate in tier I exercises and recommends 
participation in tier II exercises either through the National Exercise 
Simulation Center or as determined by agency leadership.[Footnote 93] 
Departments or agencies can participate in the NEP by combining an 
existing exercise with a NEP exercise, taking part in a tier II 
exercise sponsored by a different department or agency, or requesting 
to lead a tier II exercise to obtain greater interagency participation 
and support. DOD guidance requires that components participate in or 
lead planning efforts of NEP exercises as appropriate given the 
scenario or as tasked by the Assistant Secretary of Defense, Homeland 
Defense and America's Security Affairs or the Chairman of the Joint 
Chiefs of Staff.[Footnote 94] NORTHCOM's training guidance specifies 
the NEP exercises in which the command plans to participate during the 
following 2 fiscal years.[Footnote 95] NORTHCOM combined two of its 
large-scale exercises--Vigilant Shield 08 and Ardent Sentry 08--with 
major national exercises and has taken part in two additional exercises 
sponsored by other departments (see table 6). For example, National 
Level Exercise 1-08, a tier I exercise, and NORTHCOM's Vigilant Shield 
08 were conducted October 15 -20, 2007, in parallel with Top Officials 
4[Footnote 96] and several other exercises.[Footnote 97] These 
exercises were linked together by the use of common scenarios and 
objectives intended to test existing plans, policies, and procedures to 
identify planning and resource gaps and develop corrective actions to 
improve preparedness against a weapons of mass destruction attack. 

Table 6: NORTHCOM Participation in National Level and Tier II Exercises 
Since 2007: 

Exercise and tier: National Level Exercise 1-08 (I); 
Date: October 2007; 
Main scenario: Radiological dispersal device attack; 
Primary sponsor/lead: FEMA; 
Description of NORTHCOM participation: Linked Vigilant Shield 08 to 
exercise. 

Exercise and tier: National Level Exercise 2-08 (II); 
Date: May 2008; 
Main scenario: Hurricane, terrorist chemical attack, accidental 
chemical release; 
Primary sponsor/lead: FEMA; 
Description of NORTHCOM participation: Linked Ardent Sentry 08 to 
exercise. 

Exercise and tier: Diablo Bravo (II); 
Date: July 2008; 
Main scenario: Nuclear weapons incident; 
Primary sponsor/lead: Department of Energy/National Nuclear Security 
Administration; 
Description of NORTHCOM participation: Provided initial DOD response 
support for Department of Energy. 

Exercise and tier: Empire 2009 (II); 
Date: June 2009; 
Main scenario: Radiological dispersal device Attack; 
Primary sponsor/lead: National Nuclear Security Administration; 
Description of NORTHCOM participation: Provided Defense Coordinating 
Officer support. 

Source: GAO analysis of DHS and NORTHCOM documentation. 

Note: We limited the scope of our finding to major national exercises 
sponsored at the federal level and planned after National Level 
Exercise 1-08, the first major exercise conducted under the NEP. 

[End of table] 

NORTHCOM officials told us that they generally would like to 
participate in NEP exercises to achieve the benefits of exercising with 
interagency partners, but in some cases it is not beneficial to do so. 
For example, the officials told us NORTHCOM decided not to combine 
Ardent Sentry 09 with National Level Exercise 09--a tier 1 exercise 
scheduled for July 2009--because the objectives and scenarios for the 
exercises did not meet their training needs. Although NORTHCOM 
officials will conduct Ardent Sentry 09 separately, they are using the 
National Exercise Simulation Center--FEMA's newly established training 
and exercise facility--to provide a test run for the center's use in 
National Level Exercise 09.[Footnote 98] 

DOD and NORTHCOM have taken steps to integrate exercises with the 
National Exercise Program, including posting the command's exercises on 
DHS's National Exercise Schedule, successfully applying to lead a tier 
II exercise, and publishing guidance on integration with the NEP. The 
NEP Implementation Plan recommends that federal departments and 
agencies post exercises on the NEP's National Exercise Schedule so that 
exercises and planning meetings can be synchronized across the federal 
government.[Footnote 99] NORTHCOM has posted its annual Ardent Sentry 
and Vigilant Shield exercises for the first 4 of 5 fiscal years on the 
national schedule, while FEMA's National Exercise Division has posted 
exercises for the first 3 fiscal years. As of June 2009, neither the 
Joint Staff nor any other combatant commands have posted exercises on 
the national schedule. In addition, NORTHCOM recently requested and was 
granted approval to lead Vigilant Shield 10 as a tier II exercise 
scheduled for November 2009. Vigilant Shield 10 should have greater 
interagency participation than it would have received as a tier 3 
exercise, since federal departments and agencies will be required, at a 
minimum, to participate in the National Exercise Simulation Center. As 
of May 2009, the participants of Vigilant Shield 10 include the DHS and 
the Departments of Justice, Energy, Transportation, Health and Human 
Services, and Veteran Affairs; the Joint Chiefs of Staff and U.S. Joint 
Forces Command; and other government and nongovernment organizations. 
[Footnote 100] This exercise will be the first time that NORTHCOM will 
share planning responsibilities with FEMA's National Exercise Division. 
This exercise will also be linked to a Canadian government an exercise 
to demonstrate its readiness for the 2010 Olympics in Vancouver. 

NORTHCOM Has Not Consistently Applied NEP Guidance: 

NEP guidance includes policies and tools for the design, planning, 
conduct, and evaluation of exercises--known as the Homeland Security 
Exercise and Evaluation Program, which creates a common exercise policy 
and consistent terminology for exercise planners and serves as the 
foundation of NEP exercises.[Footnote 101] FEMA requires that entities, 
such as state and local governments, receiving homeland security grant 
funding for their exercises adhere to specific Homeland Security 
Exercise Evaluation Program guidance for exercise program management, 
design, conduct, evaluation, and improvement planning.[Footnote 102] We 
reviewed key program documents, such as the Implementation Plan, and 
found that this guidance is unclear about the extent to which federal 
agencies should use the Homeland Security Exercise Evaluation Program. 
For example, the Implementation Plan states that the NEP does not 
displace a preexisting exercise program, and none of the NEP guidance 
requires that federal agencies use the Homeland Security Exercise 
Evaluation Program. However, the Implementation Plan states that the 
Homeland Security Exercise Evaluation Program will serve as the 
doctrinal foundation for NEP exercises. 

FEMA officials told us that federal agencies should use this program 
when participating in tier I and tier II exercises so that the various 
exercise participants have consistency when planning, conducting, and 
assessing exercises. We found that NORTHCOM generally has used DOD's 
Joint Training System guidance for planning NEP exercises, defining 
capabilities, and reporting exercise results.[Footnote 103] NORTHCOM 
officials told us that the Joint Training System is consistent with the 
NEP and served, in part, as the basis for the Homeland Security 
Exercise and Evaluation Program.[Footnote 104] We found that these sets 
of guidance have similar processes but use different methods for 
defining the tasks and capabilities that are performed and validated in 
an exercise. The primary differences between these sets of guidance are 
that (1) DOD's task list, which serves as the basis for its exercises, 
includes tasks that are specific to military missions, such as troop 
movements and sealifts; (2) DHS guidance provides more detailed 
criteria for the postexercise documentation, such as content and 
format; and (3) DHS's planning cycle is generally shorter--9 to 15 
months versus 12 to 18 months for DOD. (See table 7.) See appendix II 
for a more detailed comparison. 

Table 7: Comparison of Areas of Application between DOD's Joint 
Training System and DHS's Homeland Security Exercise Evaluation 
Program: 

Areas of application: Methods for defining tasks and capabilities; 
Joint Training System: 
* Derived from Joint Mission Essential Task Lists; 
* Users may also develop tasks using the Universal Joint Task List; 
Homeland Security Exercise Evaluation Program: 
* Derived from Target Capabilities List or Universal Task List; 
* Users may also develop tasks using the Universal Task List. 

Areas of application: Reporting requirements; 
Joint Training System: 
* Issue a Quick Look Report and an Executive Summary Report; 
* No detailed guidelines on content for either report; 
* No detailed guidelines on content found in the required Quick Look or 
Exercise Summary Report; 
Homeland Security Exercise Evaluation Program: 
* Issue a Quick Look Report and an After Action Report/Improvement 
Plan; 
* Provides detailed guidelines on content for both reports, including 
templates; 
* Provides detailed guidelines on content for both the required Quick 
Look Report and the required After Action Report/Improvement Plan. 

Areas of application: Planning cycle; 
Joint Training System: 
* Uses the Joint Event Life Cycle, has five stages: design, planning, 
preparation, execution, and evaluation; 
* Intended to be flexible and can be modified to any type of event. Can 
span a period of 12 to 18 months for exercises; 
Homeland Security Exercise Evaluation Program: 
* Uses the Homeland Security Exercise Evaluation System's Exercise 
Cycle, employs five phases: foundation, design and development, 
conduct, evaluation, and improvement planning; 
* Intended to be adaptable to any type of exercise. Can span a period 
of 9 to 15 months. 

Areas of application: Lessons learned/corrective actions systems; 
Joint Training System: 
* Joint Lessons Learned Information System manages and tracks Lessons 
Learned and corrective actions; 
Homeland Security Exercise Evaluation Program: 
* FEMA's Lessons Learned Information Sharing System manages and 
disseminates lessons learned; 
* DHS's Corrective Action Program system manages corrective actions. 

Source: GAO analysis of DOD and DHS documentation. 

[End of table] 

According to both the Homeland Security Exercise and Evaluation Program 
and Joint Training System guidance, it is important to link tasks and 
capabilities with exercise objectives to ensure that participants 
exercise or train as they would perform in a real-world event. The 
Homeland Security Exercise and Evaluation Program recommends using 
DHS's Target Capabilities List or the Universal Task List to formulate 
the tasks and capabilities that underlie the objectives for an 
exercise.[Footnote 105] These lists describe the capabilities 
government entities need and tasks they are expected to perform to 
prevent, protect against, respond to, and recover from incidents of 
national significance. In contrast, NORTHCOM derives its tasks and 
capabilities from the Universal Joint Task List to formulate Joint 
Mission Essential Tasks. According to NORTHCOM guidance, the command is 
required to include in its exercises the Joint Mission Essential Tasks 
associated with its Joint Training Plan, which is updated annually. 
[Footnote 106] These tasks are identified by joint force commanders as 
most essential to their assigned or anticipated missions with priority 
given to their wartime missions and describe their priority wartime 
requirements. We found that DOD's operating instruction for 
participation in the NEP does not provide guidance on how DOD 
components should incorporate tasks and capabilities derived from 
sources recommended by the Homeland Security Exercise and Evaluation 
Program when participating in NEP exercises.[Footnote 107] The primary 
differences between DHS' and DOD's lists are that DOD's task lists 
generally incorporate more descriptive metrics and criteria to assess 
performance and include tasks that are specific to military missions, 
such as troop movements and sealifts. In some cases, state National 
Guards officials have had to translate DOD task lists into DHS tasks 
lists when working with their civilian partners and vice versa. 

We also found that neither DOD's nor NORTHCOM's guidance for developing 
postexercise reports includes the same degree of specificity 
recommended in the Homeland Security Exercise Evaluation Program. For 
example, both sets of exercise guidance require postexercise reports; 
however, the Homeland Security Exercise Evaluation Program provides 
templates and guidance for these documents, including requiring an 
improvement plan to clearly outline the corrective actions needed, 
which are not included in DOD's or NORTHCOM's guidance. In addition, 
NORTHCOM's exercise summary reports for National Level Exercise 1-08 
and 2-08 did not contain all information recommended by the Homeland 
Security Exercise Evaluation Program. For example, NORTHCOM did not 
include the recommended analyses regarding the capabilities and tasks 
tied to the exercises' objectives. 

As stated above, we reviewed NEP guidance such as the Implementation 
Plan and found it does not clearly state the extent to which federal 
agencies are required to follow the Homeland Security Exercise and 
Evaluation Program. As a result of this unclear guidance, we found that 
agency officials have varying interpretations of the requirements. For 
example, a DOD and a Joint Staff official told us that NEP guidance 
does not require agencies to use the Homeland Security Exercise 
Evaluation Program even for NEP exercises. Therefore, NORTHCOM uses the 
Joint Training System rather than the Homeland Security Exercise and 
Evaluation Program as the basis for planning, conducting, and assessing 
exercises. However, officials from FEMA's National Exercise Division 
told us that all participating agencies should use the Homeland 
Security Exercise and Evaluation Program guidance for tier I and tier 
II NEP exercises.[Footnote 108] FEMA officials stated that federal 
departments and agencies should be held accountable for meeting key 
requirements, but that FEMA's authority is limited to guiding, 
supporting, and coordinating with, but not directing other federal 
departments and agencies to comply with guidance. As we have previously 
reported, we believe that FEMA's expanded leadership role under the 
Post-Katrina Act provides FEMA opportunities to instill a shared sense 
of responsibility and accountability on the part of all agencies. 
[Footnote 109] 

Neither DOD nor NORTHCOM guidance specifically addresses the extent to 
which DHS's Homeland Security Exercise and Evaluation Program planning 
and documentation requirements should be followed.[Footnote 110] We 
recognize that NORTHCOM and DOD must meet their own mission and 
exercise requirements and the Joint Training System may be best suited 
for NORTHCOM's exercises; however, all of the states we visited use 
Homeland Security Exercise and Evaluation Program guidance. We found 
that having differing sets of guidance, such as DOD's and DHS' 
capabilities task lists and postexercise documentation requirements, 
makes exercises more difficult and potentially limits the benefits for 
participating states. For example, officials from three states we 
visited told us that using NORTHCOM's exercise planning and reporting 
requirements rather than Homeland Security Exercise Evaluation Program 
guidance has made the processes more difficult. Further, the Defense 
Science Board found that inconsistent approaches to the development and 
content of postexercise documentation may affect the ability of 
organizations to fully learn lessons identified in exercises.[Footnote 
111] We also reported that when other federal entities carry out 
processes that do not specifically follow the Homeland Security 
Exercise and Evaluation Program, FEMA managers do not have the 
necessary data to measure progress, identify gaps in preparedness, and 
track lessons learned--key objectives of the NEP.[Footnote 112] We 
believe that achieving national preparedness requires a whole-of- 
government approach and is a shared responsibility among federal, 
state, local, and tribal governments and organizations and an 
integration of their various standards, policies, and procedures into 
the national system.[Footnote 113] 

Conclusions: 

There is an increasing realization within the federal government that 
an effective, seamless national response to an incident requires a 
strong partnership among federal, state, and local governments and 
organizations, including integrated planning, training, and the 
exercise of those plans. For DOD, the effective execution of civil 
support, especially amid simultaneous, multijurisdictional disasters, 
requires ever-closer working relationships with other departments and 
agencies and at all levels of government. NORTHCOM's use of DOD's Joint 
Training System has provided a robust process for planning and 
conducting exercises to improve preparedness to achieve its homeland 
defense and civil support missions, and its efforts to involve its 
interagency partners and the states in exercises have helped to reduce 
uncertainty about the process for responding to an incident. However, 
without a consistent record of what has occurred during an exercise 
that is accessible by all exercise participants, including those from 
other federal agencies and states, NORTHCOM cannot ensure that it has 
met internal standards, trained to key focus areas, or compared the 
goals and results of exercises over time. Further, a key element to 
developing effective working relationships with all states is a 
consistent process for including states in planning and executing 
NORTHCOM's exercises that incorporates state-specific knowledge and 
information. By coordinating consistently with organizations, like FEMA 
and NGB, that have knowledge and experience dealing with states, 
NORTHCOM can improve the value and effectiveness of exercises for all 
of the participants involved. 

Exercises provide an opportunity to enhance preparedness by collecting, 
developing, implementing, and disseminating lessons learned and 
verifying corrective action taken to resolve previously identified 
issues. NORTHCOM's clear procedure for capturing observations in JLLIS 
and identifying issues needing corrective action has helped to improve 
its capabilities to complete its missions. However, by not providing 
federal agencies and states greater access to its lessons learned 
process, NORTHCOM will lose opportunities to learn valuable lessons 
from an exercise, particularly observations from the states that could 
enhance coordination and build more effective interagency 
relationships. Further, the risk that issues may reoccur will be 
increased, particularly when interagency partners are not aware of key 
issues or concerns that might impede the government's overall 
responsiveness to a natural or man-made disaster. In addition, when 
corrective actions remain open until fully implemented and verified in 
a subsequent exercise, NORTHCOM will have greater assurance that issues 
raised during exercises are being adequately addressed and the 
corrections are in fact solving the problems identified. 

NEP policies and tools for the design, planning, conduct, and 
evaluation of exercises are intended to create a common exercise policy 
and consistent terminology for exercise planners across all levels of 
government to improve the federal government's ability to evaluate 
national preparedness. The steps DOD and NORTHCOM have taken to 
integrate exercises with the NEP have helped DHS to prioritize and 
coordinate federal exercise activities and enhance the federal 
government's ability to respond to an incident. We recognize that 
NORTHCOM and DOD must meet their own mission and exercise requirements 
and the Joint Training System may be best suited to meet the high 
standards required for NORTHCOM's exercises. However, achieving 
national preparedness requires shared responsibility among federal, 
state, and local governments and organizations and an integration of 
their various standards, policies, and procedures into the national 
system. We also recognize that the NEP continues to evolve and become 
more useful to federal and state partners. However, in the absence of 
clear guidance from DHS on the extent to which agencies should use 
Homeland Security Exercise Evaluation Program planning and 
documentation guidance, DOD should ensure that its components clearly 
understand when the use of this guidance is appropriate so that both 
DOD and its exercise partners, such as other federal agencies and 
states, derive the most benefits from exercises. This, in turn, 
contributes to the ultimate success of a whole-of-government approach 
to national preparedness. 

Recommendations for Executive Action: 

To improve NORTHCOM's consistency with exercise documentation, we 
recommend that the Secretary of Defense direct NORTHCOM's Commander to 
develop guidance with specific criteria for postexercise documentation, 
particularly the Exercise Summary Report as the official exercise 
record, including the content and format to be included in such reports 
that will allow the results and lessons learned of exercises to be 
easily reviewed and compared. 

To improve NORTHCOM's involvement of interagency partners and states in 
its exercises, we recommend that: 

* the Secretary of Defense, in consultation with the Chairman of the 
Joint Chiefs of Staff, the Commander, U.S. Northern Command, and other 
relevant combatant commanders, coordinate with the Department of 
Homeland Security and Federal Emergency Management Agency to develop 
guidance and procedures for consistently involving state officials in 
planning, executing, and assessing exercises that incorporate relevant 
state-specific information, and: 

* the Secretary of Defense direct NORTHCOM's Commander to develop a 
training plan for NORTHCOM headquarters staff on state emergency 
management structures and relevant issues related to working with 
civilian state and local emergency management officials. 

To improve NORTHCOM's involvement of interagency partners and states in 
its lessons learned and corrective action process and its management of 
corrective actions, we recommend that the Secretary of Defense direct: 

* NORTHCOM's Commander to establish and publicize valid and easily 
accessible procedures for non-DOD exercise participants to submit 
observations relevant to NORTHCOM, such as placing a template on 
NORTHCOM's publicly accessible Web site or DHS's Homeland Security 
Information Network, so that NORTHCOM officials have a clear, secure 
avenue to obtain observations and assess potential lessons that 
originate with its exercise partners; 

* the Chairman of the Joint Chiefs of Staff, in consultation and 
coordination with DHS, to either resolve information assurance issues 
so that the combatant commands, including NORTHCOM, can post Exercise 
Summary Reports with lessons learned and observations from NEP 
exercises on DHS's Lessons Learned Information Sharing system to make 
them easily accessible to interagency partners and states or establish 
an alternative method to systematically collect and share lessons 
learned; and: 

* the Chairman of the Joint Chiefs of Staff to revise the joint lessons 
learned operating instruction to include procedures to ensure that 
appropriate corrective actions are implemented and verified in a 
subsequent exercise or operation before being closed and that the 
reasons for closure are documented. Possible procedures might be adding 
a verification checkbox on JLLIS's issue management page or requiring 
that the directorates and subordinate commands within the combatant 
commands provide a status report when a correction is implemented and 
reobserved or closed for reasons other than re-observation. 

To improve NORTHCOM's ability to work with interagency partners on 
major national exercises and further achieve the objectives of the NEP, 
we recommend that the Secretary of Defense revise the instruction on 
DOD participation in the NEP and/or direct the Chairman of the Joint 
Chiefs of Staff to revise the operating instruction regarding DOD 
participation in the NEP to provide the general conditions under which 
the combatant commands are expected to follow the Homeland Security 
Exercise and Evaluation Program planning and documentation requirements 
or the DOD's Joint Training System should be modified for those civil 
support exercises. 

Agency Comments and Our Evaluation: 

In comments on a draft of this report, DOD generally agreed with the 
intent of our recommendations and discussed steps it is taking or plans 
to take to address these recommendations. DOD also provided technical 
comments, which we have incorporated into the report where appropriate. 
DHS also reviewed a draft of this report and provided technical 
comments, which we have incorporated where appropriate. 

In response to our recommendation that NORTHCOM develop guidance with 
specific criteria for postexercise documentation to allow the results 
and lessons learned of exercises to be reviewed and compared, DOD 
agreed that such information should be provided in a standardized 
format that can be easily accessed and understood by authorized 
organizations which might benefit from such knowledge. DOD cautioned 
that any actions in response to this recommendation must accommodate 
constraints regarding classified information. We agree that properly 
securing classified information is a critical responsibility and 
believe this can easily be accomplished without undermining the intent 
of the recommendation, which is to improve the consistency and 
completeness of formal exercise documentation and thereby its overall 
value. 

In response to our recommendation that DOD coordinate with DHS and FEMA 
to develop guidance and procedures for consistently involving state 
officials in planning, executing, and assessing exercises that 
incorporate relevant state-specific information, DOD agreed that better 
coordination for interfacing with state officials can be achieved. DOD 
also pointed out that NORTHCOM continues to expand its efforts to work 
through defense coordinating officers, existing state National Guard 
relationships, and FEMA regional headquarters partners to ensure that 
states are able to benefit from participation in DOD-sponsored 
exercises. However, DOD also said that while NORTHCOM has continuously 
engaged and encouraged state participation in NORTHCOM-sponsored 
exercises, the primary audience for such training is and must remain 
NORTHCOM. DOD also suggested that our recommendation has applicability 
to other federal interagency partners and that the issue should be 
addressed to the Exercise and Evaluation Sub-Interagency Planning 
Committee as a revision to the National Exercise Program Implementation 
Plan. As our report indicates, we agree that NORTHCOM has sought to 
engage and involve the states in its comprehensive exercise program. 
NORTHCOM plans for and conducts major exercises both inside and outside 
the construct of the National Exercise Program. Particularly for 
NORTHCOM-sponsored exercises focused on the command's civil support 
mission, the effective involvement of and interaction with state and 
other federal partners is a critical component of improving and 
maintaining NORTHCOM's preparedness. For NORTHCOM's participation in 
national-level exercises, the preparedness goals and objectives of all 
participants are equally important. We believe that in developing 
procedures to improve coordination with the states, DOD can (1) avoid 
situations where exercises meant to improve preparedness are not fully 
coordinated with the necessary partners; (2) capitalize on the 
structures and organizations it already has in place, such as the 
defense coordinating officers and relationships with state National 
Guard headquarters; and (3) coordinate with DHS and FEMA to improve the 
military-civilian interface. With regard to the latter, the Exercise 
and Evaluation Sub-Interagency Planning Committee may indeed be one of 
the venues at which DOD can effectively coordinate with its interagency 
partners. 

With respect to our recommendation that NORTHCOM develop a training 
plan for NORTHCOM headquarters staff on state emergency management 
structures and relevant issues related to working with civilian state 
and local emergency management officials, DOD agreed and noted that 
headquarters training is required for all newly assigned NORTHCOM 
staff. Further, DOD noted that NORTHCOM sponsors three versions of its 
defense support of civil authorities seminar that are targeted to staff 
at different seniority levels. We agree that NORTHCOM has continued to 
improve the level of awareness and training it provides staff on the 
complexities of providing defense support to civilian authorities in 
the United States. However, this does not fully address our 
recommendation. While training on the general procedures of the 
national response framework, the nature of state-federal government 
relations, and DOD's proper role in providing military support to civil 
authorities is invaluable for NORTHCOM staff, we continue to believe 
that this should be supplemented by the kinds of state-specific 
information that would provide both exercise officials and all other 
staff with an understanding of the key differences between states. 
These differences are possibly as numerous as the number of states and 
play a role in all routine interactions between the individual states 
and DOD officials as well as for effective coordination for exercise 
planning and coordination during a natural disaster or some other no- 
notice incident requiring defense support to civil authorities. 

DOD agreed with our recommendation that NORTHCOM establish and 
publicize valid and easily accessible procedures for non-DOD exercise 
participants to submit observations relevant to NORTHCOM, such as 
placing a template on NORTHCOM's Web site or DHS' Homeland Security 
Information Network, so that NORTHCOM officials have a clear, secure 
avenue to obtain observations and assess potential lessons that 
originate with its exercise partners. DOD indicated that collecting 
exercise information from all perspectives would provide additional 
opportunities to improve NORTHCOM's ability to accomplish its mission 
tasks. 

DOD also agreed with our recommendation that it work with DHS to either 
resolve information assurance issues so that NORTHCOM can post Exercise 
Summary Reports with lessons learned on DHS' Lessons Learned 
Information Sharing system or establish an alternative method to 
systematically collect and share lessons learned. DOD cautioned that 
while wide dissemination of information approved for release would be 
of great benefit to homeland security entities it continues to adhere 
to the Joint Training System and cannot mandate that DHS alter its 
Lessons Learned Information System to make accommodations. DOD also 
noted that it has procedures in place to allow specifically cleared 
individuals from outside DOD access to information contained in 
Exercise Summary Reports. We agree that DOD cannot mandate alterations 
to the Lessons Learned Information System. We also agree that the Joint 
Training System should remain the chief guidance for the conduct of DOD 
exercises. However, we continue to believe that in working with DHS on 
the proper level and mode of information sharing, DOD may be able to 
improve the dissemination of relevant exercise-related information to 
all appropriate officials. 

DOD agreed with our recommendation that the Chairman of the Joint 
Chiefs of Staff revise the joint lessons learned operating instruction 
to include procedures to ensure that appropriate corrective actions are 
implemented and verified in a subsequent exercise or operation before 
being closed and that the reasons for closure are documented. DOD 
indicated that the Chairman of the Joint Chiefs of Staff Instruction 
3150.25D could be expanded to provide more guidance and the Joint 
Lessons Learned Information System could be updated to provide a 
technological solution to address the issue once the process and 
procedures are in place. DOD also indicated that the process of 
verifying corrective action and closing issues will become more 
effective with the modifications it outlined in response to the 
recommendation. 

In response to our recommendation that DOD revise guidance on DOD 
participation in the National Exercise Program to provide the general 
conditions under which the combatant commands are expected to follow 
the Homeland Security Exercise and Evaluation Program planning and 
documentation requirements or the DOD's Joint Training System should be 
modified for those civil support exercises, DOD recognized the 
importance of ensuring effective interaction with interagency partners 
for homeland security-related exercises. However, DOD noted that the 
National Exercise Program Implementation Plan contains language placed 
there at DOD's insistence that establishes a process to resolve 
doctrinal differences during exercise planning. DOD indicated that 
together with provisions in the implementation plan establishing the 
administration, scope, and hierarchy of multiagency homeland security 
exercises and the 5-year National Exercise Program schedule, this 
should address our recommendation. DOD further noted that The Joint 
Training System remains the Secretary of Defense's guidance on DOD 
exercises and that the National Exercise Program Implementation Plan 
stipulates that individual department or agency exercise programs 
should not be replaced. We agree that the Joint Training System is and 
should be DOD's primary guidance for ensuring that DOD components train 
and exercise according to standards. However, because interagency 
exercises are becoming an ever larger part of the national preparedness 
effort, and to the extent that effective exercise planning is bolstered 
by common procedures, our recommendation is intended to help DOD 
clarify for its components the circumstances under which the specific 
planning and documentation requirements for the Homeland Security 
Exercise and Evaluation Program can be followed without detriment to 
DOD's high training and exercise standards or compromise of the Joint 
Training System. 

DOD's written comments are reprinted in appendix III. 

We are sending copies of this report to the Secretary of Defense, 
Secretary of Homeland Security, and other interested parties. In 
addition, the report will be available at no charge on GAO's Web site 
at [hyperlink, http://www.gao.gov]. 

If you or your staff have any questions about this report, please 
contact me at (202) 512-5431 or dagostinod@gao.gov. Contacts points for 
our Office of Congressional Relations and Public Affairs may be found 
on the last page of this report. Key contributors to this report are 
listed in appendix IV. 

Signed by: 

Davi M. D'Agostino: 
Director, Defense Capabilities and Management: 

List of Requesters: 

The Honorable Joseph I. Lieberman: 
Chairman: 
The Honorable Susan M. Collins: 
Ranking Member: 
Committee on Homeland Security and Governmental Affairs: 
United States Senate: 

The Honorable Bennie G. Thompson: 
Chairman: 
Committee on Homeland Security: 
House of Representatives: 

The Honorable Kit Bond: 
United States Senate: 

The Honorable Patrick Leahy: 
United States Senate: 

The Honorable Gene Taylor: 
House of Representatives: 

[End of section] 

Appendix I: Scope and Methodology: 

In conducting this review, we generally focused our scope on U.S. 
Northern Command's (NORTHCOM) large-scale exercises[Footnote 114] 
conducted since Hurricane Katrina made landfall in August 2005. To 
determine the extent to which NORTHCOM's exercise program is consistent 
with Department of Defense (DOD) training and exercise requirements and 
includes relevant exercise partners, we evaluated NORTHCOM's compliance 
with exercise reporting and documentation requirements established in 
DOD and command guidance. We reviewed available guidance to determine 
requirements for timing, format, and content. We also compared these 
requirements with guidance contained in the Department of Homeland 
Security's (DHS) Homeland Security Exercise and Evaluation Program 
documentation. We reviewed exercise documentation for all large-scale 
exercises the command performed since it was established in 2002 to 
determine the extent to which the command complied with the guidance. 

We also performed an assessment of the experiences and level of 
participation from some interagency organizations and states in 
NORTHCOM's large-scale exercises. We initially met with Nevada 
officials who participated in a NORTHCOM exercise prior to Hurricane 
Katrina--Determined Promise 03--to provide context to the extent that 
changes may have been made to NORTHCOM's exercise program and help 
develop our state selection methodology. We selected a nongeneralizable 
sample of six states based on the extent to which they have 
participated in major NORTHCOM exercise since Hurricane Katrina and the 
varying scenarios of the exercises. The states we selected played a 
major role in NORTHCOM exercises by having a portion of the exercise 
conducted in their state and having various state agencies and 
officials participate. States we selected include Arizona, California, 
Michigan, Oregon, Rhode Island, and Washington. We met with 
representatives from each state's emergency management organization and 
state national guard. Because of the methodology selected, the 
resulting data and information from these state visits could not be 
projected to make assumptions about the rest of the states and what 
they may experience exercising with NORTHCOM. We also met with 
officials from three Federal Emergency Management Agency (FEMA) 
regional offices that had exercised with NORTHCOM in three of the last 
six large-scale exercises. We also interviewed officials from the 
Office of the Assistant Secretary of Defense for Homeland Security and 
America's Security Affairs, Joint Staff, and NORTHCOM with knowledge of 
and experience with NORTHCOM's training and exercise program. 

To determine the extent to which NORTHCOM is using lessons learned 
during exercises to improve mission preparedness, we reviewed DOD, 
NORTHCOM, and DHS National Exercise Program (NEP) guidance for 
recording, tracking, and managing lessons learned and assessed 
NORTHCOM's management of exercise observations and issues identified in 
several of NORTHCOM's large-scale exercises since Hurricane Katrina in 
2005. We interviewed NORTHCOM, Joint Staff, and FEMA officials 
regarding the various lessons learned management systems, and how 
interagency and state access to these systems can be accomplished. We 
also spoke with an official in the General Services Administration 
regarding the types of federal personal identification verification 
cards used by DOD and other federal departments and agencies to access 
government computer systems. In reviewing the management of NORTHCOM's 
lessons learned program we identified and reviewed all unclassified 
exercise observations from its last six large-scale exercises that had 
been activated in NORTHCOM's area of the Joint Lessons Learned 
Information System (JLLIS). Our review of the records in JLLIS entailed 
determining each record's status (open or closed), its type (issue or 
lesson learned), and each record's disposition after NORTHCOM staff 
have acted on these records to respond to the issues or lessons learned 
documented. Based on our review, we generally placed these records into 
one of several categories: open; closed, nonconcur; issue closed with 
reobservation; issue closed with no reobservation; and lesson learned. 
In addition we reviewed several records that had been merged with other 
original records because each related to the same issue; however, the 
original record for that issue was not part of our universe. Therefore, 
without reviewing the lead record the merged records lacked sufficient 
information regarding their disposition and that condition became 
another category. 

Finally, to determine the extent to which NORTHCOM is integrating its 
training and exercises with the NEP we reviewed DOD, NORTHCOM, and 
Department of Homeland Security guidance to identify any differences in 
exercise planning and documentation between DOD's guidance and that for 
the NEP. We used that analysis to determine under what conditions 
NORTHCOM should apply standards related to the NEP, and how DOD and its 
subordinate commands should participate in the NEP tier I or II 
exercises. We reviewed NORTHCOM documentation from two major national 
exercises conducted during fiscal year 2008 to determine the extent to 
which NORTHCOM employed the guidance from the Homeland Security 
Exercise Evaluation Program. We determined that national exercises that 
are operations-based exercises in that they involved the deployment of 
personnel would be the best candidates for evaluating NORTHCOM's 
participation in such exercises. We also interviewed state emergency 
management and National Guard officials from six states that have 
exercised with NORTHCOM since 2005, to understand the extent to which 
NORTHCOM is integrating its exercise planning and conduct with the 
interagency as well as various state governments. 

In addressing our objectives, we reviewed plans and related documents, 
obtained information, and interviewed officials at the following 
locations: 

* NORTHCOM Headquarters, Peterson Air Force Base, Colorado Springs, 
Colorado: 

* Joint Forces Command, Joint Warfighting Center, Suffolk, Virginia: 

* The Office of the Secretary of Defense, Washington, D.C. 

* The Joint Staff, Washington, D.C. 

* U.S. Army North, Fort Sam Houston, San Antonio, Texas: 

* National Guard Bureau, Arlington, Virginia: 

* Department of Homeland Security, Washington, D.C. 

* U.S. Coast Guard, Atlantic Area, Portsmouth, VA: 

* FEMA's National Preparedness Directorate, Washington, D.C. 

* FEMA Region 1, Boston, Massachusetts: 

* FEMA Region 9, Oakland, California: 

* FEMA Region 10, Bothell, Washington: 

* General Services Administration, Washington, D.C. 

* Arizona Division of Emergency Management, Phoenix, Arizona: 

* Arizona National Guard, Joint Force Headquarters, Phoenix, Arizona: 

* California Emergency Management Agency, Sacramento, California: 

* California National Guard, Joint Force Headquarters, Sacramento, 
California: 

* Michigan State Police, Emergency Management and Homeland Security 
Division, Lansing, Michigan: 

* Michigan National Guard, Lansing, Michigan: 

* Nevada State Division of Emergency Management, Carson City, Nevada: 

* Nevada National Guard, Joint Force Headquarters, Carson City, Nevada: 

* Rhode Island Emergency Management Agency, Cranston, Rhode Island: 

* Rhode Island National Guard, Joint Force Headquarters, Cranston, 
Rhode Island: 

* Oregon Military Department, Office of Emergency Management, Salem, 
Oregon: 

* Oregon Military Department, National Guard Joint Force Headquarters, 
Salem, Oregon: 

* Washington Military Department, Emergency Management Division, Camp 
Murray, Washington: 

* Washington National Guard, Joint Force Headquarters, Camp Murray, 
Washington: 

We conducted our review from June 2008 to September 2009 in accordance 
with generally accepted government auditing standards. Those standards 
require that we plan and perform the audit to obtain sufficient, 
appropriate evidence to provide a reasonable basis for our findings and 
conclusions based on our audit objectives. We believe that the evidence 
obtained provides a reasonable basis for our findings and conclusions 
based on our audit objectives. 

[End of section] 

Appendix II: Comparison of DOD's Joint Training System with DHS's 
Homeland Security Exercise and Evaluation Program: 

We reviewed the time lines and milestones for developing exercises 
found in the Department of Defense's Joint Training System and U.S. 
Northern Command's (NORTHCOM) implementing guidance and compared them 
with the Department of Homeland Security's Homeland Security Exercise 
and Evaluation Program guidance to determine the similarities and 
differences between them. We used the guidance associated with 
operation-based exercises rather than discussion-based exercises to 
present the full spectrum of Homeland Security Exercise and Evaluation 
Program processes and planning events. 

Table 8: Comparison of DOD Joint Training System and Homeland Security 
Exercise and Evaluation Program: 

DOD Joint Training System and NORTHCOM Exercise Program: Joint Event 
Life Cycle; NORTHCOM uses the Joint Events Life Cycle to design, plan, 
and execute exercises; 
* Consists of 5 stages: design, planning, preparation, execution, and 
evaluation; 
* Is a flexible guide that can be modified to apply to various types of 
events; 
* Can span a period of 12 to 18 months for exercise development; 
Homeland Security Exercise and Evaluation Program Guidance: Homeland 
Security Exercise and Evaluation Program Exercise Cycle; Homeland 
Security Exercise and Evaluation Program policy and guidance is the 
doctrine for the development of National Exercise Program exercises; 
* Employs 5 phases, which are collectively known as the exercise cycle: 
foundation, design and development, conduct, evaluation, and 
improvement planning; 
* Planning time line for the national level exercise will begin 
approximately 15 months prior to exercise conduct. 

DOD Joint Training System and NORTHCOM Exercise Program: Concept 
Working Group; 
Time frame: 12 to 15 months prior to start of exercise; 
Purpose: 
* Commences start of design stage; 
* Conduct initial planning to develop the candidate scenario setting, 
determine funding constraints, and define training requirements; 
* Result in the validation of the exercise purpose, exercise 
objectives, identification of the training audience, intelligence 
concepts, draft exercise event flow, planning milestone, and draft 
initial planning conference agenda; 
Homeland Security Exercise and Evaluation Program Guidance: Concept and 
Objective Meeting; 
Time frame: Minimum 6 months prior to start of exercise, lasts 2 to 4 
hours; 
Purpose: 
* Marks the start of the planning process and can be conducted in 
conjunction with the initial planning conference; 
* Identify the type, scope, objectives, and purpose of the exercise, as 
well as help planners identify, among other things, exercise planning 
team members; 
* Reach agreement regarding exercise type, scenario, capabilities, 
tasks, objectives, the target exercise time frame, and the date and 
time of the next planning conference. 

DOD Joint Training System and NORTHCOM Exercise Program: Initial 
Planning Conference; 
Time frame: 10 to 12 months prior to start of exercise, lasts 3 to 4 
days; 
Purpose: 
* First major planning conference in the Joint Events Life Cycle 
process; 
* Identify levels of participation, develop deployment and logistic 
requirements, validate the exercise milestones, develop outline of the 
exercise plan and other planning documents, determine modeling and 
simulation requirements, and develop the mid-planning conference 
agenda; 
Homeland Security Exercise and Evaluation Program Guidance: Initial 
Planning Conference; 
Time frame: Minimum 6 months prior to start of exercise, lasts 3 to 6 
hours; 
Purpose: 
* Begins planning process; 
* Determine exercise scope by gathering input from the exercise 
planning team, design requirements and conditions (such as assumptions 
and artificialities), objectives, extent of play, and scenario 
variables (such as time, location, hazard selection); 
* Used to develop exercise documentation by obtaining the planning 
team's input on exercise location, schedule, duration, and other 
relevant details. 

DOD Joint Training System and NORTHCOM Exercise Program: Mid-Planning 
Conference; 
Time frame: 7-8 months prior to start of exercise, lasts approximately 
3 days; 
Purpose: 
* Develop a draft exercise plan, scenario, modeling and simulation 
architecture, after-action review concept of operations, detailed 
augmentation requirements; 
* Update milestones and identify funding issues. In addition, 
workgroups should have master scenario events lists identified with an 
understanding of who is responsible for completing one; 
Homeland Security Exercise and Evaluation Program Guidance: Mid-Term 
Planning Conference; 
Time frame: Minimum 3 months prior to start of exercise, lasts 
approximately 1 day; 
Purpose: 
* Discuss and determine exercise organization and staffing concepts, 
scenario and time line development, scheduling, logistics, and 
administrative requirements; 
* Review draft documentation (such as scenario, exercise plan, 
controller and evaluator handbook, master scenario events list); 
* Conduct a walkthrough of the proposed exercise site at the end of the 
conference. 

DOD Joint Training System and NORTHCOM Exercise Program: The Master 
Scenario Events List Development Conference; 
Time frame: Conducted before, after, or in conjunction with the mid-
planning conference; 
Purpose: 
* Forum for the initial development of detailed scenario events and the 
master scenario events list--a detailed narrative description of the 
exercise scenario in chronological sequence; 
Homeland Security Exercise and Evaluation Program Guidance: The Master 
Scenario Events List Conference; 
Time frame: Conducted separately or combined with mid-planning 
conference or final planning conference; length varies from 4 to 8 
hours; 
Purpose: 
* Focuses on developing the master scenario events list--a 
chronological list that supplements the exercise scenario with event 
synopses; expected participant responses; capabilities, tasks, and 
objectives to be addressed; and responsible personnel. 

DOD Joint Training System and NORTHCOM Exercise Program: Final Planning 
Conference; 
Time frame: 3 months prior to start of exercise; 
Purpose: 
* Provide a last check of the status of the planning and preparation 
for exercise and to finalize, publish, and distribute key exercise 
plans and documents (such as exercise plan, supporting plans, scenario, 
after-action review collection management plan, exercise manning 
status, control plan, exercise battle rhythm, and modeling and 
simulation status); 
* Finalize the construct of the exercise with conference 
representatives; 
Homeland Security Exercise and Evaluation Program Guidance: Final 
Planning Conference; 
Time frame: Minimum 6 weeks prior to start of exercise, lasts 
approximately 1 day; 
Purpose: 
* Final forum for reviewing exercise processes and procedures; 
* No major changes to the design or scope of the exercise, or its 
supporting documentation, should take place; 
* Ensures that all logistical requirements have been met, all 
outstanding issues have been identified and resolved, and all exercise 
products are ready for printing. 

DOD Joint Training System and NORTHCOM Exercise Program: Master 
Scenario Events List Synchronization Conference; 
Time frame: Held shortly after the final planning conference and at 
least 3 weeks prior to start of exercise; 
Purpose: 
All events and implementers are reviewed for timing, realism, and 
completeness; 
Homeland Security Exercise and Evaluation Program Guidance: See master 
scenario events list conference above. 

DOD Joint Training System and NORTHCOM Exercise Program: Tabletop 
Exercises and Senior Leadership Seminars; 
Time frame: Prior to start of exercise; 
Purpose: Tabletop exercises are discussion-based exercises involving 
key personnel discussing hypothetical scenarios; 
* Facilitating understanding of concepts and processes, and identifying 
strengths and shortfalls; 
* Provide an opportunity to address new or side issues, or work out 
processes prior to the start of the exercise; 
Senior leadership seminars provide senior leaders (flag officer, 
general officer, and senior executive service level) the opportunity to 
discuss lessons learned from previous exercises and the upcoming 
exercise; 
Homeland Security Exercise and Evaluation Program Guidance: Tabletop 
Exercises and Seminars; 
Time frame: Not specified; 
Purpose: Tabletop exercises involve key personnel discussing 
hypothetical scenarios in an informal setting; 
* Used to assess plans, policies, and procedures or to assess the 
systems needed to guide the prevention of, response to, and recovery 
from a defined incident; 
Seminars are informal discussions, unconstrained by real-time portrayal 
of events and led by a presenter; 
* Orient participants to, or provide an overview of, authorities, 
strategies, plans, policies, procedures, protocols, response resources, 
and/or concepts and ideas. 

DOD Joint Training System and NORTHCOM Exercise Program: Event Tests; 
Time frame: Prior to start of exercise; 
Purpose: Conducted to ensure the modeling and simulation architecture 
is sufficient to meet exercise requirements; 
Homeland Security Exercise and Evaluation Program Guidance: Exercise 
Setup; 
Timeframe: Prior to start of exercise; 
* Exercise planning team assigned to setup should visit the exercise 
site at least 1 day prior to the event to arrange the room and test 
audio/video equipment; 
* On the day of the exercise, planning team members should arrive 
several hours before start of exercise to handle any remaining 
logistical or administrative items pertaining to setup, and to arrange 
for registration. 

DOD Joint Training System and NORTHCOM Exercise Program: Communications 
Exercise; 
Time frame: Conducted approximately 2 weeks prior to start of exercise; 
Purpose: Confirm all exercise NORAD and NORTHCOM staff and external 
battle staff members have access to and are proficient with all 
required communications systems and collaborative tools; 
Homeland Security Exercise and Evaluation Program Guidance: See 
Exercise Setup above. 

DOD Joint Training System and NORTHCOM Exercise Program: Exercise 
Execution; 
Time frame: Start of exercise; 
* Executed per the exercise plan; 
* Depending on the exercise, it may only involve the execution of a 
command post exercise or be greater in scope and involve the 
simultaneous and/or concurrent execution of other training events; 
Homeland Security Exercise and Evaluation Program Guidance: Exercise 
Conduct; 
Time frame: Start of exercise; 
* Includes setup and wrap-up activities; 
* For an operations-based exercise, conduct encompasses all operations 
occurring between the designated start of the exercise and the end of 
the exercise. 

DOD Joint Training System and NORTHCOM Exercise Program: Facilitated 
After-Action Review; 
Time frame: After the end of the exercise; 
Purpose: A facilitated discussion held at the end of the exercise 
convened for the benefit of the training audience to examine actions 
and results during a training event; 
Homeland Security Exercise and Evaluation Program Guidance: Hot Wash; 
Time frame: Maximum 2 hours after end of exercise; 
Purpose: 
* Provides opportunity to ascertain the level of satisfaction with the 
exercise, identify issues or concerns, and propose areas for 
improvement; 
* Players complete and submit their Participant Feedback Forms during 
the hot wash; 
* All evaluators take notes during play and hot washes for later 
compilation with other observations from their functional areas. 

DOD Joint Training System and NORTHCOM Exercise Program: After-Action 
Review; 
Time frame: May occur during and after the exercise; 
Purpose: An informal process designed by a commander or director to 
provide commanders direct feedback on the accomplishment of selected 
joint mission-essential tasks, conditions, and standards stated in 
terms of training objectives for the commander to evaluate training 
proficiency; 
Homeland Security Exercise and Evaluation Program Guidance: Controller 
and Evaluator Debrief; 
Time frame: Maximum 1 day after end of exercise; 
Purpose: Exercise planning team leader facilitates this debrief, which 
provides each controller and evaluator with an opportunity to provide 
an overview of the functional area he or she observed and to discuss 
both strengths and areas for improvement. 

DOD Joint Training System and NORTHCOM Exercise Program: See 
facilitated after-action review; 
Homeland Security Exercise and Evaluation Program Guidance: After-
Action Conference; 
Time frame: 5 weeks after end of exercise, lasts approximately 1 day; 
Purpose: 
* Present, discuss, and refine the draft after-action report, and to 
develop an improvement plan; 
* Solicit feedback and make necessary changes to after-action report. 

Source: GAO analysis of DOD and DHS documentation. 

[End of table] 

[End of section] 

Appendix III: Comments from the Department of Defense: 

Assistant Secretary Of Defense: 
Homeland Defense And Americas' Security Affairs: 
2600 Defense Pentagon: 
Washington, D.C. 20301-2600: 

August 11, 2009: 

Ms. Davi M. D'Agostino: 
Director, Defense Capabilities and Management: 
U.S. Government Accountability Office: 
441 G Street, N.W. 
Washington, DC 20548: 

Dear Ms. D'Agostino: 

This is the Department of Defense (DoD) response to the GAO draft 
report, GAO09-849, "Homeland Defense: U.S. Northern Command Has a 
Strong Exercise Program, but Involvement of Interagency Partners and 
States Can Be Improved," dated July 17, 2009 (GAO Code 351178). DoD 
concurs with four recommendations, and partially concurs with three 
recommendations. Our response to the recommendations is enclosed. 

Our point of contact for this action is Mr. Johnnie Wauchop, Office of 
the Assistant Secretary of Defense for Homeland Defense and Americas' 
Security Affairs (OASD (HD&ASA)), (703) 693-1968, or 
Johnnie.Wauchop@osd.mil. 

Sincerely, 

Signed by: 

Paul N. Stockton: 

Enclosure: As stated: 

[End of letter] 

GAO Draft Report - Dated July 17, 2009
GAO 09-849 (351178): 

"Homeland Defense: U.S. Northern Command Has a Strong Exercise Program, 
but Involvement of Interagency Partners and States Can Be Improved" 

Department Of Defense Comments To The Recommendations: 

Recommendation 1: The GAO recommends that the Secretary of Defense 
direct the U.S. Northern Command's (USNORTHCOM) Commander to develop 
guidance with specific criteria for post-exercise documentation, 
particularly the Exercise Summary Report as the official exercise 
record, including the content and format to be included in such reports 
that will allow the results and lessons learned of exercises to he 
easily reviewed and compared. 

DOD Response: Partially Concur. DoD agrees that such information, when 
appropriate and approved for release, should be provided in a 
standardized format that can be easily accessed and understood by 
authorized organizations which may benefit from the knowledge acquired 
through our exercise activities. Any solutions must be able to 
accommodate constraints regarding classified information. 

Recommendation 2: The GAO recommends that the Secretary of Defense, in 
consultation with the Chairman of the Joint Chiefs of Staff, the 
Commander, USNORTHCOM, and other relevant Combatant Commanders, 
coordinate with the Department of Homeland Security (DHS) and Federal 
Emergency Management Agency (FEMA) to develop guidance and procedures 
for consistently involving state officials in planning, executing, and 
assessing exercises that incorporate relevant state-specific 
information. 

DOD Response: Partially Concur. DoD concurs that better coordination 
for interfacing with State officials can be achieved. However, 
USNORTHCOM must rely on State and local exercise planning 
representatives to accurately and thoroughly represent their State 
Emergency Management structure and their training objectives during the 
12-month planning process leading up to each exercise. To this end, 
USNORTHCOM continues to expand its efforts to work through its Defense 
Coordinating Officers/Elements (DCOs/DCEs), its existing relationships 
with State National Guard forces, and with FEMA regional headquarters 
partners to ensure that States are also able to benefit from 
participating in DoD-sponsored homeland security related exercises. 

USNORTHCOM, throughout the history of its exercise program. has 
continuously engaged and encouraged the participation of State 
emergency management officials/structures in order to maximize 
available training opportunities for all stakeholders. Additionally, as 
noted in the GAO report, there are ample references to the amount of 
State "play" and the benefits that States have derived from USNORTHCOM-
sponsored exercises. Ultimately, however, it must be acknowledged that 
the primary training audience in any USNORTHCOM-sponsored exercise is 
USNORTHCOM. 

Finally, the GAOs proposed recommendation has applicability to other US 
Government interagency partners, as well as DoD. As such, the issue 
should be addressed by the Exercise and Evaluation Sub-Interagency 
Planning Committee as a revision to the National Exercise Program (NEP) 
Implementation Plan. 

Recommendation 3: The GAO recommends that the Secretary of Defense 
direct USNORTHCOM's Commander to develop a training plan for USNORTHCOM 
headquarters staff on State emergency management structures and 
relevant issues related to working with civilian State and local 
emergency management officials. 

DOD Response: Concur. Defense Coordinating Officers, who report to 
USNORTHCOM via U.S. Army North (USARNORTH), are well-versed in their 
respective States' procedures. Additionally, the headquarters staff 
training specified in the recommendation, which has been in existence 
since 2002, is required for all newly assigned USNORTHCOM staff 
members. The training includes a full discussion of how States request 
and receive Defense Support of Civil Authorities (DSCA). Further, 
USNORTHCOM sponsors three versions of its formal DSCA seminar which are 
described as follows: 

* A three phase DSCA course that includes eight hours of online 
instruction, a one-week classroom seminar, and hi-monthly post-training 
e-mail updates on DSCA news items. 

* A DSCA Executive Seminar offered four times per year and tailored to 
senior official training audiences in DoD (including the National 
Guard), interagency organizations, State emergency management agencies, 
local governments, and non-governmental organizations. 

* A soon-to-be available DSCA Senior Executive Seminar, a focused half-
day session intended for Governors. State Adjutants General, and State 
Directors of Emergency Management. 

Recommendation 4: The GAO recommends that the Secretary of Defense 
direct USNORTHCOM's Commander to establish and publicize valid and 
easily accessible procedures for non-DOD exercise participants to 
submit observations relevant to USNORTHCOM, such as placing a template 
on USNORTHCOM's publicly accessible website or the DHS's Homeland 
Security Information Network, so that USNORTHCOM officials have a 
clear, secure avenue to obtain observations and assess potential 
lessons that originate with its exercise partners. 

DOD Response: Concur. Collecting relevant exercise and training 
information tom all perspectives will undoubtedly provide additional 
opportunities to improve USNORTHCOM's ability to accomplish its mission 
essential tasks. 

Recommendation 5: The GAO recommends that the Secretary of Defense 
direct the Chairman of the Joint Chiefs of Staff, in consultation and 
coordination with DHS, to either resolve information assurance issues 
so that the combatant commands, including USNORTHCOM, can post Exercise 
Summary Reports with lessons learned and observations from the National 
Exercise Program (NEP) exercises on DHS's Lessons Learned Information 
Sharing system to make them easily accessible to interagency partners 
and states or establish an alternative method to systematically collect 
and share lessons learned. 

DOD Response: Concur. DoD agrees that such information, when 
appropriate and approved for release, should be disseminated as widely 
as possible to those homeland security related entities which may 
benefit from the knowledge acquired through our exercise activities. 
Concurrence, however, should not be misconstrued as support for 
altering or departing from the Joint Training System and its 
components, specifically the Joint Lessons Learned Information System. 
Additionally, DoD cannot mandate that DHS alter its Lessons Learned 
Information System nor require DHS to commit resources to do so. There 
are procedures in place that allow specifically cleared individuals 
from outside DoD access to information contained in the Exercise 
Summary Reports. These manual procedures ensure that only appropriate 
and authorized information is offered outside the Department. 

Recommendation 6: The GAO recommends that the Secretary of Defense 
direct the Chairman of the Joint Chiefs of Staff to revise the joint 
lessons learned operating instruction to include procedures to ensure 
that appropriate corrective actions are implemented and verified in a 
subsequent exercise or operation before being closed and that the 
reasons for closure are documented. Possible procedures might be adding 
a verification checkbox on the Joint Lessons Learned Information 
System's (JLLIS) issue management page or requiring that the 
directorates and subordinate commands within the combatant commands 
provide a status report when a correction is implemented and re-
observed or closed for reasons other than re-observation. 

DOD Response: Concur. Chairman of the Joint Chiefs of Staff Instruction 
(CJCSI) 3150.25D provides policy and guidance for the Joint Lessons 
Learned Program and depicts the "revalidation" of issues to ensure 
lessons have been learned (Figure A-2, Joint Lessons Learned Program 
Process Overview). This discussion section could be expanded to provide 
more guidance and the JLLIS could he updated to provide a technological 
solution to address the issue once the process and procedures are in 
place. 

Regarding internal tracking, JLLIS provides sufficient capability for 
directors and subordinate commands to effectively manage internal 
issues. JLLIS Tier I administrators can make local modifications to 
facilitate this process. The process of verifying corrective action and 
closing issues will become more effective with the suggested 
modification, continued education, and further maturation of JLLIS and 
the Joint Lessons Learned Program. 

Recommendation 7: The GAO recommends that the Secretary of Defense 
revise the instruction on DOD participation in the NEP and/or direct 
the Chairman of the Joint Chiefs of Staff to revise the operating 
instruction regarding DOD participation in the NEP to provide the 
general conditions under which the combatant commands are expected to 
follow the Homeland Security Exercise and Evaluation Program planning 
and documentation requirements or the DOD's Joint Training System 
should be modified for those civil support exercises. 

DOD Response: Partially Concur. DoD recognizes the importance of 
ensuring the Department is sufficiently able to interact with our 
interagency partners at every level regarding the planning and 
execution of Homeland Security-related exercises. To this end, DoD has 
fully supported and, when appropriate, has directly assisted in the 
development of the NEP and its underlying doctrine, methodologies, and 
technologies. Evidence of these efforts are readily found in the GAO 
report where the similarities between DoD exercise methodology and 
National Exercise Program doctrine are obvious. 

The NEP Implementation Plan (I-Plan) clearly delineates how DoD 
participates in the NEP. Additionally, largely due to DoD's insistence, 
the I-Plan contains language in paragraph 6.2 establishing processes to 
resolve doctrinal differences during the initial stages of NEP exercise 
planning. Further, the NEP I-Plan establishes the administration, 
scope, and hierarchy of homeland security-related exercises involving 
multiple interagency entities. Combined with its associated NEP five-
year schedule which includes USNORTHCOM homeland security-related 
exercises, the 1-Plan and the five-year schedule appropriately address 
the GAO's recommendation. 

It should be emphasized, however, that the Secretary of Defense retains 
sole authority for training and equipping the Department of Defense. 
The Joint Training System, as it is currently administered, constitutes 
the Secretary's guidance regarding this matter. This position is 
further reinforced in paragraph 4 of the NEP Implementation plan, which 
states "The NEP is intended to provide a framework for prioritizing and 
focusing Federal exercise activities, without replacing any individual 
department or agency exercise program". 

[End of section] 

Appendix IV: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Davi M. D'Agostino, (202) 512-5431 or dagostinod@gao.gov. 

Acknowledgments: 

In addition to the contact named above, Joseph Kirschbaum, Assistant 
Director; Gilbert Kim; David Hubbell; Joanne Landesman; Christopher 
Mulkins; Erin Noel; Terry Richardson; and Richard Winsor made key 
contributions to this report. 

[End of section] 

Related GAO Products: 

National Preparedness: FEMA Has Made Progress, but Needs to Complete 
and Integrate Planning, Exercise, and Assessment Efforts. [hyperlink, 
http://www.gao.gov/products/GAO-09-369]. Washington, D.C.: April 30, 
2009. 

Emergency Management: Observations on DHS' Preparedness for 
Catastrophic Disasters. [hyperlink, 
http://www.gao.gov/products/GAO-08-868T]. Washington, D.C.: June 11, 
2008. 

National Response Framework: FEMA Needs Policies and Procedures to 
Better Integrate Non-Federal Stakeholders in the Revision Process. 
[hyperlink, http://www.gao.gov/products/GAO-08-768]. Washington, D.C.: 
June 11, 2008. 

Homeland Defense: Steps Have Been Taken to Improve U.S. Northern 
Command's Coordination with States and the National Guards Bureau, but 
Gaps Remain. [hyperlink, http://www.gao.gov/products/GAO-08-252]. 
Washington, D.C.: April 16, 2008. 

Homeland Defense: U.S. Northern Command Has Made Progress but Needs to 
Address force Allocation, Readiness Tracking Gaps, and Other Issues. 
[hyperlink, http://www.gao.gov/products/GAO-08-251]. Washington, D.C.: 
April 16, 2008. 

Continuity of Operations: Selected Agencies Tested Various Capabilities 
during 2006 Governmentwide Exercise. [hyperlink, 
http://www.gao.gov/products/GAO-08-185]. Washington, D.C.: November 19, 
2007. 

Homeland Security: Preliminary Information on Federal Action to Address 
Challenges Faced by State and Local Information Fusion Centers. 
[hyperlink, http://www.gao.gov/products/GAO-07-1241T]. Washington, 
D.C.: September 27, 2007. 

Homeland Security: Observations on DHS and FEMA Efforts to Prepare for 
and Respond to Major and Catastrophic Disasters and Address Related 
Recommendations and Legislation. [hyperlink, 
http://www.gao.gov/products/GAO-07-1142T]. Washington, D.C.: July 31, 
2007. 

Influenza Pandemic: DOD Combatant Commands' Preparedness Efforts Could 
Benefit from More Clearly Defined Roles, Resources, and Risk 
Mitigation. [hyperlink, http://www.gao.gov/products/GAO-07-696]. 
Washington, D.C.: June 20, 2007. 

Homeland Security: Preparing for and Responding to Disasters. 
[hyperlink, http://www.gao.gov/products/GAO-07-395T]. Washington, D.C.: 
March 9, 2007. 

Catastrophic Disasters: Enhanced Leadership, Capabilities, and 
Accountability Controls Will Improve the Effectiveness of the Nation's 
Preparedness, Response, and Recovery System. [hyperlink, 
http://www.gao.gov/products/GAO-06-618]. Washington, D.C.: September 6, 
2006. 

Homeland Defense: National Guard Bureau Needs to Clarify Civil Support 
Teams' M9ission and Address Management Challenges. [hyperlink, 
http://www.gao.gov/products/GAO-06-498]. Washington, D.C.: May 31, 
2006. 

Hurricane Katrina: Better Plans and Exercises Needed to Guide the 
Military's Response to Catastrophic Natural Disasters. [hyperlink, 
http://www.gao.gov/products/GAO-06-808T]. Washington, D.C.: May 25, 
2006. 

Hurricane Katrina: Better Plans and Exercises Needed to Guide the 
Military's Response to Catastrophic Natural Disasters. [hyperlink, 
http://www.gao.gov/products/GAO-06-643]. Washington, D.C.: May 15, 
2006. 

Hurricane Katrina: GAO's Preliminary Observations Regarding 
Preparedness, Response, and Recovery. [hyperlink, 
http://www.gao.gov/products/GAO-06-442T]. Washington, D.C.: March 8, 
2006. 

Emergency Preparedness and Response: Some Issues and Challenges 
Associated with major Emergency Incidents. [hyperlink, 
http://www.gao.gov/products/GAO-06-467T]. Washington, D.C.: February 
23, 2006. 

Statement by Comptroller General David M. Walker on GAO'S Preliminary 
Observations Regarding Preparedness and Response to Hurricanes Katrina 
and Rita. [hyperlink, http://www.gao.gov/products/GAO-06-365R]. 
Washington, D.C.: February 1, 2006. 

Homeland Security: DHS' Efforts to Enhance First Responders' All- 
Hazards Capabilities Continue to Evolve. [hyperlink, 
http://www.gao.gov/products/GAO-05-652]. Washington, D.C.: July 11, 
2005. 

Homeland Security: Process for Reporting Lessons Learned from Seaport 
Exercises Needs Further Attention. [hyperlink, 
http://www.gao.gov/products/GAO-05-170]. Washington, D.C.: January 14, 
2005. 

Homeland Security: Federal Leadership and Intergovernmental Cooperation 
Required to Achieve First Responder Interoperable Communications. 
[hyperlink, http://www.gao.gov/products/GAO-04-740]. Washington, D.C.: 
July 20, 2004. 

Homeland Defense: DOD Needs to Assess the Structure of U.S. Forces for 
Domestic Military Missions. [hyperlink, 
http://www.gao.gov/products/GAO-03-670]. Washington, D.C.: July 11, 
2003. 

Combating Terrorism: Selected Challenges and Related Recommendations. 
[hyperlink, http://www.gao.gov/products/GAO-01-822]. Washington, D.C.: 
September 20, 2001. 

Joint Training: Observations on the Chairman, Joint Chiefs of Staff, 
Exercise Program. [hyperlink, 
http://www.gao.gov/products/NSIAD-98-189]. Washington, D.C.: July 10, 
1998. 

[End of section] 

Footnotes: 

[1] Homeland Security Council, National Strategy for Homeland Security 
(Washington, D.C: October 2007), 1, 9, 13. 

[2] GAO, National Preparedness: FEMA Has Made Progress, but Needs to 
Complete and Integrate Planning, Exercise, and Assessment Efforts, 
[hyperlink, http://www.gao.gov/products/GAO-09-369] (Washington, D.C.: 
Apr. 30, 2009). 

[3] DOD, Quadrennial Defense Review Report (Washington, D.C.: Feb. 6, 
2006), 87. 

[4] An exercise is a training event where scenarios are enacted to 
assess training proficiency and identify lessons learned to improve 
preparedness. Exercises range from tabletop computer simulations to the 
full-scale deployment of personnel and equipment. 

[5] U.S. Northern Command, Commander's Training Guidance for FY09-10 
(Colorado Springs, Colo.: Apr. 8, 2008), A-20. 

[6] DOD, Strategy for Homeland Defense and Civil Support, 5. 

[7] DOD, Strategy for Homeland Defense and Civil Support, 5. 

[8] DOD, Strategy for Homeland Defense and Civil Support, 5-6. 

[9] DOD, National Defense Strategy (Washington, D.C.: June 2008), 17- 
18. 

[10] Defense Science Board, Unconventional Operational Concepts and the 
Homeland (Washington, D.C.: March 2009), 37, 48, 50. 

[11] U.S. Northern Command, Commander's Training Guidance for FY09-10, 
A-4. 

[12] DOD, Quadrennial Defense Review Report, 26-27. 

[13] DHS, National Exercise Program Implementation Plan (Washington 
D.C.: Apr. 11, 2007). DHS issued a revised Implementation Plan in June 
2008. 

[14] GAO, Homeland Security: Observations on DHS and FEMA Efforts to 
Prepare for and Respond to Major and Catastrophic Disasters and Address 
Related Recommendations and Legislation, [hyperlink, 
http://www.gao.gov/products/GAO-07-835T] (Washington, D.C.: May 15, 
2007). 

[15] GAO, Hurricane Katrina: Better Plans and Exercises Needed to Guide 
the Military's Response to Catastrophic Natural Disasters, [hyperlink, 
http://www.gao.gov/products/GAO-06-643] (Washington, D.C.: May16, 
2006). 

[16] [hyperlink, http://www.gao.gov/products/GAO-09-369]. 

[17] GAO, Homeland Defense: U.S. Northern Command Has Made Progress but 
Needs to Address Force Allocation, Readiness, Tracking Gaps, and Other 
Issues, [hyperlink, http://www.gao.gov/products/GAO-08-251] 
(Washington, D.C.: Apr. 16, 2008); and Homeland Defense: Steps Have 
Been Taken to Improve U.S. Northern Command's Coordination with the 
States and the National Guard Bureau, but Gaps Remain, [hyperlink, 
http://www.gao.gov/products/GAO-08-252] (Washington, D.C.: Apr. 18, 
2008). 

[18] Large-scale exercises train and evaluate actual forces deployed in 
a field environment under simulated scenarios. NORTHCOM generally 
conducts two large-scale exercises annually. 

[19] Chairman of the Joint Chiefs of Staff, Joint Training Manual for 
the Armed Forces of the United States, 3500.03B (Washington, D.C.: Aug. 
31, 2007), A-1; U.S. Northern Command, Operations: Exercise Program, 
Instruction 10-156 (Colorado Springs, Colo.: June 2, 2008). 

[20] U.S. Northern Command, Commander's Training Guidance FY 09-10, A- 
4. 

[21] Chairman of the Joint Chiefs of Staff, Joint Lessons Learned 
Program, Instruction 3150.25D (Washington, D.C.: Oct. 10, 2008), A-1 
and A-6; U.S. Northern Command, Commander's Training Guidance FY 09-10 
, A-4, A-8, A-20, A-2-8; U.S. Northern Command, Lessons Learned Program 
and Corrective Action Board Process, Instruction 16-166 (Colorado 
Springs, Colo.: Mar. 16, 2009), 2. 

[22] DHS, National Exercise Program Implementation Plan (Washington, 
D.C.: June 20, 2008). 

[23] FEMA officials stated that federal departments and agencies should 
be held accountable for meeting key Homeland Security Exercise 
Evaluation Program requirements, but that FEMA lacks the authority to 
ensure compliance. While we recognize that FEMA's role is generally to 
coordinate, guide, and support, we believe that FEMA's expanded 
leadership role under the Post-Katrina Act of 2006 provides FEMA 
opportunities to instill a shared sense of responsibility and 
accountability on the part of all agencies. GAO-09-369. The Post- 
Katrina Act was enacted as Title VI of the Department of Homeland 
Security Appropriations Act, 2007, Pub. L. No. 109-295, (2006). 

[24] NORTHCOM guidance states the Vigilant Guard Program should use 
Homeland Security Exercise and Evaluation Program formats to encourage 
civilian agency participation. U.S. Northern Command, Commander's 
Training Guidance: Fiscal Year 2009-2010, p. A-18. 

[25] Defense Science Board, Unconventional Operational Concepts and the 
Homeland. 

[26] [hyperlink, http://www.gao.gov/products/GAO-09-369]. 

[27] A training event focuses primarily on improving individual or 
collective ability to perform, such as academic or field training. An 
exercise focuses primarily on evaluating capability or an element of 
capability, such as a plan or policy. Necessary training should take 
place prior to an exercise. 

[28] U.S. Northern Command, Commander's Training Guidance for FY09-10, 
A-20. 

[29] U.S. Northern Command, Lessons Learned Program and Corrective 
Action Board Process, Instruction 16-166 (Colorado Springs, Colo.: Mar. 
16, 2009), 2. 

[30] The framework is a guide to how the nation conducts all-hazards 
response and is intended to capture specific authorities and best 
practices for managing incidents that range from the serious but purely 
local, to large-scale terrorist attacks or catastrophic natural 
disasters. DHS, National Response Framework (Washington, D.C.: January 
2008). 

[31] Chairman of the Joint Chiefs of Staff, Instruction 3500.01E, Joint 
Training Policy and Guidance for the Armed Forces of the United States 
(Washington, D.C.: May 31, 2008). 

[32] DOD, Strategy for Homeland Defense and Civil Support, 8. 

[33] Homeland defense is considered DOD's portion of the broader area 
of homeland security. DHS is the lead primary federal agency for 
homeland security issues, which is a concerted national effort to 
prevent terrorist attacks within the United States, reduce America's 
vulnerability to terrorism, and minimize the damage and recover from 
attacks that do occur. Homeland Security Council, National Strategy for 
Homeland Security, 3; DOD, Strategy for Homeland Defense and Civil 
Support, 5. 

[34] DHS, National Response Framework; Joint Chiefs of Staff, Joint 
Pub. 3-28, Civil Support (Washington, D.C.: Sept. 14, 2007). 

[35] For purposes of this report, large-scale exercises are those 
involving multiple agencies and organizations, venues, and events, such 
as NORTHCOM's Vigilant Shield and Ardent Sentry exercises. 

[36] DOD established the Joint Training System in fiscal year 1994. 
GAO, Military Training: Actions Needed to Enhance DOD's Program to 
Transform Joint Training, [hyperlink, 
http://www.gao.gov/products/GAO-05-548] (Washington, D.C.: June 21, 
2005) 3, 19. 

[37] The documents that provide the written framework for the Joint 
Training System are Chairman of the Joint Chiefs of Staff, Instruction 
3500.01E, Joint Training Policy and Guidance for the Armed Forces of 
the United States (Washington, D.C.: May 31, 2008); Manual 3500.03B, 
Joint Training Manual for the Armed Forces of the United States 
(Washington, D.C.: Aug. 31, 2007); and Manual 3500.04E, Universal Joint 
Task Manual (Washington, D.C.: Aug. 25, 2008). 

[38] The Joint Mission Essential Task List outlines those tasks that 
are essential to a combatant command's ability to perform assigned 
missions. The list is a key element in readiness assessment and 
reporting. 

[39] Specifically, the Joint Training Information System provides the 
joint community with an integrated software capability to identify, 
collect, analyze, store, and disseminate the data required to execute 
the Joint Training System. 

[40] Joint exercises have been characterized by some form of 
multiechelon, computer-assisted exercises which embody complex 
simulation. These may be either a field training exercise, command 
postexercise, or a tabletop exercise. 

[41] Chairman of the Joint Chiefs of Staff, Joint Lessons Learned 
Program, 3150.25D, C-1. 

[42] DHS, National Exercise Program Implementation Plan. 

[43] Other steering committee members include DOD, the Office of the 
Director of National Intelligence, the Federal Bureau of Investigation, 
and the Departments of State, Justice, Energy, Transportation, and 
Health and Human Services. 

[44] DHS, Homeland Security Exercise and Evaluation Program, Volumes 1- 
5 (Washington, D.C.: February 2007). 

[45] [hyperlink, https://hseep.dhs.gov/pages/1001_Toolk.aspx]. 

[46] [hyperlink, https://llis.gov]. 

[47] U.S. Northern Command, Commander's Training Guidance for FY 09-10, 
A-11. 

[48] The Joint Events Life Cycle provides a sequence of the inputs, 
process steps, and outputs necessary to execute any size exercise. This 
process consists of five stages including design, planning, 
preparation, execution, and evaluation. A series of planning 
conferences provide discrete breakpoints between each stage. Chairman 
of the Joint Chiefs of Staff, Joint Training Manual for the Armed 
forces of the United States, E-6. 

[49] NORTHCOM is given an overall rating in the system for the 
command's current status of "trained", "needs practice," or "untrained" 
for each Joint Mission Essential Task. U.S. Northern Command, 
Memorandum on NORAD and USNORTHCOM Joint Training Plan FY08-09(Colorado 
Springs, Colo.: July 30, 2007, A-32-A-35; Chairman of the Joint Chiefs 
of Staff, Joint Training Manual for the Armed Forces of the United 
States, F-5-F-6. 

[50] Chairman of the Joint Chiefs of Staff, Joint Training Manual for 
the Armed Forces of the United States, E-6-E-12. 

[51] U.S. Northern Command, Operations: Exercise Program, Instruction 
10-156 (Colorado Springs, Colo.: June 2008), 20. 

[52] While NORTHCOM was unable to provide exercise summary or quick- 
look reports for United Defense 03, they did provide pre-exercise 
briefings. NORTHCOM officials told us that the exercise started late 
due to the Columbia shuttle disaster and ended early due to a level 
Orange alert in the Homeland Security Alert System. 

[53] U.S. Northern Command, Operations: Exercise Program. The 
instruction, issued in June 2008, also requires exercise quick look 
reports be issued within 30 days after the completion of an exercise. 
The instruction is only applicable to Ardent Sentry 08 and Vigilant 
Shield 09, as the post-exercise documents for these reports were issued 
after June 2008. Post-exercise documents for all other exercises 
reviewed by GAO were issued prior to June 2008. 

[54] U.S. Northern Command, Operations: Exercise Program, 20. 

[55] U.S. Northern Command, Operations: Exercise Program, 20. 

[56] Access to JLLIS is restricted in order to safeguard sensitive 
information because of concerns that adversaries may be able to exploit 
any weaknesses contained in those records. For users to access the 
unclassified portion of the JLLIS they must have a DOD Common Access 
Card, a card reader, and appropriate computer software prior to 
requesting access. 

[57] Collection Management Plans are generally linked to the Joint 
Mission Essential Tasks and exercise objectives. Focus areas include a 
wide range of operations, such as coordinating and integrating regional 
interagency activities and coordinating consequence management in 
theater. U.S. Northern Command, Operations: Exercise Program, 20. 

[58] DHS, Homeland Security Exercise Evaluation Program, Volume III: 
Exercise Evaluation and Implementation Plan, (Washington, DC: February 
2007), A-1 - B-2. 

[59] U.S. Northern Command, North American Aerospace Defense Command 
and United States Northern Command National Level Exercise 2-08: 1-8 
May 2008 Exercise Summary Report, (Colorado Springs, Colo.: Aug. 15, 
2008)1-25. 

[60] U.S. Northern Command, Commander's Training Guidance for FY09-10; 
Chairman of the Joint Chiefs of Staff, Joint Training Manual for the 
Armed Forces of the United States, CJCSM 3500.03B (Aug. 31, 2007). 

[61] We visited all of these states except for Connecticut and Indiana. 
We visited Indiana during a review of NORTHCOM's planning and 
interagency coordination in the spring of 2007. We also met with Nevada 
officials who participated in Determined Promise 03. The remaining 9 of 
the 17 states played lesser roles in the exercises. 

[62] [hyperlink, http://www.gao.gov/products/GAO-08-252]. 

[63] Injects are pieces of information presented to the training 
audience during an exercise in a manner similar to real-world 
operations to guide the training audience toward the accomplishment of 
the training objectives. 

[64] Defense Science Board, Unconventional Operational Concepts and the 
Homeland, viii, xiii, 38. 

[65] Creating an exercise scenario event that is believable is critical 
to engaging participants in a level of play and experiential learning 
that will be long lasting. Defense Science Board, Unconventional 
Operational Concepts and the Homeland, 55. 

[66] U.S. Northern Command, Northern American Aerospace Defense Command 
and United States Northern Command Ardent Sentry 06 Exercise Summary 
Report (Colorado Springs, Colo.: Sept. 27, 2006), 4-6. 

[67] State emergency management officials told us that they attended 
planning meetings with NORTHCOM and raised their concern about the 
scenarios, but NORTHCOM officials made the decision to use the scenario 
anyway in order to meet their own training objectives. As a result, 
these officials told us that several other states dropped out of the 
exercise for this scenario during the planning stages. 

[68] The Defense Science Board recently reported that national 
exercises typically focus on a top-down approach where the supporting 
organizations are training aids to the senior-level players, instead of 
a bottom-up approach focusing on an integrated and layered response. 
Defense Science Board, Unconventional Operational Concepts and the 
Homeland, xv. 

[69] In some cases, NORTHCOM's coordination with the National Guard has 
been inconsistent. For example, National Guard officials from one state 
told us that it is unclear whether communication with NORTHCOM should 
occur directly or through the National Guard Bureau. 

[70] Defense Coordinating Officers are senior-level military officers 
who provide liaison support and serve as a point of contact for 
federal, state, and local agencies requiring DOD support during an 
incident. [hyperlink, http://www.gao.gov/products/GAO-08-252]. 

[71] Chairman of the Joint Chiefs of Staff, Joint Lessons Learned 
Program, Instruction 3150.25D, A-1 and A-6; U.S. Northern Command, 
Commander's Training Guidance FY 09-10, A-4, A-8, A-20, A-2-8: U.S. 
Northern Command, Lessons Learned Program and Corrective Action Board 
Process, 2. 

[72] The Joint Staff develops joint lessons learned policy and serves 
as the program manager for JLLIS. 

[73] JLLIS became the system of record in October 2008. Prior to that 
NORTHCOM's lessons learned system was the Advanced Lesson Management 
System. 

[74] U.S. Northern Command, Lessons Learned Policy and Guidance, 
Operating Instruction 36-104, (Colorado Springs, Colo.: Oct. 1, 2003). 

[75] Chairman of the Joint Chiefs of Staff, Instruction 3150.25B, Joint 
Lessons Learned Program, A-2 - A-3 (Washington, D.C.: February 15, 
2005). 

[76] Those component commands, such as Army Forces North or Air Force 
North, may also have a lessons learned program of their own. 

[77] The Corrective Action Board process is intended to provide a means 
for addressing commandwide issues and ensuring these issues are 
tracked, resolved, and verified. 

[78] The Corrective Action Board Working Group is chaired by the head 
of the Joint Training and Exercises Directorate, and includes 
representatives from each NORTHCOM directorate and subcommand and 
reviews issues requiring extensive coordination across directorates and 
subordinate commands or action by an organization(s) external to 
NORTHCOM. The Executive Corrective Action Board is chaired by 
NORTHCOM's Chief of Staff and comprised of the heads of the 
directorates and subcommands and reviews sensitive issues requiring 
decisions by senior leaders or actions of organizations outside of 
NORTHCOM, such as the Joint Staff. For example, lessons learned that 
are shared with DHS are reviewed by the Executive Corrective Action 
Board before they are forwarded to the Joint Staff. 

[79] NORTHCOM has a process for sharing lessons learned from National 
Level Exercises that are applicable to the interagency to DHS through 
the Joint Staff. Thus far, NORTHCOM has provided three3 issues 
requiring corrective action from National Level Exercise 2-08 conducted 
in May 2008. 

[80] Joint Staff officials told us that about 10Department of State 
officials have access to JLLIS, and DHS and Department of Energy 
officials have requested access for some of their staff. Other 
departments and agencies, such as the Federal Bureau of Investigation 
and Department of Health and Human Services, have received briefings 
about JLLIS, but have not requested access. The Joint Staff official 
also told us that his office does not have enough staff to support a 
large number of non-common access card users requesting JLLIS access, 
and granting access would be a lengthy process due to the software and 
security requirements that must be addressed. 

[81] NORTHCOM's lessons learned manager told us that those without 
cards that have a valid need for access to the portal must work with 
NORTHCOM's Help Desk staff to register for an account. 

[82] DHS's Web-based Homeland Security Information Network facilitates 
information sharing and collaboration between federal, state, local, 
tribal, private sector, and international partners involved in homeland 
security. 

[83] Depending on the exercise, for example if it is linked to a 
national level or regional exercise, multiple Facilitated After Action 
Reviews may take place. 

[84] NORTHCOM reports posted on FEMA's Lessons Learned Information 
Sharing system include exercise summary reports for National Level 
Exercise 1-08 (Vigilant Shield 08), and Ardent Sentry 2007; two copies 
of the Quick Look Report for National Level Exercise 2-08; NORTHCOM 
Support for 2008 Hurricane Season Operations Summary Report; and 
NORTHCOM Support for 2008 Political Conventions Operations Summary 
Report. 

[85] We found one document where details were provided on 12 
observations. 

[86] U.S. Northern Command, National Level Exercise 2-08 Exercise 
Summary Report (Colorado Springs, Colo.: Aug. 15, 2008), 16. 

[87] Chairman of the Joint Chiefs of Staff, Instruction 3150.25D, Joint 
Lessons Learned Program (Washington, D.C.: Oct. 10, 2008), A-8; U.S. 
Northern Command, Lessons Learned Program and Corrective Action Board 
Process, Instruction 16-166 (Colorado Springs, CO: Mar. 16, 2009), 14; 
Prior Chairman of the Joint Chiefs of Staff guidance also required that 
corrective actions be tracked and remain open until the solutions are 
completed and verified. 

[88] We reviewed active records from NORTHCOM's large-scale exercises 
that were conducted between May 2006 and November 2008 for Ardent 
Sentry 2006, 2007, and 2008 and Vigilant Shield 2007, 2008, and 2009. 

[89] The corrective action included a set of checklists and processes 
added to the command's battle staff operating procedures. 

[90] The scenarios developed for NORTHCOM's next two applicable major 
exercises were not of sufficient scope to induce another nation, other 
than Canada, to offer military assistance. 

[91] The Command Assessment Element is a team supporting NORTHCOM's 
commander by providing situational awareness from a disaster or 
incident site. 

[92] For the purpose of this report, we are defining major national 
exercises as operations-based NEP exercises involving the deployment of 
personnel or forces to conduct operations from the federal, state, and 
local levels. 

[93] DHS, National Exercise Program Implementation Plan. 

[94] DOD, DOD Participation in the National Exercise Program, 
Instruction 3020.47, (Washington, D.C.: Jan. 29, 2009), 10. 

[95] The command's Joint Training Plan, which includes the commander's 
training guidance, is revised and updated annually. U.S. Northern 
Command, Commander's Training Guidance for FY 09-10. 

[96] The Top Officials exercise, better known as TOPOFF, is the 
nation's terrorism preparedness exercise involving officials at every 
level of government and representatives from the international 
community and private sector. Sponsored by DHS, TOPOFF 4 shared the 
same radiological dispersal device scenario in Vigilant Shield 08 and 
included more than 100 organizations. 

[97] National Level Exercise 1-08 included Global Lightning 08 at U.S. 
Strategic Command, Positive Response 08-1 for the Chairman, Joint 
Chiefs of Staff, Blue Flame in the United Kingdom, Pacific Challenge in 
Australia, a U.S. Southern Command Response Cell due to Able Warrior 08-
1 date change, and an unnamed exercise at U.S. Pacific Command. 

[98] FEMA's National Exercise Simulation Center opened on January 12, 
2009, to provide a national, shared facility to train and exercise and 
improve coordination for all-hazards preparedness. The goal was to pool 
resources, maximize efficiency, and provide sustained exercise and 
training support to all participants. 

[99] FEMA officials told us that only tier 1 and 2 exercises are 
required to be included in the national schedule, but encouraged the 
inclusion of Tier 3 exercises. 

[100] The list of participating departments or agencies may change as 
the planning for Vigilant Shield 10 progresses. 

[101] This plan was revised in June 2008, but has not been formally 
approved by the current administration. However, FEMA officials told us 
that they intend to execute the NEP using the latest revision of the 
plan. DHS, DRAFT National Exercise Program Implementation Plan 
(Washington, D.C.: June 20, 2008). 

[102] [hyperlink, http://www.gao.gov/products/GAO-09-369], p. 35. 

[103] NORTHCOM officials told us that they have used Homeland Security 
Exercise Evaluation Program guidance in one case when they drafted the 
exercise plan for NLE 2-08, because their state and local exercise 
partners were required to use this guidance. 

[104] NORTHCOM's training guidance for fiscal years 2008-2009 stated 
that the command will adhere to the guiding principles of the Homeland 
Security and Exercise Evaluation Program, but this reference has been 
removed from guidance for fiscal years 2009-2010. 

[105] DHS, Homeland Security Exercise and Evaluation Program, Volume 1: 
HSEEP Overview and Exercise Program Management (Washington, D.C.: 
February 2007). 

[106] U.S. Northern Command, Operations: Exercise Program, pp. 4-5. 

[107] DOD, DOD Participation in the National Exercise Program, 
Instruction 3020.47, 7-8. 

[108] According to FEMA officials, since tier I and II exercises are 
interagency exercises, the Homeland Security Exercise and Evaluation 
Program should be used because it was designed for the interagency 
community. 

[109] [hyperlink, http://www.gao.gov/products/GAO-09-369]. 

[110] NORTHCOM guidance states the Vigilant Guard Program should use 
Homeland Security Exercise and Evaluation Program formats to encourage 
civilian agency participation. U.S. Northern Command, Commander's 
Training Guidance: Fiscal Year 2009-2010, A-18. 

[111] Defense Science Board, Unconventional Operational Concepts and 
the Homeland. 

[112] [hyperlink, http://www.gao.gov/products/GAO-09-369], p. 42. 

[113] [hyperlink, http://www.gao.gov/products/GAO-09-369], p. 68. 

[114] For purposes of this report, large-scale exercises are those 
involving multiple agencies and organizations, venues, and events, such 
as NORTHCOM's Vigilant Shield and Ardent Sentry exercises. NORTHCOM 
generally conducts two large-scale exercises annually. 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO’s Web site, 
[hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional 
information. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, dawnr@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: