This is the accessible text file for GAO report number GAO-07-650 
entitled 'Weather Forecasting: National Weather Service's Operations 
Prototype Needs More Rigorous Planning' which was released on July 9, 
2007. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to Congressional Requesters: 

United States Government Accountability Office: 

GAO: 

June 2007: 

Weather Forecasting: 

National Weather Service's Operations Prototype Needs More Rigorous 
Planning: 

GAO-07-650: 

GAO Highlights: 

Highlights of GAO-07-650, a report to congressional requesters 

Why GAO Did This Study: 

Using advanced systems and trained specialists located in 122 weather 
forecast offices throughout the country, the National Weather Service 
(NWS) provides storm and flood warnings and weather forecasts to 
protect life and property and to enhance the national economy. To 
improve the efficiency of its operations, in November 2006, NWS 
approved an effort to develop a prototype of an alternative way of 
operating. Under this prototype, weather forecasting offices would 
share selected responsibilities. 

GAO (1) determined the status of and plans for the prototype, (2) 
evaluated whether the prototype’s justification was sufficient, (3) 
determined whether NWS’s plans to evaluate the prototype are adequate, 
(4) evaluated whether NWS is sufficiently involving stakeholders in its 
prototype plans, and (5) determined how NWS plans to ensure that there 
will be no degradation of service during and after the prototype. To do 
so, GAO analyzed agency documentation and interviewed program officials 
and stakeholders. 

What GAO Found: 

NWS’s prototype is currently on hold pending a reevaluation of the 
agency’s approach. The agency had just begun the first phase of its 
three-phased prototype to demonstrate a new concept of operations over 
a 2-year period when, in late March 2007, the Department of Commerce’s 
Under Secretary suspended the prototype because of concerns about the 
agency’s approach. In the first phase, NWS established a program 
manager and began planning for the next two phases. During the 
remaining phases, NWS planned to have 20 weather forecasting offices 
share responsibilities in 2-office pairs and then in 4-office clusters. 
NWS then planned to decide whether to implement the new concept of 
operations on a national basis. 

The justification for the prototype was not sufficient. Before the 
prototype was suspended, the agency had approved moving forward with 
its prototype without conducting a cost-benefit analysis. NWS estimated 
that the prototype would cost approximately $9.3 million and would 
offer qualitative benefits, such as increased efficiency and an 
improved ability to focus on severe weather events, but did not 
quantify benefits or the expected return on its investment. If NWS were 
to proceed with the prototype without a cost-benefit analysis, it would 
lack assurance that its approach would be a cost-effective investment 
for the agency. 

NWS identified goals and selected measures to evaluate during its 
prototype activities, but it did not establish a rigorous evaluation 
plan. Specifically, NWS did not define a full set of needed measures, 
how it planned to compare prototype results with baseline performance, 
or how its selected measures supported the prototype goals. If NWS were 
to proceed without a rigorous evaluation plan, the agency would run an 
increased risk of not sufficiently measuring the impact of changes on 
its performance and could make decisions affecting the nation’s weather 
on the basis of incomplete or flawed data. 

Although NWS involved internal stakeholders in planning its prototype, 
it did not (1) involve external stakeholders or (2) establish a plan 
that identified key stakeholders, the stakeholders’ responsibilities, 
and a time line for involving stakeholders and addressing their 
comments. If NWS were to proceed with its prototype without such a 
plan, it could not ensure that stakeholder interests would be 
identified and addressed. 

NWS planned to mitigate the risk of degradation during and after the 
prototype by conducting laboratory exercises to understand the impact 
of the prototype, using a dedicated workstation in each office so that 
the office could switch to original systems if warranted, and 
monitoring its systems and products during the prototype. NWS officials 
stated that should the agency decide to implement the prototype on a 
national basis, it has standard procedures for testing and validating 
systems and software to avoid any degradation of service. 

What GAO Recommends: 

GAO is recommending that if NWS decides to proceed with its prototype, 
it undertake a more rigorous approach to justifying, evaluating, and 
involving stake-holders in the prototype. In written comments, Commerce 
agreed with GAO’s recommendations. 

[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-650]. 

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact David Powner at (202) 512-
9286 or pownerd@gao.gov. 

[End of section] 

Contents: 

Letter: 

Results in Brief: 

Background: 

NWS Initiated a 2-Year Prototype and Then Suspended It: 

NWS Has Not Yet Sufficiently Justified Its Prototype: 

NWS Lacks a Rigorous Evaluation Plan for Assessing Its Prototype: 

NWS Involved Internal Stakeholders in Planning the Prototype, but Did 
Not Involve External Stakeholders or Establish a Plan for Stakeholder 
Involvement: 

NWS Planned to Ensure That the Prototype Would Not Degrade Its Service: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Objectives, Scope, and Methodology: 

Appendix II: NWS Prototype Plans, as of March 2007: 

Appendix III: Comments from the Department of Commerce: 

Appendix IV: GAO Contact and Staff Acknowledgments: 

Table: 

Table 1: Clustered Peer Concept of Operations Prototype Costs, as of 
March 2007: 

Figures: 

Figure 1: NWS's 122 Weather Forecast Offices: 

Figure 2: Overview of Key Systems and Technologies Supporting NWS 
Forecasts: 

Figure 3: An AWIPS Workstation: 

Figure 4: NWS Offices Involved in the Clustered Peer Concept of 
Operations Prototype: 

Figure 5: Preliminary Schedule for NWS's Clustered Peer Concept of 
Operations Prototype, as of November 14, 2006: 

Abbreviations: 

AWIPS: Advanced Weather Interactive Processing System: 

FAA: Federal Aviation Administration: 

NOAA: National Oceanic and Atmospheric Administration: 

NWS: National Weather Service: 

United States Government Accountability Office: 
Washington, DC 20548: 

June 8, 2007: 

The Honorable Nick Lampson: 
Chairman: 
The Honorable Bob Inglis: 
Ranking Member: 
Subcommittee on Energy and Environment: 
Committee on Science and Technology: 
House of Representatives: 

The Honorable David Wu: 
House of Representatives: 

The Honorable Vernon J. Ehlers: 
House of Representatives: 

The National Weather Service's (NWS) ability to forecast the weather 
affects the life and property of every American. The agency's basic 
mission is to provide storm and flood warnings and weather forecasts 
for the United States, its territories, and adjacent oceans and waters, 
in order to protect life and property and to enhance the national 
economy. NWS also supports other federal agencies' operations by 
providing aviation-and marine-related weather forecasts and warnings. 
To fulfill its mission, NWS operates 122 weather forecasting offices 
and other specialized weather centers across the country. In these 
offices, trained meteorologists use advanced systems to provide 
forecasts and warnings for local geographic areas. 

In an effort to improve its operational efficiency, in August 2005, NWS 
chartered a working group to evaluate the roles, responsibilities, 
functions, and supporting technology of weather forecast offices 
nationwide and to make a proposal for a more efficient concept of 
operations. In December 2005, the working group proposed an alternative 
way of operating in which weather offices could share some weather 
forecasting functions--particularly during high-intensity weather 
events. After receiving this proposal, NWS chartered another team to 
develop plans for a prototype of this concept to more fully evaluate 
this approach. 

Because of your interest in weather service operations, we (1) 
determined the current status of and plans for the prototype, (2) 
evaluated whether the justification for the prototype was sufficient, 
(3) determined whether NWS's plans to evaluate the prototype are 
adequate, (4) evaluated whether NWS is sufficiently involving 
stakeholders in its prototype plans, and (5) determined how NWS plans 
to ensure that there will be no degradation of service during and after 
the prototype. 

In late March 2007, after our audit work was nearly completed, the 
Department of Commerce's Under Secretary for Oceans and Atmosphere 
directed the Director of the National Weather Service to place all 
activities associated with the agency's concept of operations on hold 
pending a review by the Deputy Under Secretary. NWS officials stated 
that the prototype was suspended because of concerns raised about the 
agency's approach by some Members of Congress and by National Oceanic 
and Atmospheric Administration (NOAA) leadership, the private sector, 
NWS's workforce, and its critical emergency management partners. 
Shortly after this decision was made, we met with your staffs and 
agreed to continue with plans to issue this report to offer insight and 
recommendations to NWS, should it decide to proceed with the prototype. 

To address our objectives, we reviewed prototype plans and 
presentations. We compared program documents with best practices for 
justifying and evaluating prototypes and for involving stakeholders. We 
also interviewed internal and external stakeholders and agency 
officials. We conducted our work at NWS headquarters in the Washington, 
D.C., metropolitan area; NWS offices in Tampa, Florida, Detroit, 
Michigan, and Kansas City, Missouri; and a NOAA office in Boulder, 
Colorado, because of the relevance of these sites to the prototype. We 
performed our work from October 2006 to April 2007, in accordance with 
generally accepted government auditing standards. Additional details on 
our objectives, scope, and methodology are provided in appendix I. 

Results in Brief: 

NWS had begun the first phase of its three-phased prototype to 
demonstrate a new concept of operations in which weather offices would 
share selected forecasting responsibilities, when the Under Secretary 
for Oceans and Atmosphere suspended the prototype because of concerns 
about the agency's approach. As of March 2007, NWS had established a 
program manager and begun to document how the prototype would be 
executed and to develop training for employees who would participate. 
During the second and third phases, NWS planned to have 20 weather 
forecasting offices share forecasting responsibilities in 2-office 
pairs and then in 4-office clusters. At the conclusion of the 2-year 
prototype effort, NWS planned to make a decision on whether to proceed 
to implement the new concept of operations on a national basis. 
However, these efforts were halted when the agency suspended the 
prototype pending a reevaluation on how best to proceed. 

Before the prototype was suspended, NWS had approved moving forward 
with its prototype plan without conducting an analysis of the relative 
costs and benefits of the initiative. NWS estimated that the prototype 
would cost approximately $9.3 million and identified expected 
qualitative benefits--including increased efficiency, improved ability 
to focus on high-impact weather events, and the ability to switch to a 
backup site more quickly when a weather forecast office is disabled due 
to loss of power or communications. However, the agency has not 
conducted a cost-benefit analysis or determined whether there would be 
a return on investment for the prototype. If NWS were to proceed with 
the prototype without a cost-benefit analysis, it would lack assurance 
that its approach would be a cost-effective investment for the agency. 

NWS identified goals and selected measures to evaluate during its 
prototype activities, but it did not establish a rigorous evaluation 
plan. NWS's prototype plan identified evaluation goals, such as to 
validate that a group of weather forecast offices, operating as one 
unit, could be more effective and efficient than each office on its 
own. Also, the prototype plan identified quantitative and qualitative 
measures to be used in evaluating the prototype, including the number 
of times that work was transferred between offices to focus on high- 
impact events, the number of additional hours made available during 
benign weather for nonforecast work, and customer perceptions of 
service quality and timeliness. However, NWS did not establish an 
evaluation plan that identified a comprehensive set of applicable 
measures, what baseline performance its prototype would be compared 
with, and how its measures supported the goals of the prototype. If NWS 
were to proceed with its prototype without a rigorous evaluation plan, 
it would run an increased risk of not sufficiently measuring the impact 
of planned changes on its performance, and of subsequently making 
decisions affecting the nation's weather service on the basis of 
incomplete and flawed data. 

NWS involved internal stakeholders in planning its prototype, including 
having selected internal stakeholders review plans and provide feedback 
on work products and laboratory activities. However, NWS did not 
involve external stakeholders in its planning for the prototype or 
establish a plan for stakeholder involvement during the prototype. 
Specifically, NWS did not request input from external stakeholders, 
including emergency managers, in the development and planning of the 
prototype. In addition, the agency did not establish a plan that 
identified key external stakeholders or stakeholder representatives, 
determined the responsibilities of internal or external stakeholders, 
and established a time line for involving stakeholders and addressing 
their comments. If NWS were to proceed with its prototype without such 
a plan, the agency would be unable to ensure that employee and customer 
interests are anticipated, identified, and addressed. 

Prior to suspending the prototype, NWS stated that it would ensure that 
there would be no degradation of service resulting from the prototype, 
and that it had plans in place to mitigate the risk of degradation. To 
these ends, NWS planned to conduct multiple laboratory exercises to 
understand the impact of the prototype. Also, NWS planned to conduct 
the prototype on a dedicated workstation in each cluster office that 
was separate from existing systems, so that it could revert to normal 
operations should the need arise. In addition, NWS planned to monitor 
its systems and products under the prototype, so that the agency could 
revert to baseline systems if warranted. NWS officials stated that, 
should the agency decide to implement the prototype on a national 
basis, it has standard procedures for testing and validating systems 
and software to avoid any degradation of service before they are put 
into use. 

We are making recommendations to the Secretary of Commerce to ensure 
that if NWS decides to proceed with its prototype, it undertake a more 
rigorous approach to justifying, evaluating, and involving stakeholders 
in its prototype effort. We also are recommending that the agency 
evaluate moving forward with technology upgrades to allow offices to 
switch to backup service more quickly during high-impact weather 
events. 

Commerce provided written comments on a draft of this report in which 
it agreed with our recommendations, and stated that our findings were 
included in the Under Secretary's recent review of the concept of 
operations prototype. Based on that review, the NOAA Administrator 
directed NWS to cease all activities associated with the concept of 
operations prototype, and to undertake a comprehensive and analytical 
review to determine new and revised requirements for NWS products and 
services. A reprint of the department's comments is provided in 
appendix III. 

Background: 

The mission of NWS--an agency within Commerce's NOAA--is to provide 
weather, water, and climate forecasts and warnings for the United 
States, its territories, and its adjacent waters and oceans to protect 
life and property and to enhance the national economy. NWS is the 
official source of aviation-and marine-related weather forecasts and 
warnings as well as warnings about life-threatening weather situations. 

NWS Office Structure: An Overview: 

The coordinated activities of weather facilities throughout the United 
States allow NWS to deliver a broad spectrum of climate, weather, 
water, and space weather services. These facilities include weather 
forecast offices, river forecast centers, national centers, and 
aviation center weather service units. The functions of these 
facilities are described in the following text: 

* 122 weather forecast offices are responsible for providing a wide 
variety of weather, water, and climate services for their local county 
warning areas, including advisories, warnings, and forecasts. See 
figure 1 for a map of the locations of these offices. 

* 13 river forecast centers provide river, stream, and reservoir 
information to a wide variety of government and commercial users as 
well as to local weather forecast offices for use in flood forecasts 
and warnings. 

* 9 national centers constitute the National Centers for Environmental 
Prediction, which provide nationwide computer model output and manual 
forecast information to all NWS field offices and to a wide variety of 
government and commercial users. These centers include the 
Environmental Modeling Center, Storm Prediction Center, Tropical 
Prediction Center/National Hurricane Center, Climate Prediction Center, 
Aviation Weather Center, and Space Environment Center, among others. 

* 21 aviation center weather service units, which are colocated with 
key Federal Aviation Administration (FAA) air traffic control centers 
across the nation, provide meteorological support to air traffic 
controllers. 

Figure 1: NWS's 122 Weather Forecast Offices: 

[See PDF for image] 

Sources: NWS (data) and Map Resources (map). 

[End of figure] 

NWS Relies on Key Systems and Technologies to Fulfill Its Mission: 

To fulfill its mission, NWS relies on a national infrastructure of 
systems and technologies that gather and process data from the land, 
sea, and air. NWS collects data from many sources, including ground- 
based Automated Surface Observing Systems, the Next Generation Weather 
Radars, and operational environmental satellites. These data are 
integrated by advanced data processing workstations--called an Advanced 
Weather Interactive Processing System (AWIPS)--and used by 
meteorologists to issue local forecasts and warnings. Also, these data 
are fed into sophisticated computer models running on high-speed 
supercomputers, which are then used to help develop forecasts and 
warnings. Figure 2 depicts the integration of the various systems and 
technologies and is followed by a description of each. 

Figure 2: Overview of Key Systems and Technologies Supporting NWS 
Forecasts: 

[See PDF for image] 

Source: GAO. 

[End of figure] 

Automated Surface Observing System: 

The Automated Surface Observing System is a system of sensors, 
computers, display units, and communications equipment that automates 
the ground-based observation and dissemination of weather information 
nationwide. This system collects data on temperature, dew point, 
visibility, wind direction and speed, pressure, cloud height and 
amount, and types and amounts of precipitation. There are currently 
1,001 units deployed across the United States, with NWS, FAA, and the 
Department of Defense operating 312, 571, and 118 units, respectively. 

Next Generation Weather Radar: 

The Next Generation Weather Radar is a Doppler radar system that 
detects, tracks, and determines the intensity of storms and other areas 
of precipitation; determines wind velocities in and around detected 
storm events; and generates data and imagery to help forecasters 
distinguish hazards, such as severe thunderstorms and tornadoes. The 
radar system also provides information about heavy precipitation that 
leads to warnings about flash floods and heavy snow. A network of these 
radars is made up of 158 operational radar systems and 8 nonoperational 
systems that are used for training and testing. Of these, NWS, the 
United States Air Force, and FAA sponsor 120, 26, and 12 radars, 
respectively. 

Operational Environmental Satellites: 

Although NWS does not own or operate satellites, geostationary and 
polar-orbiting environmental satellites[Footnote 1] are key sources of 
data for its operations. These satellite systems continuously collect 
environmental data about Earth's atmosphere, surface, and cloud cover 
and the electromagnetic environment. These data are used by 
meteorologists to develop weather forecasts and other services, and are 
critical to the early and reliable prediction of severe storms, such as 
tornadoes and hurricanes. 

Advanced Weather Interactive Processing System: 

AWIPS is a data processing workstation that integrates and displays 
hydrometeorological data at NWS weather forecast offices, river 
forecast centers, and national centers. This system integrates data 
from a variety of sources (including the ground systems, radars, and 
satellites previously listed) to produce rich graphical displays to aid 
forecaster analysis and decision making. AWIPS includes a graphical 
forecast editing tool that allows forecasters to display and manipulate 
detailed graphical depictions of expected weather, and to use these 
depictions to generate text and graphical forecasts. AWIPS is used to 
disseminate weather information to the national centers; weather 
offices; the media; and other federal, state, and local government 
agencies. Figure 3 shows a standard AWIPS workstation. 

Figure 3: An AWIPS Workstation: 

[See PDF for image] 

Source: NOAA. 

[End of figure] 

Numerical Weather Models: 

Numerical weather models are advanced software programs that assimilate 
data from satellites and ground-based observing systems and provide 
short-and long-term weather pattern predictions. Meteorologists 
typically use a combination of models and their own experience to 
develop local forecasts and warnings. In addition, numerical weather 
models are a critical source for forecasting weather up to 2 weeks in 
advance and forecasting long-term climate changes. 

Supercomputers: 

NWS leases high-performance supercomputers to execute numerical 
calculations supporting weather prediction and climate modeling. In 
2002, NWS awarded a $227 million contract to lease high-performance 
supercomputers to run its environmental models from 2002 through 
September 2011. Included in this contract are an operational 
supercomputer used to run numerical weather models, an identical backup 
supercomputer located at a different site, and a research and 
development supercomputer on which researchers can test new analyses 
and models. 

Weather Forecast Office Operations: An Overview: 

NWS delivers a broad spectrum of climate, weather, water, and space 
weather services through the coordinated activities of the weather 
forecast offices; river forecast centers; and national centers, which 
comprise the National Centers for Environmental Prediction, aviation 
center weather service units, and other support offices around the 
country. Each weather forecast office currently has a fixed geographic 
area for which it provides local warnings and forecasts. These offices 
are staffed by meteorologists who produce a suite of weather products, 
including short-term and 3-to 5-day extended forecasts as well as 
aviation and marine weather products, and monitor for severe weather 
events. Predictive weather services (outlooks, forecasts, watches, 
warnings, advisories, and other supporting information) are produced 
through a process that incorporates observational data, numerical model 
output and statistical guidance, and human expertise. These services 
are delivered in many forms, including textual, digital, and graphical 
formats. Local offices and regional and national centers coordinate as 
needed in the preparation of these services. 

When a severe weather event occurs, forecast office managers often ask 
their staffs to work overtime so that there are enough personnel 
available to do both the normal forecasting work and the watches and 
warnings required by the severe event. If a weather forecast office is 
unable to provide forecast and warning functions due to an interruption 
in communications or power, an adjacent office will temporarily assume 
those duties. For example, during Hurricane Katrina, several weather 
forecast offices in Louisiana, Mississippi, and Alabama were damaged 
and inoperable. These offices' responsibilities were transferred to 
backup sites in Texas, Florida, Alabama, and Tennessee. During this 
transfer of operations, called service backup, the backup office gains 
access to the original office's system data and calls in extra 
personnel to handle the workload of both offices. 

According to NWS officials, due to software limitations in AWIPS, 
switching to a backup service arrangement can take 60 to 90 minutes, 
and, when invoked, the backup office must assume all products and 
services for the defunct office. In addition, the current backup 
arrangement can cause duplicate work. For example, one office could 
produce a forecast for its own geographic area, and then would need to 
produce a separate forecast for its backup partner. 

NWS Explored Options for a New Concept of Operations and Proposed One 
Option for Further Study: 

Over the past few years, NWS has evaluated options for an alternate 
concept of operations. According to agency officials, in 2005, NWS's 
board of executive advisors (called the Corporate Board)[Footnote 2] 
noted that a constrained budget, high labor costs, difficulty in 
training and developing its employees, and a lack of flexibility in how 
the agency was operating were making it more difficult for the agency 
to continue to perform its mission. In August 2005, the board chartered 
a working group to evaluate the roles, responsibilities, and functions 
of weather offices nationwide and to make a proposal for a more 
efficient concept of operations. The group was given a set of guiding 
principles, including that the proposed concept should (1) be cost- 
effective, (2) ensure that there would be no degradation of service, 
(3) ensure that weather services nationwide were equitable, and (4) not 
reduce the number of forecast offices nationwide. 

The group gathered input from various agency stakeholders and other 
partners within NOAA and considered multiple alternatives. They 
dismissed all but one of the alternative concepts because they were not 
consistent with the guiding principles. In its December 2005 proposal, 
the working group proposed the remaining concept, in which several 
offices with the same type of weather and warning responsibilities, 
climate, and customers would be grouped together to share 
responsibilities, particularly when there is a high-impact weather 
event affecting one of the offices. NWS called this a "clustered peer 
concept of operations." 

Under the clustered peer concept of operations, offices would share the 
workload associated with routine services, such as 7-day forecasts. 
During a high-impact weather event--such as a severe storm, flood, or 
wildfire--the offices would redistribute the workload to allow the 
impacted office to focus solely on the event, while the other offices 
in the cluster would pick up the impacted office's routine services. In 
addition, offices in a cluster could occasionally allocate the work so 
that certain offices would take on forecasting responsibilities while 
other offices focused on other tasks, such as training, customer 
outreach, and research. In this manner, peer offices could help 
supplement staffing needs, and the workload across multiple offices 
could be more efficiently balanced. 

Some of these benefits were demonstrated in recent years by a pair of 
weather forecasting offices. The Detroit and Grand Rapids offices 
established a technology workaround to allow them to more efficiently 
switch to backup operations between the two offices. This technology 
improvement allows them to switch to backup mode instantaneously, 
rather than taking 60 to 90 minutes using current AWIPS technology. To 
date, the two offices have used this backup capability to share 
workload during high-impact weather events; during system upgrades; and 
while conducting officewide training, research, and public outreach. 

NWS Developed a Prototype Plan for the Alternative Concept of 
Operations: 

In February 2006, NWS's Corporate Board chartered a team to develop a 
prototype plan for the clustered peer concept of operations. Between 
February and November 2006, the team established a preliminary concept 
for what the prototype would entail, conducted a laboratory exercise to 
determine AWIPS's ability to support the prototype, and--on the basis 
of the findings from the laboratory exercise--developed a prototype 
plan. 

The team's preliminary concept was for the prototype to consist of four 
clusters ranging in size from four to eight weather forecast offices. 
However, in September 2006, a laboratory exercise conducted between 
NOAA's Global Systems Division in Boulder and NWS's Central Region 
Headquarters in Kansas City showed that AWIPS was not capable of 
supporting clusters of more than four offices. The laboratory results 
showed that with minor modifications, AWIPS was capable of allowing a 
more efficient and timely switch of operations between two weather 
forecast offices for backup purposes, and that, in some cases, 
functions could be shared among four offices. Officials in charge of 
the laboratory exercise estimated that larger clusters would not be 
possible until the next version of AWIPS is deployed--a milestone 
currently planned for 2010. The laboratory exercise also identified 
several problems with system performance when weather forecast offices 
work as a cluster of two or more offices. Specifically, each weather 
forecast office has its own set of customized tools supporting each 
office's graphical forecast editor. The laboratory exercise found that 
running these tools when sharing functions among multiple offices 
caused AWIPS to slow down or crash. 

On the basis of the findings from the laboratory exercise, the 
prototype team developed a plan to demonstrate opportunities for 
improved efficiency. The goal of the prototype was to validate that a 
group of weather forecast offices, operating as one unit, could be more 
efficient than each office operating on its own. The plan entailed 
sharing forecasting responsibilities between 2, and subsequently among 
4, weather forecast offices using a single workstation in each office. 
The prototype was to be a hands-on demonstration conducted at 20 
weather forecast offices, clustered in 4 different areas of the 
country: the Pacific Northwest, Great Plains, Great Lakes, and 
Southeast (see fig. 4). These areas were selected because they present 
different forecasting challenges, involve all of the regions in the 
contiguous United States, and engage a variety of field offices. In 
November 2006, the Corporate Board approved moving forward with the 
prototype plan. 

Figure 4: NWS Offices Involved in the Clustered Peer Concept of 
Operations Prototype: 

[See PDF for image] 

Sources: NWS (data) and Map Resources (map). 

[End of figure] 

NWS Initiated a 2-Year Prototype and Then Suspended It: 

Following the Corporate Board's decision to proceed, NWS initiated the 
first of three phases of its prototype, which it planned to continue 
over the next 2 years. The first phase was expected to take place 
between November 2006 and June 2007 and was to include planning 
activities. As of March 2007, NWS had established a program manager and 
initiated efforts to plan for the prototype, including documenting how 
the prototype would be executed and developing training for employees 
who would participate in the prototype. 

During the second phase, which NWS expected to take place between 
August 2007 and February 2008, the agency planned to have 20 weather 
offices share forecasting responsibilities in 2-office pairs. During 
the third phase, which NWS expected to take place between April 2008 
and April 2009, the agency planned to have the same 20 weather offices 
share forecasting responsibilities in 4-office clusters. Each phase was 
to be preceded by both a laboratory exercise to test the new expanded 
functions and a board decision on whether to move ahead, based on the 
results of the laboratory exercise. NWS planned to conclude prototype 
operations and issue a report on the prototype by July 2009. By August 
2009, NWS intended to make a decision on whether to implement the new 
concept of operations on a national basis. NWS's schedule for the 
prototype is provided in figure 5. A more detailed description of the 
phases of the prototype is provided in appendix II. 

Figure 5: Preliminary Schedule for NWS's Clustered Peer Concept of 
Operations Prototype, as of November 14, 2006: 

[See PDF for image] 

Source: GAO analysis of NWS data. 

[End of figure] 

These efforts were halted when the prototype was suspended. On March 
23, 2007, Commerce's Under Secretary for Oceans and Atmosphere directed 
NWS's director to place all activities associated with NWS's concept of 
operations on hold pending a review by the Deputy Under Secretary. 
NWS's director subsequently issued a memorandum noting that the 
prototype was being suspended because of concerns about the agency's 
approach expressed by some Members of Congress and by NOAA leadership, 
the private sector, the agency's workforce, and the agency's emergency 
management partners. The director noted that the prototype would be put 
on hold until a reevaluation could be undertaken. 

NWS Has Not Yet Sufficiently Justified Its Prototype: 

Before the prototype was suspended, NWS developed and approved a 2-year 
plan for its prototype without conducting an analysis of the relative 
costs and benefits of the initiative. Leading organizations have 
developed best practices for justifying prototyping efforts--including 
analyzing the expected costs and quantified benefits of alternatives 
before beginning prototyping activities--and identifying plans, 
schedules, and risks.[Footnote 3] Furthermore, federal 
guidance[Footnote 4] notes that in conducting a cost-benefit 
assessment, tangible and intangible benefits and costs should be 
identified, assessed, and reported. 

The prototype plan approved by the Corporate Board in November 2006 
identifies the schedules, risks, and expected benefits of the 
prototype. The prototype was estimated to cost $9.3 million, and the 
Corporate Board's financial investment review committee approved 
spending $1.5 million on the prototype in the first half of fiscal year 
2007. As of March 2007, NWS had spent approximately $724,000 on the 
prototype (see table 1). Moreover, the prototype plan identifies 
qualitative benefits of the new concept of operations--including 
increased efficiency and time for other activities (such as outreach 
and research), the ability to switch to a backup site more quickly when 
a weather forecast office is disabled due to loss of power or 
communications, and an improved ability to focus on high-impact events. 

Table 1: Clustered Peer Concept of Operations Prototype Costs, as of 
March 2007: 

Fiscal year: 2005; 
Approved expenditures: $4,125; 
Actual expenditures: $4,125; 
Estimated expenditures[A]: --; 
Total: $4,125. 

Fiscal year: 2006; 
Approved expenditures: 430,879; 
Actual expenditures: 430,879; 
Estimated expenditures[A]: --; 
Total: 430,879. 

Fiscal year: 2007; 
Approved expenditures: 1,500,000; 
Actual expenditures: 288,907; 
Estimated expenditures[A]: $2,911,093; 
Total: 3,200,000. 

Fiscal year: 2008; 
Approved expenditures: [Empty]; 
Actual expenditures: --; 
Estimated expenditures[A]: 3,000,000; 
Total: 3,000,000. 

Fiscal year: 2009; 
Approved expenditures: [Empty]; 
Actual expenditures: --; 
Estimated expenditures[A]: 2,700,000; 
Total: 2,700,000. 

Fiscal year: Total; 
Approved expenditures: $1,935,004; 
Actual expenditures: $723,911; 
Estimated expenditures[A]: $8,611,093; 
Total: $9,335,004. 

Source: GAO analysis of NWS data. 

[A] NWS has not yet approved these estimates. 

[End of table] 

However, NWS did not conduct a cost-benefit analysis to justify its 
plans for the prototype or the $1.9 million that has already been 
approved for the prototype. 

By not conducting a cost-benefit analysis before beginning the 
prototype, NWS lacked assurance that its approach was cost-effective. 
If NWS decides to proceed with the prototype without conducting a cost- 
benefit analysis, the agency will have little basis to ensure that its 
investment is sound or that the options it is undertaking provide the 
greatest return on investment. Specifically, a cost-benefit analysis 
could help ascertain which aspects of the prototype promise the most 
benefits and enable NWS to adjust its plans accordingly. For example, a 
cost-benefit assessment might show that more benefits could be gained 
for the least cost by moving to a 2-office cluster, rather than a 4- 
office cluster. 

NWS Lacks a Rigorous Evaluation Plan for Assessing Its Prototype: 

In its prototype plan, NWS identified goals and selected measures to 
evaluate during its prototype activities, but it did not establish a 
rigorous evaluation plan. Specifically, the agency had not defined a 
comprehensive set of measures, how it planned to compare prototype 
results with baseline performance, or how its measures supported the 
goals of the prototype. 

According to best practices in leading organizations, well-developed 
evaluation plans increase the likelihood that evaluations will yield 
sound results, thereby supporting effective policy decisions. Key 
features of a well-developed evaluation plan include well-defined, 
clear, and measurable objectives or goals; quantitative measures for 
determining program performance as compared with baseline performance; 
and a linkage between the measures and the program goals. Such plans 
are essential to ensuring, among other things, that programs are 
executed properly and that funding requests are developed 
appropriately. In July 2006, we reported that NWS had not yet 
established plans, time lines, and measures for evaluating its 
prototype, and we recommended that NWS do so before beginning the 
prototype.[Footnote 5] 

NWS's prototype plan identifies evaluation goals and measures, and 
notes that NWS plans to contract with an independent entity to perform 
the evaluation. The prototype goals were to (1) validate that a group 
of weather forecast offices, operating as one unit, could be more 
effective and efficient than each office on its own; (2) increase focus 
during high-impact events through more efficient resource allocation; 
(3) gain efficiencies through the use of resource allocation during 
benign weather that can be applied to training, outreach, and 
professional development; and (4) use some of the efficiencies gained 
through the prototype to further the NOAA mission. Quantitative and 
qualitative measures that were to be used in evaluating the prototype 
included the number of times that work was transferred to another 
forecast office to focus on high-impact events, the number of 
additional hours made available during benign weather for nonforecast 
work, and customer perceptions of service quality and timeliness. The 
agency stated that system performance and hardware/software 
functionality would have been evaluated using staff feedback, 
performance monitoring software, and usage logs, while customer 
perceptions would have been evaluated using customer satisfaction 
surveys. 

However, NWS did not establish an evaluation plan that identifies a 
full set of applicable measures, what baseline performance its 
prototype would be compared with, and how its measures would support 
the goals of the prototype. Specifically, while the agency listed 
various items that it planned to measure in its prototype plan, it did 
not include quantitative measures of product quality, timeliness, the 
impact on the efficiency of the office that is receiving the increased 
workload, or plans to assess the sensitivity of its results to weather- 
related situations. In addition, the agency did not establish the 
baseline performance against which it would compare its prototype, so 
it was not clear how prototype results would be assessed. Furthermore, 
the measures that NWS identified did not directly support the goals. 
For example, the number of times functions are transferred to another 
site does not demonstrate increased efficiency. For this measure to 
support the desired goal of increased efficiency, the measure would 
need to be supplemented with an assessment of how the office that 
transferred its workload used the newly available time and how the 
receiving site handled its increased workload. 

If NWS were to proceed with its prototype without a rigorous evaluation 
plan, the agency would run an increased risk that it could not 
sufficiently measure the impact of planned changes on its performance, 
and would subsequently make decisions affecting the nation's weather 
service on the basis of incomplete and flawed data. 

NWS Involved Internal Stakeholders in Planning the Prototype, but Did 
Not Involve External Stakeholders or Establish a Plan for Stakeholder 
Involvement: 

NWS involved internal stakeholders in planning its prototype, including 
having selected stakeholders review plans and provide feedback on work 
products and laboratory activities. However, NWS did not involve 
external stakeholders or establish a plan for stakeholder involvement 
throughout the prototype. Leading organizations routinely involve 
relevant stakeholders when considering operational or process 
changes.[Footnote 6] In addition, NWS has agreed to negotiate with its 
employees' union, the National Weather Service Employees Organization, 
whenever organizational changes could affect working 
conditions.[Footnote 7] Moreover, best practices call for developing a 
plan for stakeholder involvement that includes identifying relevant 
stakeholders, the roles and responsibilities of the relevant 
stakeholders, and a schedule for stakeholder involvement throughout the 
project. 

In planning its prototype, NWS involved internal stakeholders, who were 
identified by the NWS Director of Strategic Plans as forecasters, other 
field and headquarters staff, and union representatives. For example, 
when deciding which offices to group into clusters, meteorologists-in- 
charge were asked for their input on cluster criteria and preferences. 
Furthermore, one meteorologist from each of eight potential prototype 
weather forecast offices participated in the laboratory test. Their 
feedback on the new concept of operations was acquired through 
questionnaires, electronic logs, and debriefings at the end of each 
week. In addition, the employees' union provided input before the 
decision was made to begin the prototype. Both the initial working 
group and the prototype team included union representatives. Also, the 
union president participated in Corporate Board meetings that included 
discussions of the clustered peer prototype, and a union representative 
observed the final week of the laboratory test. 

NWS also intended to work with external stakeholders (including 
emergency managers and other public safety officials, the media and 
other users of NWS information services, and the general public) after 
the prototype was complete to assess the results. Specifically, NWS 
planned to identify customer perceptions of service as one basis on 
which to evaluate the prototype. Furthermore, in an effort to make its 
plans available to employees and customers, NWS developed a public Web 
site that contains the plans and reports for the prototype.[Footnote 8] 

However, NWS did not request input from external stakeholders, 
including emergency managers, during the past 20 months that it was 
planning and initiating the prototype. NWS officials reported that they 
did not involve external stakeholders because changes in the concept of 
operations during the prototype should not be noticeable to outside 
stakeholders; that is, external stakeholders should not notice any 
difference in service. Although this goal has merit, external 
stakeholders need to be more involved in the prototype for it to gain 
acceptance and succeed. Specifically, external stakeholders need to 
understand what is being tested, have input into the tests, be alert 
for differences under the prototype, and have formal mechanisms for 
sharing their observations. In addition, NWS did not have a 
comprehensive plan for involving stakeholders in its prototyping 
activities. Specifically, it did not identify key external stakeholders 
or stakeholder representatives, determine the responsibilities of 
internal or external stakeholders, or develop a time line for involving 
stakeholders and addressing their comments. 

If NWS were to proceed with its prototype without a plan for involving 
internal and external stakeholders, it would not be able to ensure that 
employee and customer interests are anticipated, identified, and 
addressed. 

NWS Planned to Ensure That the Prototype Would Not Degrade Its Service: 

NWS stated that it would ensure that there would be no degradation of 
service during or after its prototype, and that it had plans in place 
to mitigate the risk of any service degradation. Specifically, the 
agency conducted and planned to continue to conduct laboratory 
exercises to assess the impact of the prototype before moving into each 
phase. In addition, NWS planned to run the prototype software on a 
single, isolated workstation in each office, while maintaining the 
existing AWIPS software on all other workstations. Established 
procedures would allow offices to return to the baseline software and 
operational practices should problems have developed. Furthermore, NWS 
planned to monitor the performance of all activities and the quality 
and timeliness of products. 

Looking forward, an NWS official stated that they also would ensure 
that there would be no degradation of service if the Corporate Board 
decided to implement this alternative concept of operations on a 
national basis after the prototype was completed. Agency officials 
explained that the agency would use its standard procedures for testing 
and validating systems and software to ensure that there would be no 
degradation of service before moving them into operations. 

Conclusions: 

Although there is merit in NWS's goal of improving its operational 
efficiency and allowing faster backup service during severe weather, 
the agency lacked the rigorous planning it needed to justify, evaluate, 
and obtain stakeholder involvement in its planned--and now suspended-- 
prototype of a new concept of operations. Specifically, the agency 
developed a prototype plan that identified activities, gates, and 
decisions, but it did not perform a cost-benefit assessment to justify 
the prototype. Also, NWS identified goals and selected measures for 
evaluating its prototype, but it did not establish an evaluation plan 
that clearly identifies a comprehensive set of measures or how these 
measures would be compared with baseline performance. Furthermore, the 
agency involved internal stakeholders in planning for its prototype, 
but it did not seek input from external stakeholders or establish a 
plan for stakeholder involvement throughout the prototype. These 
matters will be important to consider should NWS decide to proceed with 
the prototype. Without undertaking a more rigorous approach, NWS would 
run an increased risk that its prototype will not be a sound 
investment, and that any resulting decisions could be based on flawed 
data and analysis. Taking this more rigorous approach to the prototype 
should not preclude NWS from considering options for improving the 
ability to switch to a backup site when a weather forecast office 
becomes disabled due to loss of power or communications. 

Recommendations for Executive Action: 

We are making four recommendations to the Secretary of Commerce. If NWS 
decides to proceed with its prototype, we recommend the Secretary 
direct the Assistant Administrator for the National Weather Service to: 

* develop a cost-benefit analysis for the clustered peer approach 
before implementing the prototype in any weather forecasting office; 

* develop an evaluation plan that includes a comprehensive set of 
measures that are linked to prototype goals and identifies the baseline 
performance that the prototype will be compared with before 
implementing the prototype in any weather forecasting office; and: 

* develop a plan for internal and external stakeholder involvement, 
which includes a list of relevant stakeholders, roles and 
responsibilities for these stakeholders, and a schedule for when 
stakeholders should be involved before implementing the prototype in 
any weather forecasting office. 

In addition, we recommend that the Secretary direct the Assistant 
Administrator for the National Weather Service to evaluate moving 
forward with technology upgrades to the AWIPS processing system to 
allow weather forecast offices to switch to backup service more quickly 
during high-impact weather situations; these upgrades could be handled 
as an initiative that is separate and distinct from broader changes in 
the agency's concept of operations. 

Agency Comments and Our Evaluation: 

We received written comments on a draft of this report from Commerce 
(see app. III). In the department's response, the Deputy Secretary of 
Commerce agreed with our recommendations to conduct a thorough cost- 
benefit analysis, develop a rigorous evaluation plan with fully defined 
performance measures, and involve internal and external stakeholders. 
The response stated that our findings were included in the Under 
Secretary's recent review of the concept of operations prototype. On 
the basis of that review, the NOAA Administrator directed NWS to cease 
all activities associated with the concept of operations prototype, and 
to undertake a comprehensive and analytical review to determine new and 
revised requirements for NWS products and services. This review is 
expected to determine what, if any, changes are required. Should NWS 
decide to proceed with a new concept of operations, we believe that our 
recommendations to justify, evaluate, and involve stakeholders in any 
such changes would still be relevant and useful. 

In addition, Commerce agreed with our recommendation to evaluate moving 
forward with technology upgrades to the AWIPS processing system to 
allow weather forecast offices to switch to backup service more quickly 
during high-impact weather situations. The department stated that NWS 
has established a proposal for improving its service backup capability, 
expects to approve requirements for the backup capability by September 
2007, and plans to deploy this capability by September 2008. 

As we agreed with your offices, unless you publicly announce the 
contents of this report earlier, we plan no further distribution of it 
until 30 days from the report date. At that time, we will send copies 
of this report to interested congressional committees, the Secretary of 
Commerce, the Director of the Office of Management and Budget, and 
other interested parties. In addition, this report will be available at 
no charge on our Web site at www.gao.gov. 

If you have any questions about this report, please contact me at (202) 
512-9286 or by e-mail at pownerd@gao.gov. Contact points for our 
Offices of Congressional Relations and Public Affairs may be found on 
the last page of this report. GAO staff who made major contributions to 
this report are listed in appendix IV. 

Signed by: 

David A. Powner: 
Director, Information Technology Management Issues: 

[End of section] 

Appendix I: Objectives, Scope, and Methodology: 

Our objectives were to (1) determine the current status of and plans 
for the prototype, (2) evaluate whether the justification for the 
prototype is sufficient, (3) determine whether the National Weather 
Service's (NWS) plans to evaluate the prototype are adequate, (4) 
evaluate whether NWS is sufficiently involving stakeholders in its 
prototype plans, and (5) determine how NWS plans to ensure that there 
will be no degradation of service during and after the prototype. 

To determine the current status of and plans for the prototype, we 
reviewed agency documentation on the planned prototype's goals, major 
components, and milestones, such as the Corporate Board's meeting 
minutes, the prototype plan, and the preprototype laboratory report. We 
also interviewed NWS officials to obtain clarifications on agency 
plans. 

To evaluate whether the justification for the prototype is sufficient, 
we reviewed program documents such as the prototype plan and other NWS 
planning documents, presentations, and communications about the goals 
and costs of the prototype. We compared these planning documents with 
best practices for justifying process-improvement proposals by the 
Carnegie Mellon University Software Engineering Institute's Capability 
Maturity Model®Integration for Development[Footnote 9] and federal 
guidance on conducting cost-benefit analyses.[Footnote 10] We also 
interviewed agency officials and employees who were involved in the 
planning of the prototype and laboratory exercise to determine the 
potential costs, benefits, and risks of the prototype. 

To determine whether NWS plans to evaluate the prototype are adequate, 
we reviewed planning documents, such as the prototype plan and NWS 
customer satisfaction scores, and compared them with best practices for 
evaluating prototypes from the Capability Maturity Model®Integration 
for Development and GAO's guidance for developing and assessing 
evaluation plans.[Footnote 11] In addition, we interviewed NWS 
officials who were responsible for planning the prototype activities. 

To evaluate whether NWS is sufficiently involving stakeholders in its 
prototype plans, we reviewed NWS's plans, presentations, and other 
communications about the prototype, including an external Web site that 
presented these documents. We compared NWS's plans for involving 
stakeholders with best practices from the Capability Maturity Model® 
Integration for Development. We spoke with members of stakeholder 
groups to determine their knowledge of the prototype and their 
interaction with NWS. In addition, we interviewed NWS officials and the 
president of the National Weather Service Employees Organization to 
determine the level of involvement of the employee union in planning 
for the prototype. 

To determine how NWS plans to ensure that there will be no degradation 
of service during and after the prototype, we reviewed planning 
documents--including the prototype plan, presentations, and 
communications regarding the prototype--and interviewed NWS officials. 

We performed our work at NWS headquarters in the Washington, D.C., 
metropolitan area; at NWS offices in Tampa, Florida, Detroit, Michigan, 
and Kansas City, Missouri; and at a National Oceanic and Atmospheric 
Administration (NOAA) office in Boulder, Colorado. We visited the Tampa 
weather forecast office because the president of the employee union and 
the vice president of the International Association of Emergency 
Managers are located in the Tampa area. The Detroit weather forecast 
office was included because of its prior experiences in working in a 
paired backup with expanded domains. We visited the NWS Central Region 
Headquarters in Kansas City because of its participation in the first 
laboratory exercise and because it is where the prototype team chair 
worked. The NOAA Global Systems Division office in Boulder was chosen 
because of its participation in the first laboratory exercise. We 
performed our work from October 2006 to April 2007 in accordance with 
generally accepted government auditing standards. 

[End of section] 

Appendix II: NWS Prototype Plans, as of March 2007: 

Prior to the suspension of the prototype by the Director of the 
National Weather Service, the prototype team developed a plan for a 
three-phased prototype of the clustered peer concept of operations. As 
described in the plan, phase 1 would include planning activities, such 
as training; phase 2 was meant to establish clustered peer operations 
in 10 pairs of weather forecast offices across the country; and phase 3 
would include offices grouped into 4-office clusters. NWS planned to 
conclude prototype operations and to issue a report on the prototype by 
July 2009. NWS was to decide at that time if it would proceed to 
implement the clustered peer concept of operations on a national basis. 
Details about each of the planned prototype phases are included in the 
following text. 

Phase 1, the planning phase, began in November 2006 with the Corporate 
Board's approval of the prototype plan. NWS established a program 
manager to oversee the prototype. In addition, cluster management 
teams[Footnote 12] representing each cluster were to meet to decide on 
the rules of engagement, common tools, and common forecast 
methodologies. At the end of this phase, a second laboratory exercise 
was planned to validate that software upgrades and hardware additions 
to support the prototype were operationally ready. The second 
laboratory exercise was scheduled for June 2007 and was to involve only 
weather forecast offices that are currently service-backup pairs within 
the original four geographic cluster areas. Upon completion of this 
exercise, the Corporate Board was to decide on whether to initiate 
Phase 2 of the prototype. 

Phase 2 was planned to begin in August 2007. During this phase, NWS 
planned to conduct training, develop guidance for how and when offices 
would take over activities for other offices (called playbooks), define 
the cluster methodology, and have 10 pairs of offices operate as 
clustered peers. Also during this phase, NWS planned to develop the 
software upgrades needed for expanded operations in the next phase. 
Near the end of this phase, in February 2008, NWS planned to conduct a 
third laboratory exercise to test the software necessary for moving to 
clusters of 4 offices. After completion of this exercise, the Corporate 
Board was to decide on whether to initiate Phase 3 of the prototype. 

Phase 3 was planned to begin in April 2008 and was to determine if 
clusters of 4 offices can work as a multioffice team sharing 
forecasting responsibilities. NWS planned to complete a preliminary 
prototype report in July 2009 and present it to the Corporate Board. At 
that time, the board would have decided whether to implement the 
clustered peer concept of operations across the country. 

[End of section] 

Appendix III: Comments from the Department of Commerce: 

The Deputy Secretary Of Commerce: 
Washington, D.C. 20230: 

May 16, 2007: 

Mr. David A. Powner: 
Director, Information Technology Management Issues: 
U.S. Government Accountability Office: 
441 G Street, NW: 
Washington, D.C. 20548: 

Dear Mr. Powner: 

Thank you for the opportunity to review and comment on the Government 
Accountability Office's draft report entitled Weather Forecasting: 
National Weather Service's Operations Prototype Needs More Rigorous 
Planning (GAO-07-650). On behalf of the Department of Commerce, I 
enclose the National Oceanic and Atmospheric Administration's 
programmatic comments on the draft report. 

Sincerely, 

Signed by: 

David A. Sampson: 

Enclosure: 

Department of Commerce Comments on the Draft GAO Report Entitled 
"Weather Forecasting: National Weather Service's Operations Prototype 
Needs More Rigorous Planning" (GAO-07-650/June 2007): 

General Comments: 

The Department of Commerce's National Oceanic and Atmospheric 
Administration (NOAA) appreciates the opportunity to review this draft 
report. We are pleased GAO acknowledges the importance of continuing 
upgrades to the Advanced Weather Interactive Processing System (AWIPS) 
and associated services' backup capabilities. These upgrades are 
critical to ensure the National Weather Service's (NWS) continuity of 
services during high impact events, and to continue to help meet the 
Nation's growing demand for improved weather, water, and climate 
information. 

NOAA agrees with the need to conduct a thorough cost benefit analysis, 
involve internal and external stakeholders, and develop a rigorous 
evaluation plan with fully defined performance measures to assess 
impact of the concept of operation and prototype. These conclusions 
were part of the findings and conclusions identified in the recent 
review led by NOAA's Deputy Under Secretary. Based on the review, the 
NWS has been directed to cease all activities associated with the 
concept of operation and concept of operation prototype, and to 
undertake a comprehensive and analytical review to determine emerging 
new/revised requirements for products and services. This review will 
consider internal NOAA and external user requirements, and involve the 
external user community. The results of this review will determine what 
if any, changes are required. 

NOAA Response to GAO Recommendations: 

The draft GAO report states, "We are making four recommendations to the 
Secretary of Commerce. If NWS decides to proceed with its prototype, we 
recommend the Secretary direct the Assistant Administrator for the 
National Weather Service to:" 

Recommendation 1: "Develop a cost-benefit analysis for the clustered 
peer approach before implementing the prototype in any weather 
forecasting office." 

NOAA Response: NWS has been directed by the NOAA Administrator to cease 
all activities associated with the concept of operation and concept of 
operation prototype. 

Recommendation 2: "Develop an evaluation plan that includes a 
comprehensive set of measures that are linked to prototype goals and 
identifies the baseline performance that the prototype will be compared 
to before implementing the prototype in any weather forecasting 
office." 

NOAA Response: NWS has been directed by the NOAA Administrator to cease 
all activities associated with the concept of operation and concept of 
operation prototype. 

Recommendation 3: "Develop a plan for internal and external stakeholder 
involvement, which includes a list of relevant stakeholders, roles and 
responsibilities for these stakeholders, and a schedule for when 
stakeholders should be involved before implementing the prototype in 
any weather forecasting office." 

NOAA Response: NWS has been directed by the NOAA Administrator to cease 
all activities associated with the concept of operation and concept of 
operation prototype. 

Recommendation 4: "In addition, we recommend that the Secretary direct 
the Assistant Administrator for the National Weather Service to: 
Evaluate moving forward with technology upgrades to the AWIPS 
processing system to allow weather forecast offices to switch to backup 
service more quickly during high-impact weather situations; these 
upgrades could be handled as an initiative that is separate and 
distinct from broader changes in the agency's concept of operations." 

NOAA Response: NOAA concurs with this recommendation. NWS has 
established a Statement of Need for a project within its internal 
Operations and Services Improvement Process (OSIP) for an improved 
service backup capability (OSIP 07-20, Service Backup Improvements). 
Service backup requirements are scheduled to be approved by September 
30, 2007. Targeted delivery of Phase 1, service backup improvements in 
the AWIPS software in Operational Build 8.3 are scheduled to be 
deployed by September 30, 2008. 

[End of section] 

Appendix IV: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

David A. Powner, (202) 512-9286 or pownerd@gao.gov: 

Staff Acknowledgments: 

In addition to the contact person named above, Colleen Phillips, 
Assistant Director; Kate Agatone; Neil Doherty; Amos Tevelow; and 
Jessica Waselkow made key contributions to this report. 

FOOTNOTES 

[1] Geostationary environmental satellites orbit above Earth's surface 
at the same speed as Earth rotates, so that each satellite remains over 
the same location on Earth. Polar satellites constantly circle Earth in 
an almost north-south orbit, providing global coverage of conditions 
that affect the weather and climate. 

[2] NWS's Corporate Board is chaired by the Director of the National 
Weather Service and composed of senior officials responsible for 
different aspects of the agency's mission, including the Chief 
Information Officer and the Directors of the Office of Climate, Water, 
and Weather Services; the National Centers for Environmental 
Prediction; and the NWS regions. The board meets at least twice 
annually to discuss NWS's budget and strategic issues. It also holds 
meetings, as needed, to focus on special topics, such as an assessment 
of weather services provided during Hurricane Charley in 2004. 

[3] Carnegie Mellon University Software Engineering Institute, 
Capability Maturity Model® Integration for Development, Version 1.2 
(Pittsburgh, PA: August 2006). The Carnegie Mellon University Software 
Engineering Institute, recognized for its expertise in software and 
system processes, has developed the Capability Maturity Model® 
Integration for Development to evaluate, improve, and manage system and 
software development processes. 

[4] Office of Management and Budget, Guidelines and Discount Rates for 
Benefit-Cost Analysis of Federal Programs, OMB Circular A-94 
(Washington, D.C.: Oct. 29, 1992). 

[5] GAO, Weather Forecasting: National Weather Service Is Planning to 
Improve Service and Gain Efficiency, but Impacts of Potential Changes 
Are Not Yet Known, GAO-06-792 (Washington, D.C.: July 14, 2006). 

[6] GAO, Information Technology Investment Management: A Framework for 
Assessing and Improving Process Maturity, GAO-04-394G (Washington, 
D.C.: March 2004); and Capability Maturity Model®Integration for 
Development, Version 1.2. 

[7] NWS, National Collective Bargaining Agreement Between The National 
Weather Service and The National Weather Service Employees Organization 
(Oct. 25, 2001). This agreement calls for NWS to negotiate with the 
union on the impact and implementation of changes affecting working 
conditions before they can be implemented. The agreement also allows 
the agency to avoid the need for negotiations if the union has 
sufficient predecisional involvement. 

[8] NWS, Focus on the Future (Silver Spring, MD: Apr. 5, 2007), 
hyperlink, http://www.weather.gov/com/digitalera/ (viewed on Apr. 10, 
2007). 

[9] Carnegie Mellon University Software Engineering Institute, 
Capability Maturity Model® Integration for Development, Version 1.2 
(Pittsburgh, PA: August 2006). Capability Maturity Model® and 
Capability Maturity Modeling are registered in the U.S. Patent and 
Trademark Office. CMM is a service mark of Carnegie Mellon University. 

[10] Office of Management and Budget, Guidelines and Discount Rates for 
Benefit-Cost Analysis of Federal Programs, OMB Circular A-94 
(Washington, D.C.: Oct. 29, 1992). 

[11] GAO, Equal Employment Opportunity: DOD's EEO Pilot Program Under 
Way, but Improvements Needed to DOD's Evaluation Plan, GAO-06-538 
(Washington, D.C.: May 5, 2006); and Earned Income Tax Credit: 
Implementation of Three New Tests Proceeded Smoothly, But Tests and 
Evaluation Plans Were Not Fully Documented, GAO-05-92 (Washington, 
D.C.: Dec. 30, 2004). 

[12] The cluster management team provides operational oversight to the 
cluster. The team is composed of the meteorologists-in-charge from the 
cluster weather forecast offices, hydrologists-in-charge from the river 
forecast centers, and a regional representative. 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site (www.gao.gov). Each weekday, GAO posts 
newly released reports, testimony, and correspondence on its Web site. 
To have GAO e-mail you a list of newly posted products every afternoon, 
go to www.gao.gov and select "Subscribe to Updates." 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office 441 G Street NW, Room LM 
Washington, D.C. 20548: 

To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202) 
512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400 U.S. 
Government Accountability Office, 441 G Street NW, Room 7125 
Washington, D.C. 20548: 

Public Affairs: 

Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800 
U.S. Government Accountability Office, 441 G Street NW, Room 7149 
Washington, D.C. 20548: