This is the accessible text file for GAO report number GAO-04-379R 
entitled 'Defense Acquisitions: Risks Posed by DOD's New Space Systems 
Acquisition Policy' which was released on January 29, 2004.

This text file was formatted by the U.S. General Accounting Office 
(GAO) to be accessible to users with visual impairments, as part of a 
longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov.

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately.

January 29, 2004:

The Honorable Wayne Allard:

Chairman:

The Honorable Bill Nelson:

Ranking Minority Member:

Subcommittee on Strategic Forces:

Committee on Armed Services:

United States Senate:

Subject: Defense Acquisitions: Risks Posed by DOD's New Space Systems 
Acquisition Policy:

On November 18, 2003, we testified before the Subcommittee on the 
Department of Defense's (DOD) new acquisition policy for space systems. 
The new acquisition policy, issued in October 2003, sets the stage for 
decision making for DOD's investment in space systems, which currently 
stands at more than $18 billion annually and is expected to grow 
considerably over the next decade. You requested that we provide 
additional comments on several issues relating to the new policy and 
other space acquisition issues. Your specific questions and our answers 
are discussed below.

1. The Air Force maintains that its Defense Space Acquisition Board 
(DSAB) process allows earlier identification of problems and senior 
level attention, which will improve management and lower risk. Does 
GAO have any concerns with the DSAB process?

Our concern is not with earlier identification of problems or the added 
senior level attention the new process calls for, but with earlier 
investment decisions, which are also called for. Under the new process, 
the DSAB may approve product development to begin before DOD knows 
whether technologies can work as intended. As a result, it will make 
major investment commitments without really knowing what resources will 
be required to deliver promised capability. Moreover, the policy 
encourages development of leading-edge technology within product 
development, that is, at the same time the program manager is designing 
the system and undertaking other product development activities. DOD 
believes this approach will allow space systems to better incorporate 
leading-edge technologies. But as our work has repeatedly shown, such 
concurrency within space and other weapon system programs increases the 
risk that significant problems will be discovered as the system is 
integrated and built, when it is more costly and time-consuming to fix 
them.

Moreover, as we testified, the knowledge-building approach for space 
stands in sharp contrast to that followed by successful programs and 
the approach recommended by DOD's revised acquisition policy for weapon 
systems. Successful programs will not commit to undertaking product 
development unless they have high confidence that they have achieved a 
match between what the customer wants and what the program can deliver. 
Technologies that are not mature continue to be developed in an 
environment that is focused solely on technology development. This 
system puts programs in a better position to succeed because they can 
focus on design, system integration, and manufacturing. Further, our 
work has shown that taking an evolutionary approach to improving 
capability increases the likelihood of delivering that capability to 
the war fighter sooner than the revolutionary approach the Air Force 
continues to support in the new space policy.

2. Does GAO believe that the process put into place in the new space 
acquisition policy by which cost estimates are derived will provide 
better cost estimates?

No. Although some process changes will be made, the underlying causes 
of underestimating costs remain.

DOD is adopting new methodologies and tools to enhance cost estimates, 
and it is enlisting assistance from DOD's Cost Analysis Improvement 
Group (CAIG) to conduct independent cost estimates using cost 
estimating teams drawn from a broad spectrum of the cost-estimating 
community. Moreover, programs are now required to resolve differences 
between their cost estimates and estimates produced by the independent 
teams. In the past, cost-estimating groups have developed estimates 
that were different, leaving decision makers to select one estimate or 
combine a few.

However, under the new space acquisition policy, cost estimates do not 
have to be based on the knowledge that technologies can work as 
intended. History has shown that cost estimates not based on such 
knowledge are significantly understated. Moreover, incentives that work 
against providing good estimates have not changed. Unlike the 
commercial world where the focus is on delivering a product to market, 
DOD's system focuses on competing for resources from oversubscribed 
budgets. In the competition for funding, managers are encouraged to 
launch product developments before technologies are mature. Because 
funding is competitive and DOD's forecasts of costs, schedules, and 
performance are largely based on immature technologies and other 
unknowns, estimates tend to be squeezed into insufficient profiles of 
available funding. In fact, pressures to underestimate costs may 
increase over the next decade as DOD plans to undertake a number of 
new, challenging space programs--which are expected to require an 
additional $4 billion in the next 4 years alone. Costs beyond that 
period are as yet unknown but are likely to be considerably higher.

3. What is GAO's view on the Air Force policy related to full funding? 

DOD's acquisition policy for other weapon systems requires a commitment 
for full funding at milestone B--the start of product development and 
the point at which DOD should have knowledge that technologies can work 
as intended. However, the new space acquisition policy does not require 
DOD to commit to fully fund a space program either when this knowledge 
has been obtained or at any point in the development process. Hence, 
there is no guarantee that the resources needed to meet requirements on 
any individual program will be there when needed--particularly as DOD 
moves forward with its new programs.

This represents another important departure from the development 
approach followed by successful programs. Our prior work[Footnote 1] 
has found that if a product's business case measures up, that is a 
company is assured that there is a market or need for the product and 
that it has the right knowledge in hand to develop the product with 
firm cost and schedule estimates, the company then commits to the 
entire development of the product, including the financial investment. 
In other words, corporate resources are made available to the 
development team so that product success is not compromised. As noted 
earlier, because DOD begins too many programs, its resources are always 
oversubscribed. By requiring program managers to continually justify 
funding, DOD runs a risk of foreclosing the ability for sound planning 
and execution.

4. The Young Panel was not convinced of the merits of competition in 
some circumstances, particularly when the incumbent has performed well 
and "owns" the expertise and the government would incur significant 
cost in choosing another contractor for follow-on systems. Does GAO 
have a view on the merits or demerits of competition in space programs? 

Competition can provide natural incentives for an organization to be 
more efficient and more innovative. These incentives work in DOD's 
favor. However, it is also important to recognize that competition can 
take various forms. For example, DOD can increase competition by using 
shadow contractors, pursuing alternative sensor designs, and breaking 
acquisitions into smaller blocks. DOD can also optimize its investment 
in weapon systems by competing air, land, sea, and space-based 
capabilities. By pursuing these various options, DOD would have greater 
assurance that it is obtaining the best value when it must select a 
prime contractor for follow-on systems.

5. How effective can competition be when we have so few major 
contractors capable of executing large and complex space programs?

While there are only a few contractors currently capable of 
implementing large and complex space programs, there are many more 
capable of building specific satellite components and technologies. 
Thus, by increasing competition at the mission payload or sensor level 
and breaking acquisitions into smaller pieces, DOD can expand the 
universe of contractors competing for work. Over the long run, this 
could enable more contractors to build the expertise and knowledge 
needed to manage large space programs. It would also require DOD to 
have significant insight into the lower tiers of the industry.

6. Is there a path to making competition a useful element in healthy 
programs?

Managing the industrial base is one of the most critical determinants 
of acquisition success. According to DOD studies, this not only means 
injecting competition early on to ensure that the highest performing 
and most cost-effective technologies and designs are being pursued, but 
adequately defining work; establishing shorter, more manageable 
contract periods; and providing the right incentives for contractors. 
Following an evolutionary development path would better enable programs 
to take these kinds of actions. It would also foster a healthier 
industrial base because it would get programs into production sooner. 
Also important is ensuring that programs have the right capability to 
evaluate contractor proposals and to manage the contracts once they are 
in place. As DOD's studies of space programs show, the government will 
invariably encounter problems when too much responsibility is handed 
over to contractors and too little oversight is provided.

We have also found that the path to healthier programs is characterized 
by having an open systems design. Such a design is characterized by 
(1) well defined, widely used, preferably nonproprietary interfaces and 
protocols between systems, subsystems, and components and (2) an 
explicit provision for system expansion or upgrade through 
incorporation of additional higher performance subsystems and 
components with minimal negative impact on the existing system. Open 
systems design allows competing developers to offer additional features 
and capabilities. With this approach, the government might be able to 
minimize dependence on a specific contractor. Also, upgrades can be 
added without replacing the entire system. Costs across the board--
development, production, operations, and support--can thereby be 
reduced.

7. Does GAO believe that space programs will be less schedule driven 
under the new acquisition policy?

No. In the past, DOD has taken a schedule-driven versus a knowledge-
driven approach to the acquisition process for space and other weapons 
systems with the justification that capabilities were urgently needed. 
In other words, commitments were made to achieving certain capabilities 
without knowing whether technologies being pursued could really work as 
intended. As a result, time and costs estimates were consistently 
exceeded, and steps essential to containing costs, maximizing 
competition among contractors, and testing technologies were 
shortchanged. Perversely, programs actually took longer when rushed at 
the start. Moreover, DOD often lacked assurance that it was even 
pursuing the best technical solution because alternatives were not 
analyzed or they were eliminated in order to meet schedule pressures. 
When technology did not perform as planned, assigning additional 
resources in terms of time and money became the primary option for 
solving problems, since customer expectations about the products' 
performance already became hardened.

The new space acquisition process does not change this approach or the 
incentives that drive it. Rather, it encourages programs to enter into 
product development without knowledge that technologies can work as 
intended. Moreover, for new programs like the Transformational 
Satellite (TSAT) and Space Based Radar (SBR), DOD is still setting 
initial satellite launch dates before this knowledge has been obtained. 
By contrast, DOD's acquisition policy for non-space systems establishes 
mature technologies--that is, technologies demonstrated in a relevant 
environment--as critical before entering product development. By 
encouraging programs to do so, the policy for non-space systems puts 
programs in a better position to deliver capability to the war fighter 
in a timely fashion and within funding estimates because program 
managers can focus on the design, system integration, and manufacturing 
tasks needed to produce a product.

8. The requirements for the Space-Based Infrared System High system 
(SBIRS High) still continue to change. In GAO's report, you highlight 
several examples, including batteries and solar cell panels. From the 
report, I gather that GAO finds that the Air Force's efforts to limit 
requirements changes to only those that are "urgent and compelling" are 
better, but that they are not successfully eliminating the growth of 
requirements. You mention at least $203 million in new requirements. 
Is this a correct interpretation of the new "urgent and compelling 
approach?":

Prior to the restructuring, the SBIRS High program office exerted no 
control over requirements changes, leaving many decisions on 
requirements to its contractors or within lower management levels of 
the program office. As part of the SBIRS High program restructuring, 
the Air Force established an advisory program management board to 
oversee requirements changes. The board's role is to ensure that new 
requirements are urgent and compelling, that they reflect an 
appropriate use of funds, and that decisions about requirements are 
more transparent. Air Force leadership, not the SBIRS High program 
office, made the decision that the new requirements were urgent and 
compelling enough to address.

We believe that establishing the board is a positive step and should 
help manage requirements changes more effectively. Nevertheless, the 
board will still be challenged to ensure some discipline in 
requirements setting, since there is a diverse group of Air Force and 
other DOD users that have an interest in SBIRS High and there are 
increasing demands for surveillance capabilities. Currently, there are 
several proposed requirements changes on the table that could have a 
significant impact on the program.

9. The GAO report also indicates that software development problems 
continue to be a problem. This problem is not limited to SBIRS High, 
however. What recommendations can you make to address this continuing 
problem?

Problems with software development in DOD weapons systems are well 
known. For example, the Defense Science Board reviewed selected DOD 
software intensive systems and found that programs lacked well thought-
out, disciplined program management and/or software development 
processes. The programs lacked meaningful cost, schedule, and 
requirements baselines, making it difficult to track progress against 
them. These findings are echoed by the work of DOD's Tri-Service 
initiative. Because weapon systems are becoming increasing dependent on 
software, lax management and oversight over software development can be 
detrimental to a program, as it was for SBIRS High.

There are steps we have identified in an ongoing review for the Senate 
Committee on Armed Services that DOD could take to address this 
problem. Chief among them is to require programs to apply best 
practices for software development and acquisition, many of which have 
been identified by the Software Engineering Institute at Carnegie 
Mellon University and packaged into continuous improvement models and 
guidance. In adopting these models, organizations would take a more 
disciplined and rigorous approach toward managing or overseeing 
software development. At the same time, organizations need to provide 
the right environment to reduce software development risk. This means 
establishing an environment comprised of an evolutionary software 
development approach that relies on well-understood, manageable 
requirements and a desire to continuously improve development 
processes. It also means adopting and using a host of metrics to track 
cost and scheduling deviations; requirements changes and their impact 
on software development efforts; testing efforts; as well as efforts to 
detect and fix defects. Also important is to integrate these practices 
into existing acquisition policies and improvement plans as well as to 
enforce the use of these practices within individual programs.

10. The GAO conclusion is that SBIRS High is still a program in 
trouble. To remedy this problem, GAO recommends that the Secretary 
reconvene the independent review team, or a similar body, to provide 
an assessment of the restructured program and concrete guidance for 
addressing the program's underlying problems. To play devil's advocate 
for a moment, how will another review of this program improve its 
chances of technical, budget, and schedule success?

The fundamental problem with the SBIRS High program has been the 
failure to develop key knowledge at critical junctures early in the 
development of the system, that is, before major investments were made. 
The program is now paying the price for this lack of knowledge 
development. Although the restructuring of the program in 2002 improved 
management and oversight capabilities, it did not go far enough in 
addressing the underlying problems with system design, integration, and 
software development. Another independent and in-depth technical review 
of the program is important to ensure that these problems are more 
clearly understood and that there are no other hidden problems lurking. 
At the same time, such a review will keep attention focused and 
heighten oversight of the program. Moreover, until it becomes standard 
to make knowledge-based decisions on DOD programs, ad hoc reviews such 
as the one we call for may be the only way to bring transparency to the 
decision making process.

11. SBIRS High is clearly a highly visible troubled program. How 
representative is it of space programs in general? Is it unique or are 
the problems identified present throughout the space acquisition 
effort?

We recently reported[Footnote 2] that the majority of satellite 
programs over the past couple decades, like SBIRS High, cost more than 
expected and took longer to develop than planned. SBIRS High is one of 
the few weapon systems programs to exceed the 25 percent cost threshold 
established in 10 U.S.C. 2433, but the problems affecting other 
programs have been equally dramatic. For example, cost estimates for 
the Advanced Extremely High Frequency (AEHF) communications satellite 
program grew by $1.2 billion from 1999 through 2001, while the program 
experienced a 2-year delay in the launch of the first satellite. And 
while DOD has spent several billion dollars over the past 2 decades to 
develop low-orbiting satellites that can track ballistic missiles 
throughout their flight, it has not launched a single satellite to 
perform this capability.

A key underlying problem with many programs has been the desire to 
achieve revolutionary advancements in capability instead of 
evolutionary advancements. Such an approach meant that requirements 
exceeded resources (time, money, and technology) at the time of product 
development, setting the stage for costly and time-consuming rework 
later in the program. More specifically, in reviewing our past reports, 
we found that: (1) requirements for what the satellite needed to do and 
how well it must perform were not adequately defined at the beginning 
of a program or were changed significantly once the program had already 
begun; (2) investment practices were weak, e.g., cost estimates were 
optimistic or potentially more cost-effective approaches were not 
examined; (3) acquisition strategies were poorly executed, e.g., 
competition was reduced for the sake of schedule or DOD did not 
adequately oversee contractors; and (4) technologies were not mature 
enough to be included in product development. All of these problems 
affected SBIRS High and AEHF. One or more affected the Space Tracking 
and Surveillance System (STSS) and the predecessor SBIRS programs as 
well as Milstar, the Global Positioning System (GPS), and the National 
Polar-orbiting Operational Environmental Satellite System (NPOESS).

Because DOD took a schedule-driven approach instead of a knowledge-
driven approach to the acquisition process, activities essential to 
containing costs, maximizing competition among contractors, and testing 
technologies were compressed or not done. Like SBIRS High, many 
programs also encountered problems in setting requirements due to the 
diverse array of organizations with competing interests involved in 
overall satellite development--from the individual military services, 
to testing organizations, contractors, civilian agencies, and in some 
cases international partners. Requirements setting for SBIRS High was 
particularly problematic because the government put too much 
responsibility on its contractors to balance these competing interests-
-a problem recognized in DOD's own study of SBIRS High and other 
studies of space acquisition problems.

In our view, new programs like the Transformational Satellite (TSAT) 
will likewise be unable to make a match between needs and resources at 
the onset of product development because DOD's new space acquisition 
policy encourages product development to begin without knowing that 
technologies can work as intended to meet capability needs.

In preparing answers to your questions, we relied on our prior work on 
DOD's new space acquisition policy,[Footnote 3] best practices in 
weapon system acquisitions, and our reviews of specific space 
acquisitions. Because we relied on previously issued work, we did not 
obtain comments from DOD on a draft of this letter. We conducted our 
work from December 2003 through January 2004 in accordance with 
generally accepted government auditing standards.

We are sending copies of this letter to the Secretaries of Defense and 
the Air Force and interested congressional committees. We will also 
make copies available to others upon request. In addition, the report 
will be available at no charge on the GAO Web site at http://
www.gao.gov.

If you or your staff have any questions concerning this report, please 
contact either me or Katherine Schinasi at (202) 512-4841. Key 
contributors to this letter were Cristina Chaplain, Sigrid McGinty, Art 
Gallegos, Maricela Cherveny, John Oppenheim, and Mike Hazard.

Robert E. Levin:

Director, Acquisition and Sourcing Management:

Signed by Robert E. Levin: 

(120318):

FOOTNOTES

[1] U.S. General Accounting Office, Best Practices: A More Constructive 
Test Approach Is Key to Better Weapon System Outcomes, GAO/NSIAD-00-199 
(Washington, D.C., July 31, 2000).

[2] U.S. General Accounting Office, Military Space Operations: Common 
Problems and Their Effects on Satellite and Related Acquisitions, GAO-
03-825R (Washington, D.C.: June 2, 2003).

[3] U.S. General Accounting Office, Defense Acquisitions: Improvements 
Needed in Space Systems Acquisition Management Policy, GAO-03-1073 
(Washington, D.C.: September 15, 2003) and Defense Acquisitions: 
Improvements Needed in Space Systems Acquisition Policy to Optimize 
Growing Investment in Space, GAO-04-253T (Washington, D.C.: November 
18, 2003).