This is the accessible text file for GAO report number GAO-04-379R entitled 'Defense Acquisitions: Risks Posed by DOD's New Space Systems Acquisition Policy' which was released on January 29, 2004. This text file was formatted by the U.S. General Accounting Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. January 29, 2004: The Honorable Wayne Allard: Chairman: The Honorable Bill Nelson: Ranking Minority Member: Subcommittee on Strategic Forces: Committee on Armed Services: United States Senate: Subject: Defense Acquisitions: Risks Posed by DOD's New Space Systems Acquisition Policy: On November 18, 2003, we testified before the Subcommittee on the Department of Defense's (DOD) new acquisition policy for space systems. The new acquisition policy, issued in October 2003, sets the stage for decision making for DOD's investment in space systems, which currently stands at more than $18 billion annually and is expected to grow considerably over the next decade. You requested that we provide additional comments on several issues relating to the new policy and other space acquisition issues. Your specific questions and our answers are discussed below. 1. The Air Force maintains that its Defense Space Acquisition Board (DSAB) process allows earlier identification of problems and senior level attention, which will improve management and lower risk. Does GAO have any concerns with the DSAB process? Our concern is not with earlier identification of problems or the added senior level attention the new process calls for, but with earlier investment decisions, which are also called for. Under the new process, the DSAB may approve product development to begin before DOD knows whether technologies can work as intended. As a result, it will make major investment commitments without really knowing what resources will be required to deliver promised capability. Moreover, the policy encourages development of leading-edge technology within product development, that is, at the same time the program manager is designing the system and undertaking other product development activities. DOD believes this approach will allow space systems to better incorporate leading-edge technologies. But as our work has repeatedly shown, such concurrency within space and other weapon system programs increases the risk that significant problems will be discovered as the system is integrated and built, when it is more costly and time-consuming to fix them. Moreover, as we testified, the knowledge-building approach for space stands in sharp contrast to that followed by successful programs and the approach recommended by DOD's revised acquisition policy for weapon systems. Successful programs will not commit to undertaking product development unless they have high confidence that they have achieved a match between what the customer wants and what the program can deliver. Technologies that are not mature continue to be developed in an environment that is focused solely on technology development. This system puts programs in a better position to succeed because they can focus on design, system integration, and manufacturing. Further, our work has shown that taking an evolutionary approach to improving capability increases the likelihood of delivering that capability to the war fighter sooner than the revolutionary approach the Air Force continues to support in the new space policy. 2. Does GAO believe that the process put into place in the new space acquisition policy by which cost estimates are derived will provide better cost estimates? No. Although some process changes will be made, the underlying causes of underestimating costs remain. DOD is adopting new methodologies and tools to enhance cost estimates, and it is enlisting assistance from DOD's Cost Analysis Improvement Group (CAIG) to conduct independent cost estimates using cost estimating teams drawn from a broad spectrum of the cost-estimating community. Moreover, programs are now required to resolve differences between their cost estimates and estimates produced by the independent teams. In the past, cost-estimating groups have developed estimates that were different, leaving decision makers to select one estimate or combine a few. However, under the new space acquisition policy, cost estimates do not have to be based on the knowledge that technologies can work as intended. History has shown that cost estimates not based on such knowledge are significantly understated. Moreover, incentives that work against providing good estimates have not changed. Unlike the commercial world where the focus is on delivering a product to market, DOD's system focuses on competing for resources from oversubscribed budgets. In the competition for funding, managers are encouraged to launch product developments before technologies are mature. Because funding is competitive and DOD's forecasts of costs, schedules, and performance are largely based on immature technologies and other unknowns, estimates tend to be squeezed into insufficient profiles of available funding. In fact, pressures to underestimate costs may increase over the next decade as DOD plans to undertake a number of new, challenging space programs--which are expected to require an additional $4 billion in the next 4 years alone. Costs beyond that period are as yet unknown but are likely to be considerably higher. 3. What is GAO's view on the Air Force policy related to full funding? DOD's acquisition policy for other weapon systems requires a commitment for full funding at milestone B--the start of product development and the point at which DOD should have knowledge that technologies can work as intended. However, the new space acquisition policy does not require DOD to commit to fully fund a space program either when this knowledge has been obtained or at any point in the development process. Hence, there is no guarantee that the resources needed to meet requirements on any individual program will be there when needed--particularly as DOD moves forward with its new programs. This represents another important departure from the development approach followed by successful programs. Our prior work[Footnote 1] has found that if a product's business case measures up, that is a company is assured that there is a market or need for the product and that it has the right knowledge in hand to develop the product with firm cost and schedule estimates, the company then commits to the entire development of the product, including the financial investment. In other words, corporate resources are made available to the development team so that product success is not compromised. As noted earlier, because DOD begins too many programs, its resources are always oversubscribed. By requiring program managers to continually justify funding, DOD runs a risk of foreclosing the ability for sound planning and execution. 4. The Young Panel was not convinced of the merits of competition in some circumstances, particularly when the incumbent has performed well and "owns" the expertise and the government would incur significant cost in choosing another contractor for follow-on systems. Does GAO have a view on the merits or demerits of competition in space programs? Competition can provide natural incentives for an organization to be more efficient and more innovative. These incentives work in DOD's favor. However, it is also important to recognize that competition can take various forms. For example, DOD can increase competition by using shadow contractors, pursuing alternative sensor designs, and breaking acquisitions into smaller blocks. DOD can also optimize its investment in weapon systems by competing air, land, sea, and space-based capabilities. By pursuing these various options, DOD would have greater assurance that it is obtaining the best value when it must select a prime contractor for follow-on systems. 5. How effective can competition be when we have so few major contractors capable of executing large and complex space programs? While there are only a few contractors currently capable of implementing large and complex space programs, there are many more capable of building specific satellite components and technologies. Thus, by increasing competition at the mission payload or sensor level and breaking acquisitions into smaller pieces, DOD can expand the universe of contractors competing for work. Over the long run, this could enable more contractors to build the expertise and knowledge needed to manage large space programs. It would also require DOD to have significant insight into the lower tiers of the industry. 6. Is there a path to making competition a useful element in healthy programs? Managing the industrial base is one of the most critical determinants of acquisition success. According to DOD studies, this not only means injecting competition early on to ensure that the highest performing and most cost-effective technologies and designs are being pursued, but adequately defining work; establishing shorter, more manageable contract periods; and providing the right incentives for contractors. Following an evolutionary development path would better enable programs to take these kinds of actions. It would also foster a healthier industrial base because it would get programs into production sooner. Also important is ensuring that programs have the right capability to evaluate contractor proposals and to manage the contracts once they are in place. As DOD's studies of space programs show, the government will invariably encounter problems when too much responsibility is handed over to contractors and too little oversight is provided. We have also found that the path to healthier programs is characterized by having an open systems design. Such a design is characterized by (1) well defined, widely used, preferably nonproprietary interfaces and protocols between systems, subsystems, and components and (2) an explicit provision for system expansion or upgrade through incorporation of additional higher performance subsystems and components with minimal negative impact on the existing system. Open systems design allows competing developers to offer additional features and capabilities. With this approach, the government might be able to minimize dependence on a specific contractor. Also, upgrades can be added without replacing the entire system. Costs across the board-- development, production, operations, and support--can thereby be reduced. 7. Does GAO believe that space programs will be less schedule driven under the new acquisition policy? No. In the past, DOD has taken a schedule-driven versus a knowledge- driven approach to the acquisition process for space and other weapons systems with the justification that capabilities were urgently needed. In other words, commitments were made to achieving certain capabilities without knowing whether technologies being pursued could really work as intended. As a result, time and costs estimates were consistently exceeded, and steps essential to containing costs, maximizing competition among contractors, and testing technologies were shortchanged. Perversely, programs actually took longer when rushed at the start. Moreover, DOD often lacked assurance that it was even pursuing the best technical solution because alternatives were not analyzed or they were eliminated in order to meet schedule pressures. When technology did not perform as planned, assigning additional resources in terms of time and money became the primary option for solving problems, since customer expectations about the products' performance already became hardened. The new space acquisition process does not change this approach or the incentives that drive it. Rather, it encourages programs to enter into product development without knowledge that technologies can work as intended. Moreover, for new programs like the Transformational Satellite (TSAT) and Space Based Radar (SBR), DOD is still setting initial satellite launch dates before this knowledge has been obtained. By contrast, DOD's acquisition policy for non-space systems establishes mature technologies--that is, technologies demonstrated in a relevant environment--as critical before entering product development. By encouraging programs to do so, the policy for non-space systems puts programs in a better position to deliver capability to the war fighter in a timely fashion and within funding estimates because program managers can focus on the design, system integration, and manufacturing tasks needed to produce a product. 8. The requirements for the Space-Based Infrared System High system (SBIRS High) still continue to change. In GAO's report, you highlight several examples, including batteries and solar cell panels. From the report, I gather that GAO finds that the Air Force's efforts to limit requirements changes to only those that are "urgent and compelling" are better, but that they are not successfully eliminating the growth of requirements. You mention at least $203 million in new requirements. Is this a correct interpretation of the new "urgent and compelling approach?": Prior to the restructuring, the SBIRS High program office exerted no control over requirements changes, leaving many decisions on requirements to its contractors or within lower management levels of the program office. As part of the SBIRS High program restructuring, the Air Force established an advisory program management board to oversee requirements changes. The board's role is to ensure that new requirements are urgent and compelling, that they reflect an appropriate use of funds, and that decisions about requirements are more transparent. Air Force leadership, not the SBIRS High program office, made the decision that the new requirements were urgent and compelling enough to address. We believe that establishing the board is a positive step and should help manage requirements changes more effectively. Nevertheless, the board will still be challenged to ensure some discipline in requirements setting, since there is a diverse group of Air Force and other DOD users that have an interest in SBIRS High and there are increasing demands for surveillance capabilities. Currently, there are several proposed requirements changes on the table that could have a significant impact on the program. 9. The GAO report also indicates that software development problems continue to be a problem. This problem is not limited to SBIRS High, however. What recommendations can you make to address this continuing problem? Problems with software development in DOD weapons systems are well known. For example, the Defense Science Board reviewed selected DOD software intensive systems and found that programs lacked well thought- out, disciplined program management and/or software development processes. The programs lacked meaningful cost, schedule, and requirements baselines, making it difficult to track progress against them. These findings are echoed by the work of DOD's Tri-Service initiative. Because weapon systems are becoming increasing dependent on software, lax management and oversight over software development can be detrimental to a program, as it was for SBIRS High. There are steps we have identified in an ongoing review for the Senate Committee on Armed Services that DOD could take to address this problem. Chief among them is to require programs to apply best practices for software development and acquisition, many of which have been identified by the Software Engineering Institute at Carnegie Mellon University and packaged into continuous improvement models and guidance. In adopting these models, organizations would take a more disciplined and rigorous approach toward managing or overseeing software development. At the same time, organizations need to provide the right environment to reduce software development risk. This means establishing an environment comprised of an evolutionary software development approach that relies on well-understood, manageable requirements and a desire to continuously improve development processes. It also means adopting and using a host of metrics to track cost and scheduling deviations; requirements changes and their impact on software development efforts; testing efforts; as well as efforts to detect and fix defects. Also important is to integrate these practices into existing acquisition policies and improvement plans as well as to enforce the use of these practices within individual programs. 10. The GAO conclusion is that SBIRS High is still a program in trouble. To remedy this problem, GAO recommends that the Secretary reconvene the independent review team, or a similar body, to provide an assessment of the restructured program and concrete guidance for addressing the program's underlying problems. To play devil's advocate for a moment, how will another review of this program improve its chances of technical, budget, and schedule success? The fundamental problem with the SBIRS High program has been the failure to develop key knowledge at critical junctures early in the development of the system, that is, before major investments were made. The program is now paying the price for this lack of knowledge development. Although the restructuring of the program in 2002 improved management and oversight capabilities, it did not go far enough in addressing the underlying problems with system design, integration, and software development. Another independent and in-depth technical review of the program is important to ensure that these problems are more clearly understood and that there are no other hidden problems lurking. At the same time, such a review will keep attention focused and heighten oversight of the program. Moreover, until it becomes standard to make knowledge-based decisions on DOD programs, ad hoc reviews such as the one we call for may be the only way to bring transparency to the decision making process. 11. SBIRS High is clearly a highly visible troubled program. How representative is it of space programs in general? Is it unique or are the problems identified present throughout the space acquisition effort? We recently reported[Footnote 2] that the majority of satellite programs over the past couple decades, like SBIRS High, cost more than expected and took longer to develop than planned. SBIRS High is one of the few weapon systems programs to exceed the 25 percent cost threshold established in 10 U.S.C. 2433, but the problems affecting other programs have been equally dramatic. For example, cost estimates for the Advanced Extremely High Frequency (AEHF) communications satellite program grew by $1.2 billion from 1999 through 2001, while the program experienced a 2-year delay in the launch of the first satellite. And while DOD has spent several billion dollars over the past 2 decades to develop low-orbiting satellites that can track ballistic missiles throughout their flight, it has not launched a single satellite to perform this capability. A key underlying problem with many programs has been the desire to achieve revolutionary advancements in capability instead of evolutionary advancements. Such an approach meant that requirements exceeded resources (time, money, and technology) at the time of product development, setting the stage for costly and time-consuming rework later in the program. More specifically, in reviewing our past reports, we found that: (1) requirements for what the satellite needed to do and how well it must perform were not adequately defined at the beginning of a program or were changed significantly once the program had already begun; (2) investment practices were weak, e.g., cost estimates were optimistic or potentially more cost-effective approaches were not examined; (3) acquisition strategies were poorly executed, e.g., competition was reduced for the sake of schedule or DOD did not adequately oversee contractors; and (4) technologies were not mature enough to be included in product development. All of these problems affected SBIRS High and AEHF. One or more affected the Space Tracking and Surveillance System (STSS) and the predecessor SBIRS programs as well as Milstar, the Global Positioning System (GPS), and the National Polar-orbiting Operational Environmental Satellite System (NPOESS). Because DOD took a schedule-driven approach instead of a knowledge- driven approach to the acquisition process, activities essential to containing costs, maximizing competition among contractors, and testing technologies were compressed or not done. Like SBIRS High, many programs also encountered problems in setting requirements due to the diverse array of organizations with competing interests involved in overall satellite development--from the individual military services, to testing organizations, contractors, civilian agencies, and in some cases international partners. Requirements setting for SBIRS High was particularly problematic because the government put too much responsibility on its contractors to balance these competing interests- -a problem recognized in DOD's own study of SBIRS High and other studies of space acquisition problems. In our view, new programs like the Transformational Satellite (TSAT) will likewise be unable to make a match between needs and resources at the onset of product development because DOD's new space acquisition policy encourages product development to begin without knowing that technologies can work as intended to meet capability needs. In preparing answers to your questions, we relied on our prior work on DOD's new space acquisition policy,[Footnote 3] best practices in weapon system acquisitions, and our reviews of specific space acquisitions. Because we relied on previously issued work, we did not obtain comments from DOD on a draft of this letter. We conducted our work from December 2003 through January 2004 in accordance with generally accepted government auditing standards. We are sending copies of this letter to the Secretaries of Defense and the Air Force and interested congressional committees. We will also make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at http:// www.gao.gov. If you or your staff have any questions concerning this report, please contact either me or Katherine Schinasi at (202) 512-4841. Key contributors to this letter were Cristina Chaplain, Sigrid McGinty, Art Gallegos, Maricela Cherveny, John Oppenheim, and Mike Hazard. Robert E. Levin: Director, Acquisition and Sourcing Management: Signed by Robert E. Levin: (120318): FOOTNOTES  U.S. General Accounting Office, Best Practices: A More Constructive Test Approach Is Key to Better Weapon System Outcomes, GAO/NSIAD-00-199 (Washington, D.C., July 31, 2000).  U.S. General Accounting Office, Military Space Operations: Common Problems and Their Effects on Satellite and Related Acquisitions, GAO- 03-825R (Washington, D.C.: June 2, 2003).  U.S. General Accounting Office, Defense Acquisitions: Improvements Needed in Space Systems Acquisition Management Policy, GAO-03-1073 (Washington, D.C.: September 15, 2003) and Defense Acquisitions: Improvements Needed in Space Systems Acquisition Policy to Optimize Growing Investment in Space, GAO-04-253T (Washington, D.C.: November 18, 2003).