This is the accessible text file for GAO report number GAO-04-759 entitled 'Defense Acquisitions: Space-Based Radar Effort Needs Additional Knowledge before Starting Development' which was released on July 19, 2004. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Report to Congressional Committees: United States Government Accountability Office: GAO: July 2004: Defense Acquisitions: Space-Based Radar Effort Needs Additional Knowledge before Starting Development: GAO-04-759: GAO Highlights: Highlights of GAO-04-759, a report to congressional committees: Why GAO Did This Study: Missing among the Department of Defense’s (DOD) portfolio of systems is a capability to track stationary and moving enemy vehicles on land or at sea in any type of weather, day or night, from space. To meet this need, DOD and the intelligence community are collaborating on the ambitious Space-Based Radar (SBR) program. By leveraging the newest generation of radar technologies, the SBR concept promises to deliver high-quality data to a wide array of users. DOD intends to start product development in 2006 and to field SBR satellites as quickly as possible so that warfighters, the intelligence community, and national decision makers can gain a better understanding of what adversaries are doing in specific locations around the world. GAO reviewed the SBR program to assess DOD’s progress in attaining the knowledge it needs by 2006 in terms of customer needs (or requirements) and resources. What GAO Found: Although SBR is 2 years away from product development, the program already faces major challenges. DOD officials say SBR will likely be the most expensive and technically challenging space system ever built by DOD. The acquisition time frame is much shorter than what has been achieved in the past for other complex satellite systems. Finally, DOD is setting precedence by taking the lead on developing SBR with the intelligence community as a partner. Most DOD space programs that GAO has reviewed in the past several decades were hampered by schedule and cost growth and performance shortfalls. Problems were largely rooted in a failure to match requirements with resources when starting product development. Commitments were made without knowing whether technologies being pursued would work as intended. To avoid these problems, leading commercial firms have adopted a knowledge-based model that enables decision makers to be reasonably certain about their products at critical junctures and helps them make informed investment decisions. Although DOD has taken positive steps to strengthen the involvement of senior leaders within DOD and the intelligence community in setting requirements, SBR’s concept of operations has not been approved and signed by requirements boards for either of the two partners. Without documentation and formal approval, it is unclear who will be held accountable for setting requirements or how disagreements among SBR’s partners will be resolved when DOD moves SBR into ensuing phases of acquisition. DOD has adopted noteworthy practices to gain knowledge about SBR’s resources. These include maximizing the use of systems engineering to close gaps between requirements and resources; estimating all of SBR’s costs; exploring alternatives for SBR if the Transformational Communications Architecture (TCA)—the communications infrastructure that is expected to relay SBR data across a network of users—incurs schedule and performance shortfalls; and asking contractors to propose multiple operations concepts for SBR with or without TCA. Despite these accomplishments, DOD is at risk of knowledge gaps. SBR’s critical technologies will not be mature when product development starts, as called for by best practices. One of TCA’s primary components may not be ready in time to support SBR data. These knowledge gaps make it harder for DOD to reliably estimate how much time and money are needed to complete SBR’s development. If TCA is delayed, DOD’s alternatives may involve reducing SBR’s capabilities or significantly increasing program cost. Without sufficient knowledge, DOD may not be able to determine by the time SBR’s product development starts in 2006 whether space-based radar is best suited to tracking moving targets on land or at sea or whether air-based radar would provide enough capabilities at far less cost. More specific analyses would help DOD weigh the merits of various alternatives and assess how much to invest in the SBR acquisition program versus air platforms with similar capabilities. What GAO Recommends: GAO recommends that, before committing to SBR’s acquisition program in 2006, DOD and its intelligence partners close gaps in the requirements approval process in terms of documenting decisions and be prepared to add time and money or make trade-offs with other DOD space programs to address SBR’s requirements and resources. DOD generally agreed with our findings and partially agreed with our recommendations. www.gao.gov/cgi-bin/getrpt?GAO-04-759 To view the full product, including the scope and methodology, click on the link above. For more information, contact Katherine V. Schinasi at (202) 512-4841 or schinasik@gao.gov. [End of section] Contents: Letter: Scope and Methodology: Results in Brief: Background: Gaining Knowledge about Requirements and Resources before Product Development Is Important for Space Acquisition Success: DOD Moving Forward on Acquiring Critical Knowledge but Gaps Remain in Approval for SBR Requirements: DOD Taking Proactive Steps to Gain Knowledge about Resources, but Critical Gaps May Remain at Product Development: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: TRL Scale for Assessing Critical Technologies: Appendix II: Comments from the Department of Defense: Tables: Table 1: Knowledge Provided by SBR's Initial Capabilities Document and Concept of Operations: Table 2: Technology Readiness Levels of SBR Critical Technologies: Figures: Figure 1: SBR's Acquisition Schedule (in fiscal years): Figure 2: SBR's Oversight Structure: Figure 3: Requirements-Setting Can Impact Acquisition Process: Abbreviations: AOA: analysis of alternatives: DOD: Department of Defense: R&D: research and development: SBR: Space-Based Radar: TCA: Transformational Communications Architecture: TRL: Technology Readiness Level: United States Government Accountability Office: Washington, DC 20548: July 23, 2004: The Honorable John Warner: Chairman: The Honorable Carl Levin: Ranking Minority Member: Committee on Armed Services: United States Senate: The Honorable Ted Stevens: Chairman: The Honorable Daniel Inouye: Ranking Minority Member: Subcommittee on Defense: Committee on Appropriations: United States Senate: The Honorable Duncan Hunter: Chairman: The Honorable Ike Skelton: Ranking Minority Member: Committee on Armed Services: House of Representatives: The Honorable Jerry Lewis: Chairman: The Honorable John P. Murtha: Ranking Minority Member: Subcommittee on Defense: Committee on Appropriations: House of Representatives: The Department of Defense (DOD) currently has ground-based and air-based radars but no capability to track moving targets from space. To meet a need for persistent global observation, DOD and the intelligence community are working together to develop the Space-Based Radar (SBR) system to find, identify, track, and monitor ground and sea targets--mobile or immobile--under all-weather conditions and on a near-continual basis across large swaths of the earth's surface. SBR is to enhance information gathering by providing intelligence, surveillance, and reconnaissance data in a meaningful and timely manner. This is an ambitious program for DOD. DOD's initial total cost estimate for SBR is about $28.6 billion from fiscal year 2003 to 2024. And although the technologies for tracking moving targets from space are still in development, DOD is scheduling almost 7 years between the start of product development and launch of the first SBR satellite--a time frame that is considerably shorter than what has been achieved in the past for other complex satellite systems. DOD would like to field SBR satellites as quickly as possible because it believes that SBR represents a major leap forward in providing the warfighters, intelligence community, and national decision makers with significant tactical, operational, and strategic advantages over potential adversaries. For example, DOD envisions that SBR will be able to see deep inside enemy territory without risk to personnel or resources and that it will operate over areas where conventional airborne surveillance systems are at risk from the enemy's surface-to-air missiles, which can travel increasingly longer ranges. DOD also envisions that SBR will generate high-quality radar imagery of targets and terrain and interface with ground, air, and other space systems so that users can gain a better understanding of what is occurring in specific locations. DOD is in the early exploratory phase of the SBR program--a period when it gathers knowledge about its needs, or requirements, and explores the feasibility of meeting those requirements. DOD expects to begin product development in fiscal year 2006; at which point it will make a commitment to invest in a formal acquisition program for SBR. Our past work has shown that successful weapon system programs are able to match their needs to their resources--that is, money, technology, and time-- before product development. With achievable requirements and commitment of sufficient investment to complete development, programs are better able to deliver products at cost and on schedule. Most space programs over the past several decades have not been able to achieve a match between needs and resources before product development and have incurred significant cost and schedule increases due in part to the need to rework technologies in the later stages of their acquisition. We conducted our review of the SBR program on the initiative of the Comptroller General. Given SBR's overall importance to DOD and the intelligence community, we reviewed the SBR program to assess DOD's progress in attaining the knowledge it needs by 2006 in terms of (1) requirements and (2) resources--technology, communications infrastructure, and funding. We are addressing this report to you because of your jurisdiction over weapon systems acquisition. Scope and Methodology: To assess DOD's progress in attaining the knowledge it needs before the start of product development, we examined the resources (technology, communications infrastructure, and funding) committed and planned for the program as well as the users' needs for an SBR system. We considered DOD's plans for maturing the critical technologies when we obtained technology-readiness information for each critical technology (as well as its mature backup technology) against best practice standards to determine if they will be sufficiently mature when DOD plans to start product development. We also reviewed the SBR risk management plans and concept development contract information. We discussed these documents and issues with representatives from Air Force Space Command, Peterson Air Force Base, Colorado; and the SBR Joint Program Office, Space and Missile Systems Center, Los Angeles Air Force Base, California. To determine SBR's role in a larger DOD architecture, we met with officials from the Joint Chiefs of Staff, Washington, D.C; and the Air Force Directorate of Space Acquisitions, Arlington, Virginia. We also consulted past GAO reports to determine the relationship between SBR and the Transformational Communications Architecture. To determine the scope and completeness of the analysis of alternatives and its follow-on study to identify the optimal ways to gather information on ground moving targets from radars based in space versus air, we met with officials from Air Force Space Command; DOD Office of the Director, Program Analysis and Evaluation, Washington, D.C; Air Force Directorate of Requirements for Space, Crystal City, Virginia; and the Air Force Studies and Analyses Agency, Arlington, Virginia. We also talked with an official from the Air Force Office of Aerospace Studies, Kirtland Air Force Base, New Mexico. We discussed overarching programmatic issues--including the level of coordination between DOD and the intelligence community--with representatives from the Air Force Directorate of Space Acquisitions. We were not able to obtain meetings with members of the Mission Requirements Board (a board within the intelligence community responsible for approving program requirements) or the intelligence agencies to discuss their stake in the SBR program. We performed our work from November 2003 through June 2004 in accordance with generally accepted government auditing standards. Results in Brief: DOD has inserted a great deal of management and stakeholder involvement into developing SBR so that it can gain greater knowledge about requirements before product development than it has in past programs. Senior-level officials from DOD, the military services, and the intelligence community are heading up three new SBR oversight groups to discuss what can and cannot be accomplished in terms of the desired time frame, available funding, and achievable technologies. The groups have so far attained informal agreement on requirements. However, this commitment has not been formalized, and it is unclear as to whether or how it will be formalized. Moreover, it is also unclear how disagreements that may occur later on among SBR's partners will be resolved. Given the varied interests of SBR's partners and past problems with securing agreement on requirements for space programs,[Footnote 1] it is important that DOD build on the positive steps it has already taken and find ways to formalize commitment to requirements as well as SBR's concept of operations. DOD is also taking positive steps in its effort to gain knowledge about SBR's resources. These steps include strengthening systems engineering applications; estimating not just direct costs but all of SBR's life- cycle costs; exploring alternatives for SBR if the new Transformational Communications Architecture (TCA), the infrastructure that is to help relay SBR data, falls short of its schedule and performance goals; and asking concept development contractors to propose multiple design concepts for SBR with or without TCA. Despite these accomplishments, however, decision makers will have significant knowledge gaps about SBR's resources if SBR's product development phase starts when currently planned. This is because DOD intends to start product development in fiscal year 2006 even though the two critical technologies that would enable the tracking of surface-moving targets and the timely delivery of imagery data will not have been tested in space or even in a relevant environment. Without the knowledge derived from such tests, DOD cannot adequately assess whether the technologies will work as intended, making it harder to reliably estimate how much time and money is needed to complete development. Adding to SBR's overall risk are uncertainties as to whether TCA will be available to transmit SBR's vast volumes of radar data and imagery. Although DOD has plans underway to develop alternatives for SBR if TCA is delayed, the primary alternatives would involve either reducing SBR's capabilities or significantly increasing program cost. Lastly, DOD and the Air Force may not have sufficient knowledge to make a corporate decision as to how much it should invest in space-based radar capabilities versus air-based capabilities at the time it makes a commitment to the formal SBR acquisition in 2006. While the Air Force has undertaken two analyses of the SBR investment, neither presents a comprehensive assessment as to whether space-or air-based radar platforms (or a combination of both) are better suited for tracking moving targets on land or at sea, nor does either analysis weigh the capabilities and costs of each suitable radar option in space and in the air. Before committing to SBR's acquisition program in 2006, we recommend that senior-level officials in DOD and the intelligence community close gaps in the requirements-setting process in terms of documenting decisions and establishing a formal mechanism for addressing unresolved issues or how changes to approved requirements will be assessed. We recommend that DOD's space acquisition policy be modified to identify the requirements-setting process when DOD partners with the intelligence community on space programs. We recommend that DOD delay approval to commit funding to product development for SBR until technologies are matured to higher levels, but if DOD determines that the SBR program should proceed to product development with less than mature technologies, we recommend that DOD be prepared to add time and money or make trade-offs with other DOD space programs to address SBR's requirements and resources. DOD generally agreed with our findings and our recommendation to strengthen its study of SBR alternatives. DOD partially agreed with our recommendations to strengthen its requirement setting process for SBR and to demonstrate SBR technologies in a relevant or operational environment before committing to product development. DOD did not agree with our recommendation to modify its acquisition policy to strengthen requirements setting. In commenting on our recommendations, DOD cited concerns about supplanting current requirements setting processes. Our recommendation does not advocate replacing this process, but rather strengthening it to provide more transparency, discipline, and accountability. Background: SBR represents the first time that DOD has taken the lead on developing a major national security space capability with the intelligence community as a partner. Because of this partnership, SBR's acquisition process is more complex than that used for typical DOD programs. While DOD and the intelligence community will likely use all the data that SBR produces, their priorities differ. DOD's warfighting community is particularly interested in tracking targets moving over land or sea as well as other objects of interest. The intelligence community is more focused on obtaining detailed global imagery and combining it with other data for advanced processing. SBR is expected to meet both needs and be fully integrated with other space and non-space systems, including TCA, which is to transmit SBR's data to receivers in the air, at sea, or on the ground. A key advantage of radar in space is having the ability to "see" through clouds and sand storms and any type of weather, day or night. Radar-equipped aircrafts, on the other hand, require U.S. air dominance to collect radar information and must steer clear of hostile areas--the result being limited radar coverage. The SBR concept offers other added features, including electronic steering of the radar signal toward a particular area and capturing high volumes of very fine resolution radar images of targets and terrain. With the ability to perform these functions almost simultaneously, SBR is expected to help analysts gain a better understanding of what is occurring in specific locations. To help meet some of its goals, DOD plans to leverage key technologies that were developed in the late 1990s to demonstrate a space-based radar capability. According to DOD officials, contractors developed some satellite hardware and prototype components under the Discoverer II program, which began in 1998 and was to identify and validate by 2008 the capability of tracking mobile ground targets from space. Discoverer II, comprising two radar demonstration satellites, was a joint initiative by the Air Force, DOD's Defense Advanced Research Projects Agency, and the intelligence community's National Reconnaissance Office. DOD officials told us that the Discoverer II program had reached the preliminary design review phase when it was cancelled in 2000 because of cost and schedule uncertainties, poorly explained requirements, and the lack of a coherent vision to transition the system to operational use. The Secretary of Defense concluded that space-based radar could provide a military advantage and in 2001 approved SBR as a new major defense acquisition program, delegating it to the Air Force. In July 2003, an independent cost assessment team consisting of representatives from DOD and the intelligence community estimated that $28.6 billion would be needed to pay for SBR's life-cycle costs--development, production, launch, and operation. The program entered the study phase in August 2003. The Air Force has requested $328 million for SBR in fiscal year 2005 and has programmed about $4 billion for the program from fiscal years 2005 to 2009. Given concerns about affordability and readiness, the Fiscal Year 2005 Defense Appropriations Conference Report reduced funding for SBR to $75 million, with the direction to return this effort back to the technology development phase. In 2003, Congress reduced the Air Force's $274 million budget request for SBR by $100 million due to concerns about technology maturity and schedule. DOD has scheduled the start of product development for mid-fiscal year 2006, with production starting at the end of fiscal year 2008 and the first satellite to be launched at the end of fiscal year 2012. Figure 1 shows SBR's acquisition schedule in fiscal years. Figure 1: SBR's Acquisition Schedule (in fiscal years): [See PDF for image] [End of figure] Gaining Knowledge about Requirements and Resources before Product Development Is Important for Space Acquisition Success: In the past several decades, DOD's space acquisitions have experienced problems that have driven up costs by hundreds of millions, even billions, of dollars; have stretched schedules by years; and have increased performance risks. In some cases, capabilities have not been delivered to the warfighter after decades of development. Our reports have shown that these problems, common among many weapon acquisitions, are largely rooted in a failure to match the customer's requirements (desired capabilities) with the developer's resources (technical knowledge, timing, and funding) when starting an acquisition program. In particular, our past work has shown that for space systems, product development was often started based on a rigid set of requirements that proved to be unachievable within a reasonable development time frame. Other cases involved unstable requirements. In some cases where requirements had been identified and approved, even more requirements were added after the program began. When technology did not perform as planned, adding resources in terms of time and money became the primary option for solving problems because the customer's expectations about the product's performance capabilities already had been set. The path traditionally taken by space programs--and other DOD weapon system programs--stands in sharp contrast to that taken by leading commercial firms. Our extensive body of work shows that leading companies use a product development model that helps reduce risks and increase knowledge when developing new products. This best practices model enables decision makers to be reasonably certain about their products at critical junctures during development and helps them make informed investment decisions. This knowledge-based process can be broken down into three cumulative knowledge points. * Knowledge point 1: A match must be made between the customer's requirements and the developer's available resources before product development starts. As noted earlier, DOD plans to start SBR product development in 2006. * Knowledge point 2: The product's design must be stable and must meet performance requirements before initial manufacturing begins. * Knowledge point 3: The product must be producible within cost, schedule, and quality targets and demonstrated to be reliable before production begins. Systems engineering is a technical management tool that provides the knowledge necessary at knowledge point 1 to translate requirements into specific, achievable capabilities. With systems engineering knowledge in hand, acquisition decision makers and developers can work together to close gaps between requirements and available resources--well before product development starts. Some gaps can be resolved by the developer's investments, while others can be closed by finding technical or design alternatives. Remaining gaps--capabilities the developer does not have or cannot get without increasing the price and timing of the product beyond what decision makers will accept--must be resolved through trade-offs and negotiations. Effective use of this tool enables decision makers to move on to knowledge point 2 and to produce a stable product design. DOD has recently issued a new acquisition policy for space systems, partly intended to address past acquisition problems and provide capability to users quicker. However, we recently reported that the policy is not likely to achieve these goals because it allows programs to continue to develop technologies after product development starts. Our past work has shown that this approach makes it more difficult to estimate cost and schedule at the onset of product development and increases the likelihood that programs will encounter technical problems that could disrupt design and production and require more time and money to address than anticipated. Over the long run, the extra investment required to address these problems could reduce funding for developing other technological advances, slow the overall modernization effort, delay capabilities for the warfighter, and force unplanned--and possibly unnecessary--trade-offs between space and other weapon system programs. By contrast, DOD's revised policy for other weapon acquisitions encourages programs to mature technologies to the point of being tested in an operational environment before beginning product development. We recommended that DOD modify its policy to separate technology development from product development so that needs can be matched with available technology, time, and money at the start of a new program. We also reported that DOD's space acquisition policy does not require DOD to commit to setting aside funding for space acquisitions. Hence, there is no guarantee that the resources needed to meet requirements will be there on any individual program when needed. This makes it difficult for DOD as a whole to make corporate-level and trade-off decisions--which will likely be needed when DOD begins the SBR acquisition because (1) costs are significantly increasing for other critical space systems such as the Space-Based Infrared System High, the Transformational Satellite, and the Evolved Expendable Launch Vehicle and (2) DOD is planning to undertake additional new programs, such as the Space-Based Space Surveillance system and a new version of the Global Positioning System. DOD is revising its new space acquisition policy partly to address these issues; however, the revision was not available for review at the time of this review. DOD Moving Forward on Acquiring Critical Knowledge but Gaps Remain in Approval for SBR Requirements: DOD has bolstered the SBR acquisition program by increasing senior leader and stakeholder involvement in setting requirements. However, DOD is not fully documenting commitments made during the requirements approval process before progressing to the next acquisition phase, nor has it established a process to resolve potential disagreements that may occur after approval. Clouding the approval of requirements is that DOD's current space acquisition policy does not provide specific guidance for acquisitions that involve partnerships between DOD and the intelligence community. SBR Managed by New Executive Oversight Structure: Providing senior-level oversight are three new groups created expressly for the SBR program: the Executive Steering Group, which advises the Requirements/Capabilities Group and the Joint Senior Acquisition Group. Members of these groups come from DOD, National Reconnaissance Office, and National Geospatial-Intelligence Agency. All key stakeholders are expected to have open and honest discussions about what can and cannot be done within desired time frames, budgetary constraints, and achievable technologies. Figure 2 shows how these groups work with SBR's joint program office and requirements review boards for DOD and the intelligence community. Figure 2: SBR's Oversight Structure: [See PDF for image] [End of figure] A primary benefit of having an oversight structure for the SBR program, which involves many decision makers from across multiple organizations, is that the right people are involved in the decision-making process and can work together to lock in their requirements. The intent is to avoid problems of the past in which a program incurs cost, schedule, and performance risks because decision makers continue to negotiate and make trade-offs even after designers and engineers have started technology development and design work. Figure 3 shows the likely outcomes if requirements are poorly defined and are not approved or, in the case of SBR, if requirements are adequately defined and approved early in the study phase. Figure 3: Requirements-Setting Can Impact Acquisition Process: [See PDF for image] [End of figure] SBR's Requirements-Setting Process Lacks Formal Approval and Documentation: DOD officials reported to us that the oversight groups have achieved informal consensus on requirements for SBR. However, this approval has not been formalized and it is unclear as to whether and how it might be formalized. Moreover, it is unclear how disagreements that may occur after initial approval will be resolved. Regardless of how many stakeholders have been invited to join in decision making or how much expertise is included in SBR's oversight function, overall success of the SBR program hinges in part on whether the requirements are clear, stable, and achievable and whether DOD and the intelligence community demonstrate commitment and accountability by formally approving the requirements. In an acquisition decision memorandum, the Under Secretary of the Air Force requested that DOD and the intelligence community approve the initial capabilities document and concept of operations before the request for proposals was released in January 2004 for concept development contracts. DOD officials told us that the Joint Requirements Oversight Council and the intelligence community's Mission Requirements Board approved the initial capabilities document, and there are memoranda documenting these decisions. The Joint Requirements Oversight Council reviewed the concept of operations, provided comments, but did not approve it. According to DOD officials, during a meeting of the SBR Executive Steering Group, high-level officials from the intelligence community verbally approved the concept of operations, but there is no documentation recording this approval. Agreement is critical because DOD and the intelligence community are placing different emphasis on desired capabilities for SBR. An independent assessment of the SBR program determined that requirements were adequate to enter the study phase, which started in August 2003, but cautioned that the requirements needed to be converged among all stakeholders and users. Table 1 shows the type of knowledge that decision makers expect to gain from the initial capabilities document and the concept of operations. Table 1: Knowledge Provided by SBR's Initial Capabilities Document and Concept of Operations: Document: Initial capabilities document; Key knowledge provided: * Identifies current gaps in capability, * Identifies the overall desired capabilities of the SBR system; * Lists users' performance requirements for SBR; * Identifies the functional relationships between users; * Specifies the desired capability for DOD's capability development document, which is prepared at the end of the study phase; * Identifies validated requirements. Document: Concept of operations; Key knowledge provided: * Describes the components of the SBR system; * Identifies how SBR information is to be processed and disseminated to the warfighters and others; * Describes how SBR fits into architectures involving other space and non-space systems in meeting requirements; * Identifies external threats to SBR and the perceived operational environment. Sources: DOD (data); GAO (analysis). [End of table] A defined requirements approval process helps decision makers resolve disagreements that may occur and ensure they will remain committed to their decisions after formal approval. Based on our past reports on uncovering problems and our best practice work, we believe that the steps in a formal approval process include: * explaining how decision makers' requirements and comments are obtained and addressed; * identifying the officials and/or the organizations responsible for taking specific approval action; * establishing a mechanism and time frame for providing approval or disapproval; * establishing a system for addressing unresolved issues as they relate to key program documentation; and: * assessing changes to approved requirements based on their effect on the program's cost and schedule. While DOD has taken steps to increase senior leader and stakeholder involvement in setting requirements and addressing acquisition issues, DOD is not fully documenting commitments made during the requirements approval process, nor has it established a process to resolve potential disagreements that may occur after approval. DOD Taking Proactive Steps to Gain Knowledge about Resources, but Critical Gaps May Remain at Product Development: DOD is also taking positive steps to attain the knowledge needed to understand what resources will be needed to develop SBR's capabilities and to mitigate risks. These include: * relying on systems engineering to translate requirements into specific, achievable capabilities and to close gaps between requirements and resources; * adopting a more comprehensive cost estimating technique to identify SBR's life-cycle costs; * exploring alternatives for SBR if TCA--the infrastructure that DOD is depending on to transmit SBR's data--incurs schedule slips; and: * asking two concept development contractors to each propose at least two different operations concepts for SBR with and without TCA. However, the path that SBR is on has potential for knowledge gaps when making investment decisions, the types of gaps that have hampered other space programs in the past. Specifically, it is expected that some critical SBR technologies will not be mature when product development starts, that is, not tested in a relevant or operational environment. Typical outcomes of this lack of knowledge are significant cost and schedule increases because of the need to fix problems later in development. Furthermore, TCA, a new, more robust communications infrastructure that could transmit SBR's imagery data much more quickly than the current infrastructure, is facing uncertainties. Specifically, one of TCA's primary components, the Transformational Satellite, may not be ready in time to support SBR.[Footnote 2] Without mature technologies and faced with a possible slip in the Transformational Satellite's schedule, DOD will be less able to accurately estimate total system costs before the start of product development. In addition, DOD and the Air Force may not have knowledge needed to make corporate level trade-offs between SBR and other air-based radar systems at the time it plans to make a commitment to invest in the SBR acquisition program. DOD has undertaken an analysis to weigh the merits of space-based radar. At this time, it is not known whether this analysis will be a detailed examination of the capabilities and costs of each individual radar option and combined with other radar platforms or whether the analysis will be a less rigorous examination of the mix of radar options. DOD Taking Positive Steps to Build Foundation of Knowledge about SBR Resources: DOD is planning to aggressively address technology, affordability, and integration issues by, in part, instituting robust systems engineering processes and procedures. Systems engineering is a technical management tool for gaining information on a broad array of activities related to the development of a system. For SBR, DOD plans to perform systems engineering work on requirements and their allocation, interface definitions, trade studies, risk management, performance analysis and modeling, environmental and safety planning, test planning, program protection planning, information assurance, and configuration control. Applying systems engineering to these activities would give DOD the insight and knowledge it needs to better manage the program, including ways to reduce risk and ensure the viability of concepts and requirements. DOD has also decided to take a more comprehensive approach to estimating SBR's life-cycle costs. According to the SBR program director, this marks the first time DOD has willingly presented all related costs to develop, acquire, produce, maintain, operate, and sustain the system. DOD officials stated that they wanted to identify not just direct costs, but also costs for associated infrastructure such as the costs related to modifying the ground system that will be used to support SBR as well as other systems. According to DOD, about $8 billion of the $28.6 billion life-cycle cost estimate represents costs that in the past, would not have been included in space program total cost estimates. Taking steps to more comprehensively identify SBR and SBR related costs is a positive step and will help DOD manage its portfolio of space programs. Although DOD hopes to rely on TCA to support SBR data transmissions, it is taking a proactive approach to identify and assess the viability of TCA alternatives. First, in April 2004, DOD awarded two 2-year contracts for concept development efforts that call for the identification of alternatives to TCA. For each alternative identified, the contractor is to conduct an assessment of the cost, risk, and effect on SBR's performance. DOD officials told us that when SBR initiates product development in 2006, it would know whether TCA will be available to support SBR or whether to pursue a TCA alternative. In addition, DOD also awarded two contracts totaling $510,000 for a yearlong study to propose several alternatives to TCA capable of supporting SBR's communications requirements and to analyze the viability of such alternatives. These actions have put DOD in a better position to ensure the program is successful. The two 2-year contracts that DOD awarded in April 2004 also require that at least two different viable SBR operations concepts be proposed. DOD is expecting each contractor to fully develop the alternative operations concepts. These alternative concepts could involve using unique radar processing techniques. According to DOD, it will work with each of the contractors to pare down the alternatives to a single best concept for each contractor. For the remainder of the contract performance period, the contractors would focus their attention on fleshing out the details associated with these concepts. This approach will put DOD in a better position when the time comes to select a single contractor to design the SBR system. Technologies Will Not Be Mature at Product Development Start: DOD officials have said that SBR will likely be the most technically challenging, software-intensive, and complex space system ever built by DOD. The two key pieces of hardware needed to give SBR a radar capability from space--the electronically scanned array (which steers the radar signal to an area of interest) and the on-board processor (the radar-processing unit aboard SBR)--face the highest amount of risk. The electronically scanned array can scan multiple areas of interest virtually simultaneously, allowing for simplified satellite design over conventional technology offering mechanical slew radar. The on-board processor is expected to allow the processing radar data to assure the timely and thorough delivery of imagery data that will be downlinked for transmission to the warfighter. To minimize the potential for technology development problems after the start of product development, DOD uses an analytical tool to assess technology maturity for many weapon system acquisition programs. Called Technology Readiness Levels (TRL), this tool associates a TRL with different levels of demonstrated performance, ranging from paper studies to actual application of the technology in its final form. The value of using a tool based on demonstrated performance is that it can presage the likely consequences of incorporating a technology at a given level of maturity into a product's development, enabling decision makers to make informed choices. Our previous reviews have found the use of TRLs, which range from 1 to 9, to be a best practice. (See app. I for a description of the TRL levels.) The critical technologies that will support the SBR program currently range from TRL 3 to 5. A TRL 3 means that most of the work performed so far has been based on analytical and laboratory studies. At a TRL 5, the basic technology components are integrated and tested in a simulated or laboratory environment. Table 2 shows the current TRL for each of SBR's critical technologies and the expected TRL at product development start in 2006. In general, the program office's key risk reduction efforts are scheduled to mature these technologies to TRL 5 by the middle of fiscal year 2006. These efforts include the awarding of research and development contracts to three payload contractors for efforts to continue to develop and mature these components (the electronically scanned array and on-board processor). The period of performance of each contract is about 2.5 years. Table 2: Technology Readiness Levels of SBR Critical Technologies: Critical technology: Electronically scanned array; Current TRL level[A]: 4; Expected TRL level: 5; Fiscal year[B]: 2006. Critical technology: On-board processor; Current TRL level[A]: 3; Expected TRL level: 5; Fiscal year[B]: 2006. Critical technology: Signal processing algorithms (for moving target indication); Current TRL level[A]: 4-5; Expected TRL level: 5; Fiscal year[B]: 2006. Critical technology: Information management system; Current TRL level[A]: 3; Expected TRL level: 5; Fiscal year[B]: 2006. Critical technology: Moving target indication exploitation hardware and software; Current TRL level[A]: 3-4; Expected TRL level: 5; Fiscal year[B]: 2006. Sources: DOD (data); GAO (analysis). [A] A TRL range is shown because of varying maturities between the viable suppliers. [B] Even if the SBR program office chooses to pursue a different SBR concept alternative, these technologies still would be considered critical; however, the specific technology readiness date could be different for each proposed alternative. [End of table] To mature the electronically scanned array and on-board processor technologies from a TRL 3/4 to 5, the contractors plan to conduct various developmental and integrative tasks in about 3 years. For example, one contractor plans to conduct 18 tasks to develop the electronically scanned array and 8 tasks to integrate the on-board processor with other system components. In addition, the development of the integrated circuits and programmable microcircuits that support the on-board processor requires extensive tests and evaluations and the radiation-hardening requirement further complicates the development. Given the challenges of the state-of-the-art technologies being developed and the algorithms involved, the testing programs must be rigorous and transparent and the results fully documented. We have determined that the time allotted to mature the SBR technologies to TRL 5 is ambitious given the tasks that need to be accomplished. Furthermore, the development of the signal processing algorithms and communications downlink involves significant software development. Based on our past experience of software assessments in other programs, the establishment of a structured testing regime for software development has always been underestimated. By planning to start product development in fiscal year 2006 with technologies at TRL 5, DOD is very likely to continue designing the system and to conduct other program activities at the same time it builds representative models of key technologies and tests them in an environment that simulates space conditions (such as a vacuum chamber). This approach is common with DOD space acquisitions but has a problematic history. Our past work[Footnote 3] has shown that it can lead to significant cost and schedule increases because of the need to fix problems later in development. A continuing problem is that software needs are poorly understood at the beginning of a program. We have previously recommended that DOD not allow technologies to enter into a weapon system's product development until they are assessed at a TRL 7, meaning a prototype has been demonstrated in an operational environment.[Footnote 4] DOD has accepted lower TRL thresholds for space programs because testing in an operational environment--in space, for example, or even in a relevant environment--is difficult and costly. However, DOD's new space acquisition policy does not identify what the minimum TRL level should be before starting product development for space programs, how risks should be mitigated if technologies are included in programs without full testing, or how lower TRL levels affect the confidence of cost and schedule estimates. Moreover, the policy does not address the option of maturing technologies outside a program and pulling them in once they prove to be viable. One way to mitigate technology risk is to rely on backup technologies, should newer technologies prove to be problematic during product development. According to DOD officials, there are backup technologies that are more mature for each of SBR's critical technologies. The backups are the same technologies but rely on a previous and more mature version. Using previous versions of these technologies would result in a lower level of desired performance--such as a reduced area collection rate, a reduction in the total number of targets collected per satellite per day, increased product delivery time frames to the user, an increased weight of the spacecraft, and higher cost. For example, more mature versions of the electronically scanned array exist and if used, would result in a reduction in its performance level. In addition, some previous versions of SBR technologies have not been demonstrated or tested in space. But according to DOD officials, even with backup technologies, the total performance of the SBR system can be maintained through systems engineering trades. DOD says it has been able to leverage some of the key technologies (such as the electronically scanned array) that were under development during the previous effort, Discoverer II, to demonstrate a space-based radar capability. Communications Infrastructure May Not Be Ready in Time to Support SBR: Current plans call for TCA to transmit SBR's large volume of data to ground-, air-, ship-, and space-based systems. However, one of TCA's primary components, the Transformational Satellite--which will use technologies that DOD has never before tried in space--is facing uncertainties in its scheduled 2011 launch. DOD started product development for the Transformational Satellite in December 2003 even though technologies were immature. If the Transformational Satellite falters but SBR launches as expected in 2012, then DOD will have a fully operational, new-generation satellite that is missing its primary means of data transmission. Recognizing the challenges, DOD is to decide by November 2004 whether to move forward or delay the Transformational Satellite's acquisition program and instead procure another Advanced Extremely High Frequency satellite, which already are under development and are based on mature technologies. Our analysis shows that alternatives to TCA may involve a greater reliance on processing aboard the SBR satellites, thereby increasing software development efforts. This approach would reduce the volume of data requiring transmission, allowing conventional satellite systems, such as the Advanced Extremely High Frequency satellites, to handle the transmission. Another likely alternative is to have SBR satellites transmit only selected portions of data, again, so that the Advanced Extremely High Frequency satellite could handle the lower volume of information. Finally, a dedicated system of satellites could be fielded for the sole purpose of transmitting SBR data, significantly increasing program cost and raising affordability issues. Currently, DOD is working closely with officials from the Transformational Satellite program office to evaluate the relative merits of various alternatives and to document the interfaces needed between SBR and the Transformational Satellite for each alternative. During the course of our audit work, SBR program officials met weekly with the Transformational Satellite program's integrated product teams and were coordinating efforts on a memorandum of agreement on requirements development, joint engineering practices, and studies of air-and space- based options. SBR's Cost Estimate Unlikely to Be Realistic Because of Multiple Uncertainties: Based on a notional constellation of nine (plus one spare) satellites operating in low-earth orbit, an independent cost assessment in 2003 put SBR's cost at the $28.6 billion mark, making SBR the most expensive DOD space system ever built. When this initial cost estimate is revised in 2006, before SBR's product development starts, DOD is to have decided a number of issues, such as how many satellites are to be acquired, what their capabilities will be, and at what altitude(s) the satellites are to operate. This system refinement allows DOD to develop a more realistic total system cost estimate--a critical knowledge point if a successful match between requirements and resources is to be made. However, if DOD begins product development with less than mature technologies and without knowing the availability of TCA, accurate cost estimates for SBR will be much more difficult to prepare. We have previously reported that improving the reliability of cost estimates is critical[Footnote 5] and affords DOD decision makers with the appropriate information to decide whether a weapon system is worth the overall investment and whether the time is right to proceed with such an investment. Once a total cost is known, DOD needs to secure the funding so it can design, produce, operate, and sustain the system. DOD may also lack knowledge needed to make a corporate-level decision as to how much it should invest in SBR versus air platforms with similar capabilities at the time it begins the SBR acquisition program. In November 2003, the Air Force completed an analysis of alternatives (AOA) for SBR, which was supposed to evaluate whether space-or air-based radar platforms (such as manned and unmanned aircraft with radar capabilities) or a combination of both are better suited for tracking moving targets on land or at sea and analyze the capabilities and costs of each suitable option. However, DOD officials raised a concern that the AOA only weighed the merits of various space-based solutions. The Air Force decided to undertake a follow-on study to explore the optimal ways to gather information on ground moving targets from radars based in space versus air. The plan is to also use this follow-on study as part of DOD's preparations for submitting a fiscal year 2006 budget to Congress to secure funding for SBR and other radar systems on air platforms. A more thorough AOA, completed before the start of the study phase, might conceivably have determined that air-based radar could provide many or most of the capabilities promised by space-based radar but at a fraction of the cost. Moreover, this type of analysis could help DOD officials better decide whether SBR should be initiated at a later date, when critical technologies will have been matured, or when the communications infrastructure to support SBR will be available. DOD officials have mentioned other ongoing studies that are examining the optimal mix between SBR and other platforms for specific capabilities, such as ground-moving target indication. However, it is unclear as to the extent these studies will be factored into the SBR product development start decision. Conclusions: DOD has recently embarked on a discovery and exploration phase for its SBR program. During this period, it is critical for programs to work toward closing knowledge gaps about requirements, technologies, funding, and other resources so they can be positioned to succeed when DOD decides to commit to making significant investments. For SBR, this would mean testing technologies to the point of knowing they can work as intended before starting program development, securing agreement on requirements with the intelligence community, and fully assessing the cost and benefits and risks of relying on TCA and alternatives, including different mixes of air and space-based platforms. DOD is taking positive steps toward this end, but without maturing critical technologies or securing formal commitment on requirements, it will not be able to assure decision makers that the program can be completed within cost and schedule estimates. Should DOD decide to proceed on a path that leaves open important questions, including those about technologies, then it should do so with (1) assessments of technical risks and what additional resources (in terms of time and money) would be needed to address problems that may occur during development as well as what trade-offs would need to be made with other space programs should DOD need to invest additional resources in SBR, and (2) a formal commitment for providing additional resources if problems do occur. Recommendations for Executive Action: To better ensure that DOD and its intelligence community partners obtain the additional knowledge they need to determine whether and when to begin the SBR acquisition program, we recommend that the Secretary of Defense direct the Under Secretary of the Air Force to: * Direct the SBR Executive Steering Group to ensure that outcomes from the requirements management process are formally approved and documented as the program proceeds through product development before an investment is made beyond technology and concept development for the SBR program. This group should identify how key document review comments are to be obtained and addressed and identify all the officials and/or organizations responsible for taking specific approval action. In addition, the group should establish a mechanism and time frame for providing approval/disapproval. Finally, the group should establish a formal mechanism for addressing unresolved issues as they relate to key program documentation, as well as how changes to approved requirements will be assessed. * Modify DOD's space acquisition policy to reflect protocols for setting requirements when DOD undertakes programs in partnership with the intelligence community. * Delay approval to commit funding to product development (key decision point B) for SBR until technologies have been demonstrated in a relevant or operational environment so DOD can more reliably estimate the resources needed to complete the program. If the Under Secretary determines that the program should go forward with less mature technologies, then we recommend that the Under Secretary (1) undertake an assessment of the backup technologies that may lessen capability and add cost to the program and the additional time and money that may be required to meet SBR's performance objectives to address those risks, (2) undertake an assessment of trade-offs that may need to be made with other space programs to assure SBR's successful outcome, and (3) secure formal commitments from DOD to provide funding for total estimated costs as well as costs estimated to address potential technical risks. * Strengthen the ongoing study of options for tracking ground-moving targets by ensuring this work includes: (1) a full range of air and space options; (2) measures of effectiveness that would help justify choosing SBR over air options; and (3) the possibility of having to rely on TCA alternatives for space options. This work should also consider the results of analyses being conducted by other DOD entities on tracking ground-moving targets. Agency Comments and Our Evaluation: We received written comments on a draft of this report from the Deputy Under Secretary of Defense (Programs, Requirements, and Resources) within the Office of the Under Secretary of Defense for Intelligence. DOD generally agreed with our findings and our recommendation to strengthen its study of SBR alternatives. DOD partially agreed with our recommendations to strengthen its requirement setting process for SBR and to demonstrate SBR technologies in a relevant or operational environment before committing to product development. DOD did not agree with our recommendation to modify its acquisition policy to strengthen requirements setting. In commenting on our recommendations, DOD agreed in principle with the need to extensively define, analyze, and validate requirements for SBR, but it did not believe this necessitated a different requirements setting process than the one that is in place for SBR or changes to its space acquisition policy or that additional controls were needed within the program's study phase. To clarify, our recommendation was not intended to construct a new requirements setting process or supplant activities undertaken by the Joint Requirements Oversight Council or the Mission Requirements Board, as DOD asserts. Rather, we recommend that DOD build on the positive requirements setting procedures it has already put in place by instituting controls and mechanisms that ensure transparency, discipline, and accountability with requirements setting. As noted in our report, while DOD has taken steps to increase senior leader and stakeholder involvement in requirements setting, it is not fully documenting commitments made during the requirements approval process, nor has it established a process to resolve potential disagreements that may occur after approval. It is important that this discipline be instilled in the study phase and throughout the SBR effort. As noted in previous reports, many space programs have not been executed within cost and schedule estimates because of an inability to establish firm requirements and to make and enforce trade-off decisions. For SBR, the potential for difficulty in requirements setting is higher because of the distinct needs of the intelligence community and DOD's desire to integrate SBR with other radar platforms. Moreover, revising the acquisition policy to clearly communicate protocols that should be followed when DOD undertakes space programs in the future involving diverse users--such as the intelligence community, military services, industry, and/or other agencies--would further help DOD to rationalize requirements setting and to solidify relationships with users, which DOD reported was a top SBR management issue. In regard to our recommendation to delay product development until SBR technologies are sufficiently matured, DOD stated that it has planned for critical and most other enabling technologies to be demonstrated at least at the component level in a relevant environment on the ground. DOD also stated that where technically feasible and fiscally feasible, it planned to pursue on-orbit demonstrations. It also stated it has taken some actions relating to our recommendation such as accounting for technical risks in the costing and budgeting process. DOD asserted, however, that our recommendation encourages pursuit of older, more proven, technologies. We recommended that DOD pursue relevant or operational environment demonstrations of all critical technologies and even an integrated system before committing to a formal acquisition program because this practice enables a program to align customer expectations with resources, and therefore minimize problems that could hurt a program in its design and production phase and drive up costs and schedule. Further, we agree that continuing to develop leading edge technology is important for space system capabilities. However, history has shown and we have repeatedly reported that conducting technology development within a product environment consistently delays the delivery of capability to the user, robs other programs of necessary funds through unanticipated cost overruns, and, consequently, can result in money wasted and fewer units produced than originally stated as necessary. A technology development environment is more forgiving and less costly than a delivery-oriented acquisition program environment. Events such as test "failures," new discoveries, and time spent in attaining knowledge are considered normal in this environment. Further, judgments of technology maturity have proven to be insufficient as the basis for accurate estimates of program risks as it relates to cost, schedule, and capability. Lastly, our report noted that DOD was taking positive actions to gain knowledge about technology readiness, including strengthening systems engineering, undertaking risk assessments, and assessing various technical concepts. Given the potential cost of the program, our recommendation focuses on taking these steps further by assessing what trade-offs may need to be made with other space programs should the program encounter technical problems that require more time and money than anticipated and securing commitments to provide resources needed to address such problems. DOD's detailed comments are provided in appendix II. We plan to provide copies of this report to the Secretary of Defense, the Secretary of the Air Force, and interested congressional committees. We will make copies available to others upon request. In addition, the report will be available on the GAO Web site at http:// www.gao.gov. If you or your staff has any questions concerning this report, please contact me at (202) 512-4841 or Arthur Gallegos at (303) 572-7368. Other key contributors to the report include Tony Beckham, Cristina Chaplain, Lily Chin, Maria Durant, Nancy Rothlisberger, and Hai V. Tran. Signed by: Katherine V. Schinasi: Managing Director: Acquisition and Sourcing Management: [End of section] Appendix I: TRL Scale for Assessing Critical Technologies: TRL: Basic principles observed and reported; TRL description: Lowest level of technology readiness. Scientific research begins to be translated into applied research and development (R&D). Examples might include paper studies. TRL: Technology concept and/or application formulated; TRL description: Invention begins. Once basic principles are observed, practical applications can be invented. Examples are still limited to paper studies. TRL: Analytical and experimental critical function and/or characteristic proof of concept; TRL description: Active R&D is initiated. This includes analytical studies and laboratory studies to physically validate analytical predictions of separate elements of the technology. Examples include components that are not yet integrated or representative. TRL: Component and/or breadboard validation in laboratory environment; TRL description: Basic technological components are integrated to establish that they will work together. This is relatively "low fidelity" compared to the eventual system. Examples include integration of "ad hoc" hardware in a laboratory. TRL: Component and/or breadboard validation in relevant environment; TRL description: Fidelity of breadboard technology increases significantly. Basic technological components are integrated with reasonably realistic supporting elements so that the technology can be tested in simulated environment. Examples include "high fidelity" laboratory integration of components. TRL: System/subsystem model or prototype demonstration in a relevant environment; TRL description: Representative model or prototype system, which is well beyond the breadboard tested for level 5, is tested in a relevant environment. Represents a major step up in a technology's demonstrated readiness. Examples include testing a prototype in a high- fidelity laboratory environment or in a simulated operational environment. TRL: System prototype demonstration in an operational environment; TRL description: Prototype near, or at, planned operational system. Represents a major step up from TRL 6, requiring demonstration of an actual system prototype in an operational environment such as an aircraft, vehicle, or space. TRL: Actual system completed and qualified through test and demonstration; TRL description: Technology has been proven to work in its final form and under expected conditions. In almost all cases, this TRL represents the end of true system development. Examples include Developmental Test and Evaluation of the system in its intended weapon system to determine if it meets design specifications. TRL: Actual system proven through successful mission operations; TRL description: Actual application of the technology in its final form and under mission conditions, such as those encountered in Operational Test and Evaluation. Examples include using the system under operational mission conditions. Source: DOD Interim Defense Acquisition Guidebook, app.6 (Oct. 30, 2002). [End of table] [End of section] Appendix II: Comments from the Department of Defense: Note: GAO comments supplementing those in the report text appear at the end of this appendix. OFFICE OF THE UNDER SECRETARY OF DEFENSE: 5000 DEFENSE PENTAGON: WASHINGTON, DC 20301-5000: INTELLIGENCE: Ms. Katherine V. Schinasi: Managing Director: Acquisition and Sourcing Management: U.S. General Accounting Office: 441 G Street, N.W.: Washington, D.C. 20548: Dear Ms. Schinasi, This is the Department of Defense (DoD) response to the GAO draft report, 'DEFENSE ACQUISITIONS: Space-Based Radar Effort Needs Additional Knowledge Before Starting Development,' dated May 25, 2004 (GAO Code 120267/GAO-04-759). This report did an excellent and thorough job reviewing the Space Based Radar (SBR) program and we agree with many of your findings and comments; however we can only partially concur with the entire report, particularly your recommended actions. Detailed responses to these recommendations are provided in the attachment. In summary, the recommendation to provide more formal documentation to the requirements process assumes that DoD and the Intelligence Community (IC) have committed to SBR as a joint (i.e., DoD/IC) program, which is not the case. The current SBR acquisition strategy requires an iterative process of reviewing several concept design proposals from multiple vendors to determine whether a single space radar system can satisfy the broad spectrum of DoD and IC requirements. This does not alleviate the need to extensively define, analyze, and validate requirements; however a process outside of, and more formalized than, the one currently in place is not required. Also, a modification of the space acquisition policy is not required specifically for this program. As far as your recommendation to delay funding for product development until SBR technologies have been demonstrated in an operational environment, we have planned for critical and most other enabling technologies to be demonstrated at least at the component level in a relevant environment on the ground. Where technically and fiscally possible and technically warranted, the JPO is pursuing on-orbit demonstrations. Nevertheless, we cannot accept your recommendation to employ a more mature technology which would produce "a lower level of desired performance." The SBR Joint Program Office has established a risk reduction and technology readiness process which ensures extensive testing and validation of related technologies. In addition, we are planning a classified demonstration which will prove the utility of SBR-like capabilities. If your staff has not been briefed on this project, we can arrange for appropriately cleared personnel from your office to receive this briefing. Finally, while the Air Force-conducted Analysis of Alternatives clearly demonstrates the utility of SBR (in a variety of constellations) to satisfy warfighting needs, more studies are required to analyze contributions of airborne assets in a variety of scenarios and to further refine SBR and Moving Target Indicator (MTI) contributions to intelligence problems. These studies are either underway or in various stages of completion, and both DoD and the IC are extensively involved. In addition, the relationship and dependencies of SBR and TCA are understood. The Joint Staff has tasked additional study of this relationship for formal review as part of the Joint Requirements Oversight Council process. Communications alternatives to TCA are an integral part of the SBR Phase A studies. Again, I commend your staff on a professional effort, both in their investigation, in their dealings with DoD personnel, and in the preparation of the final report. My point of contact for this effort is Mr. Brian P. Levengood, 703-607-0423. Please contact him if you have any questions or comments. Signed by: Letitia A. Long: Deputy Under Secretary of Defense: (Programs, Requirements, & Resources): GAO DRAFT REPORT -DATED MAY 26, 2004 GAO CODE 120267/GAO-04-759: "DEFENSE ACQUISITIONS: SPACE-BASED RADAR EFFORT NEEDS ADDITIONAL KNOWLEDGE BEFORE STARTING DEVELOPMENT": DEPARTMENT OF DEFENSE COMMENTS TO THE RECOMMENDATIONS: RECOMMENDATION 1: The GAO recommended that the Secretary of Defense direct the Under Secretary of the Air Force to direct the Space-Based Radar (SBR) Executive Steering Group to ensure that outcomes from the requirements management process are formally approved and documented as the program proceeds through product development before an investment is made beyond technology and concept development for the SBR program. The Steering Group should identify how key document review comments are to be obtained and addressed and identify all the officials and/or organizations responsible for taking specific approval action. The Steering Group should also establish a mechanism and time frame for providing approval/disapproval of requirements and a formal mechanism for addressing unresolved issues as they relate to key program documentation, as well as how changes to approve requirements will be addressed. (p. 17/GAO Draft Report): DOD RESPONSE: Partially Concur. If the SBR concept development phase results in a satellite design and constellation that satisfactorily provides the capabilities needed to satisfy both Department of Defense (DoD) and Intelligence Community (IC) requirements, a more formal process, with appropriate supporting documentation may be required as the acquisition progresses beyond Key Decision Point (KDP) B. The current acquisition process (now in Phase A) includes requirements from both the DoD and the IC to ascertain whether a single satellite system can satisfy the broad spectrum of cross-community needs. It is imperative that the respective requirements validation processes (the Joint Requirements Oversight Council (JROC) for DoD, and the Mission Requirements Board (MRB) for the IC) function in accordance with their charters. The process put in place specifically to review and approve requirements for SBR (Requirements and Capabilities Group (RCG) and the Executive Steering Group (ESG)) was designed to build on the existing requirements validation process, not to replace it or usurp any JROC or MRB authorities. The inference drawn in the GAO report that the apparent informality of the Executive Steering Group process renders that process ineffective in providing direction and stability for SBR requirements is not correct. The senior level representation of the ESG (e.g., Under Secretary of Defense, Deputy Director of Central Intelligence, Directors of Defense Agencies, Under Secretary of the Air Force, etc.), militates against reversals of requirements and strategic direction that have been adjudicated and approved by this group. The Under Secretary of the Air Force has stood up the National Security Space Office organization to provide valuable staff support and integration of several functional areas and will provide documentation of SBR ESG findings. The GAO report also points out that the Initial Capabilities Document (ICD) was approved by the JROC and MRB through memoranda, but that neither provided written approval of the Concept of Operations (CONOPS). In accordance with Title X, DoD Instructions, and DCI Instructions; neither the JROC nor the MRB have formal approval authority over acquisition program CONOPS. However, the JROC does routinely review CONOPS as part of the supporting documentation for capability documents and, in response to the Under Secretary of the Air Force's request, noted their support for using the current CONOPS for Phase A "Concept Development" in formal memoranda, JROCM 221-03. There is a concerted and cooperative effort between the DoD and IC to come to a convergence of capability needs using the RCG process to draft the Capabilities Development Document (CDD). The CDD is the formal statement of capability needs required for a KDP B decision to enter Phase B "Development." In accordance with CJCSI 3170 and DCID 1/10, the final draft CDD will be submitted to the JROC and MRB prior to KDP B for formal review, comment adjudication, and approval. We believe the processes in place to validate requirements for the DoD and IC are adequate and should not be replaced. Further, the SBR-unique process is sufficient, when acting in concert with the JROC and MRB, to adjudicate any differences and to support Phase A of the SBR acquisition process. We partially concur with your recommendation that a joint single SBR acquisition may require a more formalized, document- supported process beyond Phase A. RECOMMENDATION 2: The GAO recommended that the Secretary of Defense direct the Under Secretary of the Air Force to modify DOD's space acquisition policy to reflect protocols for setting requirements when DOD undertakes programs in partnership with the intelligence community. (p. 17/GAO Draft Report): DOD RESPONSE: Non-Concur. The Under Secretary of the Air Force also acts as the Director of the National Reconnaissance Office (NRO). The National Security Space (NSS) Acquisition Policy (03-01) combines elements of the DoD acquisition process with those used by the NRO. This process outlined in NSS 03-01 also requires Key Decision Points and Defense Space Acquisition Boards which include IC participation. Further, the term "partnership" throughout the GAO report implies a programmatic commitment which does not currently exist. SBR is wholly funded in the Defense budget (primarily in the Air Force Tactical Intelligence and Related Activities and partially in the Joint Military Intelligence Program). We believe that, even if the IC agreed to a co- funding arrangement for SBR, NSS 03-01 sets out procedures which do not need to be modified specifically for SBR. NSS 03-01 supports the steps already underway to improve joint requirements protocol. An example is that NSS Acquisition Policy 03-01 embraces the new Joint Capabilities Integration and Development System (JCIDS) concept as outlined in CJCSI 3170.01d, which talks extensively about working closer with the Intelligence Community MRB. Under this construct, the JROC reviews programs at the request of the USecAF (as DOD Space Milestone Decision Authority (MDA)) or the MRB. When national intelligence capabilities exist, capabilities integration and development efforts by the Intelligence Community are encouraged to follow a parallel path between the defense and national intelligence communities with resulting capabilities documents validated and approved by the JROC and the Director of Central Intelligence MRB. Further, the Functional Capability Boards Principal Membership roster includes positions for, when appropriate, Defense Intelligence Agency representatives, the Under Secretary of the Air Force (as the DOD Space Milestone Decision Authority), the Under Secretary of Defense for Intelligence, and MRB Executive Staff. JCIDS products (e.g., ICD, CDD, CPD) are required as part of the DSAB/KDP process described in NSS 03- 01. Further, the dual responsibilities of the USecAF/DNRO facilitate management of SBR acquisition while protecting DoD and IC equities and providing the capabilities required to satisfy the cross-community requirements. RECOMMENDATION 3: The GAO recommended that the Secretary of Defense direct the Under Secretary of the Air Force to delay approval to commit funding to product development (key decision point B) for SBR until technologies have been demonstrated in a relevant or operational environment so DOD can more reliably estimate the resources needed to complete the program. If the Under Secretary determines that the program should go forward with less mature technologies, then we recommend that the Under Secretary: (1) undertake an assessment of the backup technologies which may lessen capability and add cost to the program and the additional time and money that may be required to meet SBR's performance objectives to address those risks; (2) undertake an assessment of trade-offs that may need to be made with other space programs to assure SBR's successful outcome; and (3) secure formal commitments from DOD to provide funding for total estimated costs as well as costs estimated to address potential technical risks. (p. 17/GAO Draft Report): DOD RESPONSE: Partially Concur. We agree that SBR product development should not proceed until technologies have been sufficiently demonstrated; however, The Secretary of Defense need not direct the Under Secretary of the Air Force to delay approval of KDP-B to meet GAO recommended technology maturity levels, because the current program baseline already includes demonstration of all critical and most other enabling technologies in a relevant or operational environment prior to KDP-B. Further, while NSS Acquisition Policy 03-01 does state that technology development is part of Phase B, this phase is primarily focused on risk reduction; component level technology development is finished by PDR. These risk reductions activities are similar to the DoDI 5000.2 Phase B activities to reduce integration and manufacturing risks. In addition, technology maturity is assessed at each KDP: "At each KDP, the program office should identify the key technology components of the system and provide their assessment of the maturity of each key component using the Technology Readiness Level (TRL) method identified in the Interim DoD Acquisition Guidebook. The IPAT will review the program office assessment and determine if, in their view, all key technology components of the program have been identified. The IPA will also provide its own independent assessment of the maturity of the key components using the TRL method. The intent is not to require a specific TRL for each key component in order to proceed into the next acquisition phase, but to instead allow for the DoD Space MDA to be made knowledgeable of the state of key component maturity so appropriate direction can be given in the ADM for additional technology maturation/risk reduction activities." The GAO Report recommends that we pursue lower risk technology which admittedly "would result in a lower level of desired performance." (p.14) Concepts developed under Phase A should maximize capability and employ technological advances, but they must be affordable and producible, not technically feasible at some future date. We believe the Technology Readiness Level processes in place will ensure that. If the MDA, based on the findings and recommendations of the Independent Cost Assessment Team, Independent Program Assessment Team, and the larger Defense Space Acquisition Board process, is not convinced the technology is mature, he will provide the appropriate direction to the program to ensure maturation occurs, including possibly delaying entry into the next phase until ready or directing the program to conduct additional reviews to assess maturity within a given timeframe. Key and critical and most other enabling technologies will be demonstrated at least at the component level in a relevant environment on the ground. Select technologies have been identified for demonstration in a relevant environment at a system or subsystem level, and, where technically and fiscally possible and technically warranted, the JPO is pursuing on-orbit demonstrations. A classified program plans to demonstrate certain aspects of an SBR MTI capability. Appropriately cleared members of your staff are invited to receive this briefing. The SBR Program is putting a major focus on technology maturity to mitigate the performance, schedule and cost risks represented by the critical and most other enabling technologies. Going back as far as the Discoverer II (DII) activities and continuing through the recently completed Concept Definition activities and the on-going Phase A activities, the SBR Program has, and is continuing, to invest heavily in technology maturation. One example of continued technology maturation is the Electronically Scanned Array (ESA) Technology development that was initiated under DII activities, and has been leveraged since then by the JPO. To ensure that the risks remaining are accounted for in the costing and budgeting process, the cost estimating approach includes discrete technology related development risks and more general cost estimating uncertainties due to technical and schedule issues. Our cost estimating approach has been validated by the OSD CAIG at the 18 Mar 04 USECAF SBR Cost Review. (1) The SBR JPO is aggressively pursuing technology paths that will best satisfy SBR requirements. At the same time, the JPO is maintaining back up technologies, which have been proven on other programs, to ensure there are mitigation paths if the desired technologies run into issues. Phase A also includes a robust systems engineering effort which will maintain technology focus and ensure overall performance does not suffer even if the JPO has to sub-optimize with older backup technologies. The technology maturity assessment process described in NSS Acquisition Policy 03-01 will ensure an objective assessment of the most appropriate technology path for program execution. The JPO cost and schedule estimates will be updated for KDP-B to reflect the chosen path. (2) According to analysis thus far, a delay in the deployment of TCA is something SBR could absorb without substantial impact. Since the ADM was released by Mr. Teets on 19 Aug 2003, SBR JPO has been evaluating an extensive trade space for its communications architecture during its deployment period. Deliverables due by KDP-B include one schema that incorporates TCA and another that allows SBR to function independently during its deployment phase. Current trades for SBR's initial deployment period include leveraging residual national resources and future MILSATCOM assets or incorporating RF downlink capabilities directly from SBR for dissemination in theater or through the future Global Information Grid. Furthermore, the prime contractors for SBR are incorporating and expanding on the ongoing analysis done by the JPO. (3) Both the independent program assessment team and the independent cost assessment team stated that the FYDP funding was adequate for Phase A efforts. The on-going Phase A efforts with industry will result in more detailed technical and programmatic baseline definition. This more detailed definition will support KDP-B, the generation of an acquisition program baseline, and updated cost estimates and supporting budgets. We agree that SBR contains some technology risk, but we believe we can reduce this risk sufficiently to proceed with our desired SBR capabilities without reducing performance at this point in the program. Therefore, we can only partially concur with this recommendation. RECOMMENDATION 4: The GAO recommended that the Secretary of Defense direct the Under Secretary of the Air Force to strengthen ongoing study of options for tracking ground-moving targets by ensuring this work includes: (1) a full range of air and space options; (2) measures of effectiveness that would help justify choosing SBR over air options; and (3) the possibility of having to rely on Transformational Communications Architecture alternatives for space options. (P. 17/GAO Draft Report): DOD RESPONSE: Concur. We agree that SBR studies must include air/space options, comparable measures of effectiveness, and communications considerations. We believe that studies currently underway incorporate these criteria; however, not all of these studies are being performed under the auspices of the Under Secretary of the Air Force. Therefore, direction from the Secretary of Defense to the Under Secretary of the Air Force to incorporate these criteria would be redundant in some cases and not relevant in others. The Air Force Analysis of Alternatives proved the utility of SBR to warfighter requirements, and it also included airborne collectors in certain scenarios; however, it was not chartered to and did not undertake an extensive evaluation of specific airborne platforms in cooperative collection operations. For this reason, a follow-on Air Force study as well as an OSD Programs Analysis and Evaluation Study are looking more specifically at aircraft contributions. In addition, the IC continues to work with the National Geospatial Intelligence Agency to ascertain the contribution of MTI to intelligence problems. We concur that measures of effectiveness are critical in understanding the relative contributions of each collection asset and will critically review the results of these ongoing studies to ensure valid comparisons are being made. The relationship of SBR and TCA are of concern to DoD, and the JROC has recently sanctioned a study to analyze the interdependencies and impacts of SBR and TCA remaining in synch. SBR's key dependency on TCA is an area that will require close monitoring and review prior to a KDP B decision. The JROC will rely heavily on the Battlespace Awareness and Net Centric Functional Capability Boards to provide a joint assessment of related trade analysis conducted during Phase A. SBR Phase A studies, as directed by USecAF, are looking at a number of communications alternatives to TCA. The SBR program has a flexible communication strategy intended to reduce dependencies on any specific TCA implementation at this early stage (Phase A) of the program. Initial SBR constellation configurations are not expected to be fully dependent on Transformational Communications. Specific risk mitigation alternatives have been identified in order to facilitate SBR operations during the initial SBR deployment phase while the TCA is also being deployed. GAO's Comments: The following are GAO's comments on the Department of Defense's letter dated June 29, 2004. 1. DOD stated that DOD does not require formal approval for the concepts of operations from the Joint Requirements Oversight Council or the Mission Requirements Board, but noted that the Joint Requirements Oversight Council communicated agreement in a memo. As we reported, the Under Secretary of the Air Force requested that both DOD and the intelligence community approve the initial capabilities document and concept of operations in light of the complexity of SBR's acquisition process, the partnership with the intelligence community, and the proposed integration with other radar platforms. 2. DOD stated that it is not engaged in a partnership with the intelligence community on SBR, as our report states. Specifically, DOD stated that SBR is wholly funded in the defense budget and that a programmatic commitment with the intelligence community does not exist. DOD's SBR System Acquisition Strategy was signed by senior-level officials from DOD, National Reconnaissance Office, and the National Geospatial-Intelligence Agency and approved on January 14, 2004. This strategy states that the Air Force, in close partnership with the National Reconnaissance Office and National Geospatial-Intelligence Agency, is responsible for leading development of an SBR capability. This strategy further identifies the responsibilities related to SBR that each mission partner (National Reconnaissance Office and National Geospatial-Intelligence Agency) is supposed to carry out. We disagree with DOD's assertion that these organizations must provide funding to SBR in order to consummate a partnership. Because SBR is being justified on the basis of the system's ability to provide intelligence, surveillance, and reconnaissance products to both DOD and the intelligence community, the part of the budget used is not relevant to our finding. To clarify, we did not recommend that DOD pursue lower risk technologies that would result in lower levels of desired performance. Instead, we reported that DOD might have to resort to using backup technologies if the current ones prove to be problematic during product development. We recommended that DOD should assess the cost to the program of having to use the backup technologies DOD has already identified in terms of time and money. [End of section] FOOTNOTES [1] U.S. General Accounting Office, Military Space Operations: Common Problems and Their Effects on Satellite and Related Acquisitions, GAO- 03-825R (Washington, D.C.: June 2, 2003). [2] U.S. General Accounting Office, Space Acquisitions: Committing Prematurely to the Transformational Satellite Program Elevates Risks for Poor Cost, Schedule, and Performance Outcomes, GAO-04-71R (Washington, D.C.: Dec. 4, 2003). [3] U.S. General Accounting Office, Military Space Operations: Common Problems and Their Effects on Satellite and Related Acquisitions, GAO- 03-825R (Washington, D.C.: June 2, 2003). [4] U.S. General Accounting Office, Best Practices: Better Management of Technology Development Can Improve Weapon System Outcomes, GAO/ NSIAD-99-162 (Washington, D.C.: July 30, 1999). [5] U.S. General Accounting Office, Defense Acquisitions: Improvements Needed in Space Systems Acquisition Management Policy, GAO-03-1073 (Washington, D.C.: Sept. 15, 2003). GAO's Mission: The Government Accountability Office, the investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO's Web site ( www.gao.gov ) contains abstracts and full-text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics. Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as "Today's Reports," on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select "Subscribe to e-mail alerts" under the "Order GAO Products" heading. Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. Government Accountability Office 441 G Street NW, Room LM Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov Automated answering system: (800) 424-5454 or (202) 512-7470: Public Affairs: Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, D.C. 20548: