This is the accessible text file for GAO report number GAO-09-295R entitled 'Defense Acquisitions: Perspectives on Potential Changes to Department of Defense Acquisition Management Framework' which was released on February 27, 2009. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. GAO-09-295R: United States Government Accountability Office: Washington, DC 20548: February 27, 2009: Congressional Committees: Subject: Defense Acquisitions: Perspectives on Potential Changes to Department of Defense Acquisition Management Framework: The National Defense Authorization Act (NDAA) for Fiscal Year 2008 directed GAO to report on potential modifications of the organization and structure of the Department of Defense (DOD) for Major Defense Acquisition Programs (MDAP). In preparing the report, the review was required to include the feasibility and advisability of seven potential modifications of DOD's organization and structure for MDAPs.[Footnote 1] We were charged with reporting on the feasibility and advisability of (1) establishing a process in which the commanders of combatant commands (COCOM) assess and provide input on the capabilities needed to successfully accomplish their missions over a long-term planning horizon of 15 years or more; (2) establishing a materiel solutions process for addressing identified gaps in critical warfighting capabilities, under which the Under Secretary of Defense for Acquisition, Technology, and Logistics (USD (AT&L)) circulates among the military departments and appropriate defense agencies requests for proposals for technologies and systems to address such gaps; (3) revising the acquisition process by establishing shorter, more frequent acquisition program milestones; (4) requiring the milestone decision authority (MDA) to specify at the time of milestone B approval the period of time that will be required to deliver an initial operational capability (IOC) to the commanders of the relevant COCOMs; (5) establishing a new office to provide independent cost and performance estimates; (6) requiring certifications of program status to the Defense Acquisition Executive (DAE) and Congress prior to milestone approval; and (7) modifying the role played by Chiefs of Staff of the Armed Forces in the requirements, resource allocation, and acquisition processes.[Footnote 2] See enclosure II for the full wording of the potential changes. In carrying out this review, as Congress directed, we obtained the views of current and prior senior DOD acquisition officials, currently serving senior military officers involved in setting requirements, and other experts including some who participated in previous reviews of DOD's organization and structure for carrying out the acquisition of major weapon systems.[Footnote 3] We also reviewed and made extensive use of GAO's prior work and studies commissioned by DOD or the President--listed in enclosure IV. We conducted this performance audit from January 2008 to February 2009 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. For more on our scope and methodology, see enclosure I. Results in Brief: Based on our discussions with acquisition experts and reviews of our own and key studies on DOD acquisition, we believe that six of the seven potential changes included in the fiscal year 2008 NDAA could help improve DOD's acquisition processes with some modifications, but only if they are accompanied by additional changes in the overall acquisition environment. Specifically, the six changes are (1) establishing a process in which the combatant commanders assess and provide input on the long-term capabilities needed to successfully accomplish their missions; (2) establishing a materiel solutions process for addressing identified gaps in critical warfighting capabilities, under which USD (AT&L) circulates among the military departments and appropriate defense agencies requests for proposals for technologies and systems to address such gaps; (3) revising the acquisition process by establishing shorter, more frequent acquisition program milestones; (4) requiring the MDA to specify at the time of milestone B approval the period of time that will be required to deliver an IOC to the relevant combatant commanders; (5) establishing a new office to provide independent cost and performance estimates; and (6) requiring certifications of program status to the DAE and Congress prior to milestone approval. The sixth potential change appears to have been instituted already in recent legislation[Footnote 4]. Finally, we could not determine whether the remaining potential change--modifying the role played by Chiefs of Staff of the Armed Forces in the requirements, resource allocation, and acquisition processes--would improve acquisition outcomes as this potential change requires further definition and more study. Our discussions with acquisition experts indicate that these changes may not achieve the desired improvement in acquisition outcomes unless they are accompanied by changes in the overall acquisition environment, its culture, and the incentives provided for success. Some of the DOD executives we interviewed--both current and former--emphatically stated that these changes will not improve the acquisition system until the department can significantly improve its prioritization of needs so that fewer programs are competing for available dollars. Furthermore, some of the experts expressed a strong belief that none of the potential changes may be acceptable to some process stakeholders because some stakeholders like the process as it is. Another expert characterized the acquisition process as a system in equilibrium. It will take strong, consistent strategic leadership to change the current culture and environment. DOD did not provide a written response to a draft of this correspondence, but provided technical comments which we have incorporated as appropriate. Background: DOD's MDAPs are carried out under an overall defense acquisition management framework comprised of three major processes: one for requirements definition, one for resource allocation, and one for the development and production of specific systems.[Footnote 5] Within that framework, the requirements process identifies, assesses, and prioritizes warfighting needs utilizing DOD's Joint Capabilities Integration and Development System (JCIDS); the resource allocation process governs the distribution of financial resources across DOD and the military services through DOD's Planning, Programming, Budgeting, and Execution System (PPBES) to meet validated requirements; and the Defense Acquisition System (DAS) serves to manage the execution of product development and procurement.[Footnote 6] DOD's overall acquisition management framework in theory integrates three interdependent and stable processes of budget, acquisition, and requirements and is built on a foundation that requires stability and continuity. Within that framework, the DAS is the acquisition process that defines how to buy weapon systems, but it does not include requirements and budget. The system involves contracting, program management, developmental and operational test and evaluation, production, and initial fielding of weapon systems. While all these activities are necessary and important, they are a subset of the strategic-level overall management framework. Together, what DOD decides to buy at the strategic level and how it develops, tests, and produces weapon systems at the program level determine the capabilities DOD delivers to the warfighter. DOD Directive 5000.01 defines the DAS as the management process used by DOD to provide weapons systems and other equipment to users. DOD Instruction 5000.02 provides more specific guidance and identifies the entrance and exit criteria necessary for an acquisition program to move through each of the phases of the acquisition process. DOD recently updated DOD Instruction 5000.02 in December 2008 to incorporate changes to policy that were the result of new or revised sections of public law enacted since 2003; approved DOD policy appearing in over 25 policy memos; and DOD responses to GAO and DOD Office of Inspector General reports. A comparison of the new and prior processes and an explanation of the major changes are provided in enclosure III. The MDA approves the program structure, including the type and number of decision points.[Footnote 7] Milestone decision points can initiate programs and authorize entry into the major acquisition process phases. Decision reviews assess progress and authorize or halt further program activity. At each milestone, the MDA (which will be the DAE or Component Acquisition Executive (CAE) depending upon the scale or complexity of the acquisition program) conducts a management review to determine if a program is ready to progress to the next phase of the framework.[Footnote 8] Milestone A review is held between the material solution analysis and technology development phases; milestone B review is held between the technology development and engineering and manufacturing development phases; and milestone C review is held between the engineering and manufacturing development and the production and deployment phases. GAO, DOD, and other organizations have issued a number of reports over the past 20 or more years examining both the overall acquisition management framework and the programmatic implementation of the DAS and making recommendations for improvements. Five reports seemed most relevant to the seven potential changes, the challenges facing defense acquisition, and ways to mitigate them--the Quadrennial Defense Review, the Center for Strategic and International Studies' (CSIS) Beyond Goldwater-Nichols series volume two, the Defense Science Board's (DSB) Summer Study on Transformation, the Defense Acquisition Performance Assessment (DAPA) and the Packard Commission report,[Footnote 9] a 1986 study which became the foundation for the Goldwater-Nichols Department of Defense Reorganization Act of 1986.[Footnote 10] Potential Changes May Improve Acquisition Outcomes: We believe that six of the seven potential changes--some with modification--could improve the acquisition process and its outcomes if accompanied by additional changes in the overall acquisition environment. The sixth potential change appears to have been instituted already in recent legislation.[Footnote 11] It is unclear whether the potential change modifying the role played by Chiefs of Staff of the Armed Forces in the requirements, resource allocation, and acquisition processes would improve acquisition outcomes because it requires further definition and more study. Establishing a Process in Which the Combatant Commanders Assess and Provide Input on the Long-Term Capabilities Needed to Successfully Accomplish Their Missions: Based on our work and other major acquisition reform studies, we believe that enduring requirements reform will occur when the combatant commanders have more influence over the requirements process to meet their priorities. This reform is necessary to create a truly joint military and lay the foundation for an acquisition process that is responsive to operational needs. COCOMs are responsible for conducting combat operations and, ultimately, ensuring that the warfighter has the capabilities needed to defeat threats. However, these commands currently have a role in the requirements process that can be strengthened. A key vehicle for informing the process is the integrated priority list (IPL), which is nonbinding and does not obligate the military services, which currently have primary responsibility over the requirements process, to fund any of the COCOM requirements.[Footnote 12] However, many of the COCOMs do not believe that their needs, which are reflected through the IPL process, are sufficiently addressed through the department's JCIDS process. This has created well-known problems among the services including duplication in capabilities and persistent interoperability problems. The potential change would provide more authority to COCOMs to ensure their long-term needs are met. We believe if this potential change were implemented, a more COCOM-focused requirements process could improve joint warfighting capabilities by ensuring that the combatant commanders--the customers--are provided the appropriate level of input regarding the capabilities needed to execute their missions rather than relying on the military services--the suppliers-- to drive requirements. Our review of DOD's past studies and our own work provides additional rationale for this change. Previous recommendations and evidence to support this change include the following: * GAO recently reported the results of our review of documentation related to proposals for new capabilities that found that most--almost 70 percent--were sponsored by the military services, with little involvement from COCOMs.[Footnote 13] * The DAPA panel recommended that the COCOMs play the lead role in defining needed capabilities and identifying gaps, and that the military services and DOD agencies compete to provide solutions once those gaps are identified. The report further stated that decisions concerning which capabilities to acquire should be made from a joint operational perspective. It also found that within the current requirements identification process, the military services provide the primary input concerning needed capabilities and control the resources for procuring capabilities, thus the requirements frequently are not linked to the capabilities desired by the combatant commanders. * CSIS's Beyond Goldwater-Nichols body of work and DSB's 2005 Summer Study on Transformation found that only COCOMs have operational requirements and operational responsibility to employ the military services as a joint team and therefore should have more influence in establishing requirements. A small majority of the experts we interviewed who commented on this change (23 of 43) believed that a requirements process that provides the COCOMs with more influence over which needs and gaps get filled could improve acquisition outcomes if the COCOMs are provided needed personnel and financial resources to do this. Additionally, some experts stated that this change would promote joint solutions and would be more responsive to warfighters' most urgent needs. The remaining experts, however, told us the potential change would not improve the process, and some believed this was because the existing JCIDS process already considers the combatant commanders' long-term needs and that they lack the required expertise and resources. To implement a more COCOM-focused requirements process, DOD would need to ensure the COCOMs have sufficient additional resources to establish robust analytical capabilities to identify and assess their long-term requirements. Also, while some of the participants in DOD's acquisition system may object to a perceived shift in focus for the COCOMs from short-term to long-term views, this objection may be addressed by implementing a Beyond Goldwater-Nichols Phase II Report recommendation that DOD utilize functional commands, such as the Joint Forces Command, to provide robust analytical support to the geographical COCOMs to identify and assess each command's long-term requirements. According to DOD, it has taken action already that it believes is in line with the Beyond Goldwater-Nichols Phase II Report recommendation. A recent JROC memorandum directed that the lead functional COCOM for the battlespace awareness, command and control, logistics, and net-centric portfolios will coordinate on any capability documents, and the lead functional combatant command Deputy Commander will coordinate on the associated JROC or Joint Capability Board memoranda that validate those capability documents.[Footnote 14] We believe this is a step in the right direction. In its latest Defense Acquisition Transformation Report to Congress, DOD indicated that the Joint Staff is heavily engaging the COCOMs in the requirements process through participation in JROC meetings. DOD noted that in fiscal year 2008 over 75 percent of JROC meetings included one or more COCOM general officer representatives. Furthermore, JROC used the COCOMs' IPLs as the starting point for a series of assessments that identified the most critical military issues and prioritized the capability gaps for each Joint Capability Area (JCA).[Footnote 15] Additionally, twice a year JROC visits and consults with each COCOM to better understand their warfighting needs and to provide feedback on JROC's and the military services' efforts to satisfy those needs. Establishing a Materiel Solutions Process for Addressing Identified Gaps in Critical Warfighting Capabilities, Under Which USD (AT&L) Circulates Among the Military Departments and Appropriate Defense Agencies Requests for Proposals for Technologies and Systems to Address Such Gaps: If other important changes also take place, we believe this potential change could significantly improve acquisition outcomes. If implemented in conjunction with an integrated, portfolio management based investment strategy, it would establish a single point of accountability that has the potential to help ensure interoperability among multiple systems; help reduce stovepiped and sometimes duplicative acquisition programs; and enable DOD to more easily align its portfolio of systems within its budgetary resources. GAO and others have time and again reported on service parochialism in DOD's requirements process and the military services may not be in the best position to decide what the combatant commanders need to conduct their operations, ensure systems are interoperable, and eliminate duplication of systems. Previous recommendations and evidence to support this change include the following: * GAO recently reported that the JCIDS process has not yet met its objective to identify and prioritize warfighting needs from a joint capabilities perspective. Instead, capabilities continue to be driven primarily by the individual military services. GAO has repeatedly reported that DOD could improve acquisition outcomes if, at the Office of the Secretary of Defense (OSD) level, it utilized a portfolio management approach to guide its investment strategy rather than rely on each military service to choose which weapons to procure. * The DAPA panel recommended requiring the military services to compete to satisfy the combatant commanders' requirements. * DSB's 2005 Summer Study on Transformation recommended an organizational structure that clearly identifies the Secretary of Defense with support from USD (AT&L) as the decision maker for weapon system solutions with advice from the Chairman of the Joint Chiefs of Staff and the Joint Staff. Of the 49 experts we interviewed who commented on this potential change, 35 believed that it would not improve the acquisition processes. Some of those experts told us that USD (AT&L) was not properly resourced for this change to be effective. Also, there appeared to be uncertainty among the experts as to how this change would be implemented. Some agreed that the services would be strictly tasked with providing solutions to requirements that have already been vetted, while others seemed to interpret the change as one that would include requirement-setting responsibility. However, the remaining experts felt this potential change would improve the process. Some of those experts cited benefits including reducing service parochialism and increasing the role of OSD staff and the combatant commanders in the requirements process. To be effective, this potential change should be considered in conjunction with the potential change to establish a process in which the combatant commanders are given more responsibility for assessing capabilities and determining requirements. Together, these changes could help ensure focus on filling the capability gaps identified by the combatant commanders and reducing service parochialism. The department has reported efforts that may help in this regard. In its latest Defense Acquisition Transformation Report to Congress, DOD stated that it is addressing some of the perceived problems in this area by adopting concept decision and capability portfolio management processes under the auspices of the USD (ATL)--that are intended to establish better OSD leadership to improve strategic investment decisions.[Footnote 16] DOD also stated that it is seeking to challenge and empower its acquisition workforce to yield the best value for the taxpayer and capability for the warfighter. Revising the Acquisition Process by Establishing Shorter, More Frequent Acquisition Program Milestones: This potential change, if adopted in a manner consistent with the mandate and its supporting conference report, would require shorter, more frequent acquisition milestones to manage requirements, technology, design, and manufacturing risk more closely.[Footnote 17] Based on our own body of work and review of relevant DOD studies, we believe a requirement for four milestones would serve the purposes of this potential change. In our opinion, this potential change could improve program execution if the milestones are specified and aligned with demonstrating knowledge at the appropriate time and if they are tied to certifications to the DAE. GAO has previously reported that the acquisition management framework allows programs to move forward without adequate knowledge at critical junctures. DOD programs regularly skipped a pre-acquisition milestone A, passed milestone B with limited systems engineering knowledge and critical technologies that were immature and not ready for product development. Programs do not undergo another milestone review until production begins at milestone C, sometimes up to 10 years later. These practices left programs with limited knowledge and lengthy time gaps between critical reviews and few opportunities for crucial management oversight and decision making concerning the department's investment and the warfighter's continuing need. DOD issued updated guidance concerning the operation of its DAS in December 2008 to help address these issues. However, it is too soon to determine how successful these changes will be in addressing those issues. GAO has advocated that OSD exert stronger leadership in managing DOD's acquisition programs and has specifically recommended that OSD (1) develop and implement an integrated enterprise-level investment strategy that is based on a joint assessment of warfighting needs and a full set of potential and viable alternative solutions within realistic and affordable budget projections; (2) ensure DOD's acquisition programs are established with sound business cases prior to entering systems development; (3) establish a knowledge-based acquisition process; and (4) improve leadership and accountability. DOD's recently revised acquisition policy addresses some of these needed changes to some degree; for example, the policy now encourages the development of knowledge earlier in the acquisition process. However, consistent with our prior work, we believe an additional milestone review is needed midway through engineering and manufacturing development--during the product development phase--to show the design is stable, and that adequate knowledge exists and demonstrates the weapon system design will perform as expected. We believe this is necessary to reduce risk and optimize the department's investment. According to DOD, the recently revised process achieves this. In addition to the three milestone reviews, the revised instruction requires a formal MDA decision review--the Post Critical Design Review Assessment--during the Engineering and Manufacturing Development Phase. Additionally, if a program did not conduct a Preliminary Design Review prior to the milestone B decision, the MDA will conduct a Post Preliminary Design Review Assessment following the Preliminary Design Review. According to DOD, these MDA assessments are designed to achieve the same substantive benefit as a milestone review, and will result in decisions that will be documented in acquisition decision memoranda. Figure 1 compares DOD's revised acquisition process and GAO's recommended knowledge-based, evolutionary process. Figure 1: Comparison of DOD's Acquisition Process and GAO's Knowledge- Based Acquisition Process: [Refer to PDF for image: illustration} Current DOD process: Pre Systems Acquisition: Materiel solution analysis: Milestone A; Materiel development decision (decision point); Technology Development: Milestone B; Systems Acquisition: Engineering and manufacturing development: Post Prelim design review assessment (Conditional decision point [held if preliminary design review is after Milestone B]); System integration design; Post critical design review assessment (decision point); System capacity and manufacturing process demo; Milestone C; Initial Operational Capacity: Production and deployment: Low rate initial production and initial operational test and evaluation; Full-rate production decision review (decision point); Full-rate production and deployment; Sustainment: Full Operational Capability: Operations and support: Lifecycle sustainment; Disposal. Knowledge-Based Process: Concept and technology development: Materiel development decision (decision point); Milestone A; Prelim Design review (design review); Milestone B, Program initiation; Knowledge point 1; Technology available, time, funding, and other resources match requirements. Product development: Critical design review (design review); Post critical design review assessment (decision point); Milestone B'; Knowledge point 2; Design is stable and meets performance requirements; Milestone C; Knowledge point 3; Production can meet costs, schedule, and quality targets; Production: Initial Operational Capacity: Low rate initial production and initial operational test and evaluation; Full-rate production design review (decision point); Full-rate production and deployment; Full Operational Capacity: Lifecycle sustainment; Disposal. Source: GAO analysis of DOD's acquisition process and GAO's best practices. [End of figure] GAO's work has repeatedly demonstrated the need for DOD to follow a knowledge-based acquisition process and to develop knowledge earlier in the acquisition process, basing the decision to enter the system development phase of the acquisition process with preliminary design knowledge in hand. We agree with the department's new policy that prescribes conducting milestones A, B, and C. We believe a fourth milestone, centered at each program's Post Critical Design Review Assessment midway through development, is also needed. With that in mind, each program would have to pass through the following milestone reviews: * Milestone A--A review to determine the weapon system concept and identify the technologies available to meet a validated requirement for a new weapon system; * Milestone B (equivalent to Knowledge Point 1)--A review to determine through prototyping and disciplined and robust systems engineering practices that a match is made between requirements and available mature technology; funding, and other resources; * Milestone B' (equivalent to Knowledge Point 2)--A review of demonstrated systems engineering, prototyping, and test data that the system's design is stable and meets performance requirements; and: * Milestone C (equivalent to Knowledge Point 3)--A review to demonstrate that manufacturing processes are in control and that the system can be produced within cost, schedule, and quantity targets. Consequently, shorter, more frequent acquisition program milestones could improve DOD's accountability and the acquisition process by providing OSD's decision makers with improved data upon which to make decisions and more frequent opportunities to exercise oversight over these expensive investments. Recommendations from prior studies on DOD acquisition support realignment along these lines and include: * The DAPA panel reported that the chief risk reduction opportunities exist between milestone A and milestone B. Nonetheless, DOD focuses much of its efforts on reaching and passing milestone B, which leads to starting programs with immature technologies and a long cycle of program instability, requirements and budget changes, schedule delays, and repeated rebaselining.[Footnote 18] The DAPA panel recommended DOD realign the milestone B decision to occur at preliminary design review to reduce program risk by increasing knowledge prior to milestone B. * The Institute for Defense Analysis (IDA) in its Acquisition Initiatives--Phase II report recommended USD (AT&L) should reinstitute MDA oversight of the milestone A process so that a sound systems engineering management plan is put in place for each alternative that will be pursued into a formal risk reduction/demonstration/validation phase, leading to a possible milestone B formal initiation of a major new acquisition program.[Footnote 19] The report also suggested there should be a very high bar for waivers of an inadequate systems engineering plan at formal program initiation at milestone B. Regarding this potential change, 28 of the 52 experts we interviewed who commented on this potential change believed that it would not improve the process. Some of the experts who told us the potential change would not improve the acquisition process indicated that the current process and oversight reviews are adequate and noted that more reviews do not always equate to improved outcomes. However, many of the experts we interviewed also told us that this potential change could improve the process depending upon how it is implemented. Some of the experts told us that additional milestones should be accompanied with a more uniform data gathering process, quantifiable metrics, significantly more systems engineering knowledge, and incremental development strategies. We believe DOD's updated DOD Instruction 5000.02 aligns with the intent of this potential change and GAO's best practices model. However, to be more effective, we believe the revised policy should be implemented in conjunction with the potential change to require the milestone decision authority to specify, at the time of milestone B approval, the period that will be required to deliver IOC to the relevant combatant commanders, which is discussed in the following report segment. In combination, these potential changes should result in more frequent milestone reviews with shorter timeframes between milestones--both as a result of the adoption of the milestone B' and the adoption of a more incremental, evolutionary acquisition approach--helping to focus development on more manageable, less aggressive increments of capability and facilitating accountability. Additionally, strong consistent leadership that demands and rewards the proper use of knowledge by programs is needed. In its latest Defense Acquisition Transformation Report to Congress, DOD states its acquisition policy encourages the development of knowledge earlier in the acquisition process and has placed clear emphasis on the need for more systems engineering analysis and prototyping earlier in the process. For example, the policy now requires that programs involving immature technologies undergo a milestone A review and requires a disciplined preliminary design review prior to or soon after the start of a program at milestone B. While the recently updated policy does not create new milestones for design demonstration, it aligns existing milestones and other decision points with obtaining critical knowledge. Requiring the MDA to Specify at the Time of Milestone B Approval the Period of Time That Will Be Required to Deliver an IOC to the Relevant Combatant Commanders: We believe that this potential change, if modified to require a time- certain development period not to exceed 6 years would significantly improve acquisition outcomes.[Footnote 20] However, it can only be successful if the department uses an incremental, evolutionary, and knowledge-based approach to weapon system acquisition programs. New programs would attempt to develop and produce capabilities that are achievable within 6 years, requiring a new program to achieve future incremental capabilities. Limiting development to 5-or 6-year increments (or shorter increments) would force smaller, more manageable commitments in capabilities for each increment and make costs and schedules more predictable. In addition, it could help to stabilize funding by reducing the need to make up for program deviations. In today's acquisition environment, it sometimes takes more than a decade to produce an initial capability. As a result, programs face increased risks that circumstances will change around them--for example, technology changes, requirements changes, and budget pressures--driving up costs and delaying fielding.[Footnote 21] Requiring a milestone decision authority to specify the period required to deliver an IOC, essentially incorporating a time-certain philosophy, could help DOD deliver useful military capabilities in a more timely manner. Previous recommendations and evidence to support this change include the following: * GAO recommended in 2007 that DOD take decisive actions to shorten cycle times in delivering needed combat capabilities to the warfighter to include adopting a time-certain development cycle that can deliver an increment of new capability within 5 to 6 years after the start of system design and development[Footnote 22] We have cited past acquisitions, such as the F-16 and Small Diameter Bomb, as examples of successful incremental development. * The DAPA panel recommended that DOD make program schedules a key performance parameter as well as shift its acquisition approach to incorporate a time-certain development strategy. Inserting time as a key performance parameter would encourage evolutionary development and force more detailed up-front requirements statements. * The Beyond Goldwater-Nichols work found that current initiatives may prove fruitful in the next few years, for example more aggressively managing risk at the early stages of the development process by assessing technology maturity more accurately and giving increased weight to getting capabilities into the field faster. * The Packard Commission reported that an unreasonably long acquisition cycle--10 to 15 years for major weapon systems--is the central problem from which most other acquisition problems stem. It leads to unnecessarily high costs of development--a 10 year acquisition cycle is clearly more expensive than a 5 year cycle. It also leads to obsolete technology in fielded equipment. Because long-term forecasts are uncertain at best, users tend to err on the side of overstating the threat. More than half of the experts we interviewed who commented on this potential change (26 of 49) believed it could improve the acquisition process. Several of those experts told us this change could be effective if it is aligned with evolutionary acquisition and time- certain development principles. Several of the experts who commented that this potential change may not improve acquisition results told us that historically, DOD program managers (PM) do not control requirements and resources or cost estimates are not realistic. To be most effective, we believe this potential change should be implemented in conjunction with the potential change to revise the acquisition process by establishing shorter, more frequent acquisition program milestones. Furthermore, the success of implementing time- certain development may be contingent upon several factors. For example, PMs must be empowered to control their program and defer additional requirements until a future increment; there must be unity among leadership in support of the program; and PM tenure agreements need to bridge the gap between major program milestones to facilitate accountability. In its latest Defense Acquisition Transformation Report to Congress, DOD agreed with the intent, but not the language, of the DAPA panel's findings on time-certain acquisition and stated that it was implementing that intent by selecting time-defined approaches consistent with the users' needs at the concept decision point in the acquisition process. Establishing a New Office to Provide Independent Cost and Performance Estimates: We believe that establishing an independent assessment office could enhance the cost estimates produced if the organization is required to report the results of knowledge-based cost assessments for MDAPs directly to Congress and the Secretary of Defense, and is given the resources needed to develop thorough estimates. The Cost Analysis Improvement Group (CAIG) is the organization currently responsible for conducting independent cost assessments for MDAPs and reports to the department's program analysis and evaluation organization (PA&E).[Footnote 23] However, USD (AT&L)--through oversight of the DAS--is the ultimate customer for CAIG estimates, and ultimately decides whether the CAIG independent cost estimate will be used as the basis to fund the program.[Footnote 24] Establishing an office of independent assessment that reports its results directly to Secretary of Defense and Congress, similar to the office of the Director of Operational Test and Evaluation (DOT&E) would more closely align the CAIG with the Secretary of Defense and USD (AT&L) providing a new level of accountability and the opportunity to be more fully integrated into the acquisition management framework.[Footnote 25] DOD's inability to allocate funding effectively to programs is largely driven by its acceptance of unrealistic cost estimates and a failure to balance needs based on available resources. The uniqueness of each program, the lack of sufficient knowledge about system requirements, technology and design maturity, and the limited analytical tools available are often cited as factors that contribute to optimistic forecasts of development costs. Previous recommendations and evidence to support this change include the following: * GAO has reported: - Development costs for major acquisition programs are often underestimated at program initiation sometimes by 30 to 40 percent. [Footnote 26] Additionally, weapon system programs are initiated without sufficient knowledge about system requirements, technology, and design maturity and that lacking such knowledge, managers rely on assumptions that are consistently too optimistic, exposing programs to unnecessary risks and ultimately cost growth and schedule delays. [Footnote 27] - The policy concerning independent cost estimates does not require that the estimates be relied upon to set budgets, only that they be considered at key acquisition decision points.[Footnote 28] Additionally in 2008, we reported that for many of the 20 major weapons programs we reviewed, the initial approved development baseline funding was lower than the CAIG's cost estimate.[Footnote 29] - The independence of DOT&E and its resulting authority to report directly to Congress is the foundation of its effectiveness. That independence along with its legislative mandate provides sufficient freedom and authority to exercise effective oversight of the operational testing and evaluation of new systems.[Footnote 30] * The DAPA panel recommended adjusting program estimates to reflect high confidence levels--defined as a program with an 80 percent chance of completing development at or below estimated cost. Overall, 46 of the 49 experts we interviewed believe a new office to provide cost and performance estimates is not needed. Many of the experts we interviewed believe the CAIG is independent. Some suggested that the lack of resources within the CAIG, inadequate systems engineering, long acquisition time frames, or adoption of immature technologies contribute to poor cost estimates. Additionally, according to one CAIG official, its cost estimates and independence could be improved if the organization had more personnel resources with the appropriate systems engineering expertise to conduct independent systems engineering performance reviews that could then be used to prepare improved cost estimates and risk determinations. It is critical to note that for this change to be effective and result in more realistic, reliable, and independent cost estimates, we believe DOD needs to improve its cost estimating through a disciplined, knowledge-based approach that reduces risk and uncertainty over time by conducting detailed systems engineering early in weapons systems development processes, shortening acquisition periods, and adopting more mature technologies. Although many experts believed that this change would not improve the acquisition process, we believe that establishing an independent assessment office that provides independent cost estimates and reports its results directly to the Secretary of Defense and to Congress would enhance independence by better informing stakeholders outside of the DOD acquisition community. We see no reason why the CAIG couldn't form the basis of the suggested organization. Finally, that office should have the appropriate resources to conduct such knowledge-based cost assessments. In its latest Defense Acquisition Transformation Report to Congress, DOD stated that USD (AT&L) is focusing on prototyping to reduce technical risk and inform costs estimates. DOD is also working to improve control over developmental timelines through reduced risk, better knowledge, and technical maturity. Requiring Certifications of Program Status to the DAE and Congress Prior to Milestone Approvals: The National Defense Authorization Acts of Fiscal Years 2006, 2007, and 2008 changed certification standards, requiring additional certification requirements for major defense acquisition programs before proceeding to milestones A or B.[Footnote 31] In addition USD (AT&L) issued a memo in May, 2007 implementing Program Manager Agreements to establish an annual contract between the program manager and the acquisition and requirements/resource officials.[Footnote 32] These are binding agreements that detail the progress the program is expected to make during the year and the resources the program will be provided to reach these goals. USD (AT&L) also requires PMs to sign tenure agreements with the SAE stipulating that their MDAP tenure will correspond to the next major milestone review closest to 4 years. In our opinion, when certifications are used in conjunction with Program Manager Agreements, they provide more program visibility and a stronger means for holding the PM and the other key management officials accountable for the decisions they make managing their programs. Furthermore, DOD's strategy to improve acquisition outcomes rests not only with providing support for individual PMs, but more importantly on creating a department-wide environment that promotes stability, accountability, and more informed decision making.[Footnote 33] Many of the experts we interviewed cited the new legislation--which requires milestone A and B certifications--as adequate to meet the needs that this change targets. We agree. Modifying the Role Played by Chiefs of Staff of the Armed Forces in the Requirements, Resource Allocation, and Acquisition Processes: It is difficult to say whether the potential change, as presented, will generate better acquisition outcomes. In our view, this potential change requires further definition and more study. It is unclear which roles of the Chiefs of Staff of the Armed Forces should be changed and how these roles should be modified to improve acquisition outcomes. However, we note that the Beyond Goldwater-Nichols body of work concluded that a consequence of implementing the 1986 Goldwater-Nichols legislation is that the military service Chiefs of Staff no longer play a large role in managing the execution of their acquisition programs. [Footnote 34] According to the DAPA panel, the DSB 2005 Summer Study on Transformation, and the Beyond Goldwater-Nichols body of work, restoring the authority to manage the execution of their acquisition programs to the military service chiefs could in turn restore accountability for acquisition results. A chief argument of each of these studies is that by realigning acquisition programs through a uniformed services chain of command, the services' acquisition organizations would be more streamlined and accountability would be easier to enforce. A large majority of the experts we interviewed who commented on this potential change (35 of 43) told us that modifying the role of the Chiefs of Staff of the Armed Forces to provide them authority to manage the execution of their acquisition programs would not improve the acquisition process. Many of the experts told us the military services already have control over the requirements and resource allocation processes, and should not also have control over the acquisition process. On the other hand, some experts commented that changing the roles of the military Chiefs of Staff might improve the process if done in conjunction with making the combatant commanders responsible for determining requirements, a condition that would take place if the potential change to allow combatant commanders more influence concerning their long-term needs is made. Changes Needed in Acquisition Environment, Culture, and Incentives: Our discussions with acquisition experts indicated that the potential changes may not achieve the desired improvement in acquisition outcomes unless they are accompanied with changes in the overall acquisition environment, its culture, and the incentives provided for success. Specifically, some of the DOD executives we interviewed--both current and former--emphatically stated that these changes will not improve the acquisition system unless the department can significantly improve its prioritization of needs so that there are fewer programs competing for available dollars. Other experts we interviewed questioned the potential of all seven changes to improve acquisition outcomes. A few of the experts believe some process stakeholders involved in setting requirements, providing funding, or executing programs would not agree to the changes because they like the process as it is or do not want their roles in the process to change. One DOD acquisition official told us that DOD's acquisition policies are good; they just are not implemented as intended. Another expert--a former high-ranking DOD executive--categorically stated that the acquisition system is a system in equilibrium with rules and regulations that allow leadership to deviate from the process when they want. Until members of Congress and DOD leadership collectively realize this is the case as well as the fact that they all like the system as it is, nothing will change. Many believe it will take strong, consistent strategic leadership to change the existing culture and environment. Conclusions: We believe that six of the seven potential changes to DOD's acquisition framework could improve acquisitions outcomes. Although many of the experts we interviewed believed the potential changes may not achieve the desired improvement in acquisition outcomes, they provided a wide range of opinions concerning the pros and cons of each potential change. Nonetheless, based on the results of this review and our work and observations over the years of the DOD acquisition systems' imperviousness to change, we do not believe that the changes are likely to achieve the desired results unless they are (1) implemented together along with other more systemic and sweeping changes and (2) accepted by all of the stakeholders and process owners in this complex framework of requirement setting, funding, and program execution. We point specifically to comments made by those current and former DOD executives who know the environment best. Some of these experts stated that (1) the requirements setting process (JCIDS) takes too long; generally produces requirements for capabilities that aren't achievable in the short term, and often doesn't provide optimal results; (2) the funding process historically accepts cost and schedule estimates based on optimistic assumptions and little knowledge about the resources needed to develop the weapon system required; and (3) the acquisition process moves forward with those poor estimates, often adding requirements and increasing cost as the development program progresses. Finally, some of the experts we interviewed pointed to an overall lack of discipline in the process when it comes to making resource- constrained decisions. All of these factors sustain an environment of high risk, low reward, and poor outcomes in the form of higher costs, late deliveries, and less than optimal capabilities fielded to the warfighter. Potential incremental changes to the process are not likely to produce lasting and meaningful improvement until the acquisition environment and all of its process owners accept that risk in product development must be lowered by prioritizing the needs of national security jointly and with discipline; systems engineering knowledge is used to constrain requirements prior to program initiation; funding programs are based on knowledge and independent cost estimates with high levels of confidence; and evolutionary, knowledge-based practices and processes to reduce technology, design, and manufacturing risk are employed. Nonetheless, we believe the potential changes discussed in this report could have significant positive effect, but not if the environment, its incentives, and the culture continue to resist change. Agency Comments and Our Evaluation: We provided a draft of this correspondence to DOD for comment. The department did not provide a written response, but provided technical comments which we have incorporated as appropriate. We are sending copies of this report to the Secretary of Defense; the Under Secretary of Defense for Acquisition, Technology and Logistics; the Chairman of the Joint Chiefs of Staff; and the Secretaries of the Air Force, Army, and Navy. This report will also be available at no charge on GAO's Web site at [hyperlink, http://www.gao.gov]. If you have any questions about this report or need additional information, please contact me at (202) 512-4841 or sullivanm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report were Bruce Thomas, Assistant Director; Claudia Dickey, Analyst-in-Charge; Greg Campbell; Tina Cheng; Rosa Johnson; John Krump; Jean McSween; Robert Miller, and J. Andrew Walker. Signed by: Michael J. Sullivan, Director: Acquisition and Sourcing Management: List of Committees: The Honorable Carl Levin: Chairman: The Honorable John McCain: Ranking Member: Committee on Armed Services: United States Senate: The Honorable Daniel K. Inouye: Chairman: The Honorable Thad Cochran: Ranking Member: Subcommittee on Defense: Committee on Appropriations: United States Senate: The Honorable Ike Skelton: Chairman: The Honorable John M. McHugh: Ranking Member: Committee on Armed Services: House of Representatives: The Honorable John P. Murtha: Chairman: The Honorable C.W. Bill Young: Ranking Member: Subcommittee on Defense: Committee on Appropriations: House of Representatives: [End of section] Enclosure I: Scope and Methodology: The 2008 National Defense Authorization Act (NDAA) directed GAO to report on the feasibility and advisability of seven potential changes to improve the Department of Defense's (DOD) acquisition framework for Major Defense Acquisition Programs (MDAPs).[Footnote 35] See enclosure II for a list of the seven potential changes. Our objective was to determine the feasibility and advisability of the potential modifications. To determine the feasibility and advisability of the seven potential changes included in the mandate, we considered the results of GAO's own body of work and relevant works produced by DOD and others, and the results of our interviews with 53 acquisition experts. We identified and analyzed recommendations from prior GAO and DOD work that address Congress's potential changes, and considered the opinions of the 53 experts we interviewed. Enclosure IV provides a list of DOD acquisition studies and GAO reports used to inform our work. We reviewed relevant law and DOD policies, directives, and guidance as well as the history of the department's DAS and management framework. We conducted semistructured interviews with 53 identified experts. We selected the experts from among currently serving senior DOD acquisition officials, senior military officers involved in setting requirements for the joint staff, the military services, the Combatant Commands (COCOM), individuals who formerly served as senior DOD acquisition officials, participants in previous reviews of the organization and structure of DOD for the acquisition of major weapon systems, academicians, and other subject matter experts. The interview questions were designed to determine the expert's opinions concerning the seven potential changes, as well as determine other systemic problems in DOD's overall acquisition management framework and other potential changes that might result in better acquisition outcomes. We pretested the questions during initial interviews with DOD officials located in Dayton, Ohio and at the Pentagon. Based on the results of these interviews, we refined our questions to ensure they were open ended and allowed the respondents the opportunity to provide not only opinions on the potential changes, but also their reasoning. Additionally, the nature of the questions allowed the experts to provide additional opinions concerning problems with DOD's acquisition processes and potential solutions. To summarize and analyze the information obtained during our interviews, we recorded and verified the experts' responses in a Web-based data collection instrument. The content for the responses were then coded and verified to determine the extent to which the experts believed the seven potential changes, would, would not, or might improve DOD's acquisition processes, and to identify other potential challenges and solutions. We conducted this performance audit from January 2008 to February 2009 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of section] Enclosure II: Potential Modifications to the DOD Acquisition Process from the National Defense Authorization Act for 2008: 1. Revising the acquisition process for major defense acquisition programs by establishing shorter, more frequent acquisition program milestones; 2. Requiring certifications of program status to the defense acquisition executive and Congress prior to milestone approval for major defense acquisition programs; 3. Establishing a new office (to be known as the "Office of Independent Assessment") to provide independent cost estimates and performance estimates for major defense acquisition programs; 4. Requiring the milestone decision authority for a major defense acquisition program to specify, at the time of Milestone B approval, or Key Decision Point B approval, as applicable, the period of time that will be required to deliver an IOC to the relevant combatant commanders; 5. Establishing a material solutions process for addressing identified gaps in critical warfighting capabilities, under which process the Under Secretary of Defense for Acquisition, Technology, and Logistics circulates among the military departments and appropriate Defense Agencies a request for proposals for technologies and systems to address such gaps; 6. Modifying the role played by chiefs of staff of the Armed Forces in the requirements, resource allocation, and acquisition processes; and: 7. Establishing a process in which the commanders of the combatant commands assess, and provide input on, the capabilities needed to successfully accomplish the missions in the operational and contingency plans of their commands over a long-term planning horizon of 15 years or more, taking into account expected changes in threats, the geo- political environment, and doctrine, training and operational concepts. [End of section] Enclosure III: Comparison of DOD’s Current and Prior Acquisition Processes: Figure: Comparison of DOD’s Current and Prior Acquisition Processes: [Refer to PDF for image: illustration] Current DOD Process (as of December 2008): Pre Systems Acquisition: Materiel solution analysis: Materiel development design (decision point); Milestone A; Technology DEvelopment; Prelim design review (design review) (held here or after Milestone B hold system integration design review); Milestone B; Systems Acquisition: Engineering and manufacturing development: Post prelim design review assessment (conditional decision point [held if preliminary design review is after milestone B]); System integration design (design review); Post critical design review assessment (decision point); System capability and manufacturing process demo; Milestone C; Initial Operational Capability: Production and deployment; Low-rate initial production and initial operational test and evaluation; Full-rate production decision review (decision point); Full-rate production and deployment; Sustainment: Full Operational Capability: Lifecycle sustainment; Disposal. Prior DOD process (May 2003): Pre Systems Acquisition: Concept refinement; Concept decision (decision point); Milestone A; Technology development; Milestone B; Systems Acquisition: System development and demonstration; Prelim design review (design review); System integration; Critical design review (design review); Design readiness review (decision point); System demonstration; Milestone C; Initial Operational Capability: Production and deployment; Low-rate initial production and initial operational test and evaluation; Full-rate production decision review (decision point); Full-rate production and deployment; Sustainment: Full Operational Capability: Lifecycle sustainment; Disposal. Source: GAO analysis of DOD's current and prior acquisition process. [End of figure] Major differences between the 2003 and 2008 versions of the DOD instruction 5000.02 are: * The Materiel Development Decision (MDD) replaces the Concept Decision. A MDD is required regardless of where the program intends to enter the acquisition process. * The Materiel Solution Analysis (MSA) Phase replaces the Concept Refinement Phase. The Joint Capabilities Integration and Development System (JCIDS) process no longer includes an Analysis of Materiel and Non-Materiel Alternatives. Non-materiel solutions will be handled via JCIDS; however, all analysis of alternative materiel solutions will be accomplished by the Analysis of Alternatives (AOA) during MSA. The Milestone Decision Authority (MDA) will approve the materiel solution at milestone A. * The Technology Development Phase (TD) now includes a mandatory requirement for competitive prototyping of the system or key-system elements. A Preliminary Design Review (PDR) may be conducted for the candidate designs, and a PDR report will be provided to the MDA with recommended requirements trades. The final Capabilities Development Document should contain trade-offs determined during the TD phase. * The Engineering & Manufacturing Development (EMD) phase replaces System Development and Demonstration. The two major efforts have been renamed. The Program Manager must provide a PDR report, and must provide a Critical Design Review (CDR) report to the MDA. * A Post-CDR Assessment replaces the Design Readiness Review. The MDA will determine if the results of the CDR warrant continuing EMD to milestone C. [End of section] Enclosure IV: Acquisition Studies: Studies on DOD Acquisition: President's Blue Ribbon Commission on Defense Management, A Formula for Action: A Report to the President on Defense Acquisition. The Packard Commission. Washington, D.C.: April 1, 1986. [hyperlink, http://www.ndu.edu/library/pbrc/pbrc.html]. Defense Acquisition Performance Assessment Report. Department of Defense Acquisition Performance Assessment Panel. Washington, D.C.: January 27, 2006. [hyperlink, http://www.acq.osd.mil/dapaproject]. Beyond Goldwater-Nichols: Defense Reform for a New Strategic Era, Phase 1 Report. Center for Strategic and International Studies. Washington, D.C.: March 2004. [hyperlink, http://www.csis.org/isp/bgn]. Beyond Goldwater-Nichols: U.S. Government and Defense Reform for a New Strategic Era, Phase 2 Report. Center for Strategic and International Studies. Washington, D.C.: July 2005. [hyperlink, http://www.csis.org/isp/bgn]. The Future of the National Guard and Reserves: Beyond Goldwater- Nichols: Department of Defense Acquisition and Planning, Programming, Budgeting, and Execution System Reform, Phase 3 Report. Center for Strategic and International Studies. Washington, D.C.: August 2006. [hyperlink, http://www.csis.org/isp/bgn]. Beyond Goldwater-Nichols: Department of Defense Acquisition and Planning, Programming, Budgeting, and Execution System Reform, An Annotated Brief. Center for Strategic and International Studies. Washington, D.C.: August 1, 2006. [hyperlink, http://www.csis.org/isp/bgn]. Transformation: A Progress Assessment Volume I. Defense Science Board, 2005 Summer Study. Washington, D.C.: February 2006. [hyperlink, http://www.acq.osd.mil/dsb/reports.htm#2005]. Defense Acquisition Transformation, Report to Congress. John Warner National Defense Authorization Act Fiscal Year 2007, Section 807. Washington, D.C.: March 2008: [hyperlink, http://www.acq.osd.mil/at/docs/804_defense_acquisition_transformation_re port_mar_2008.pdf]. Report to Congress on Program Manager Empowerment and Accountability. Department of Defense. Washington, D.C.: August 2007. [hyperlink, http://www.acq.osd.mil/at/docs/Program_Manager_Empowerment_and_Accountab ility.pdf]. Joint Defense Capabilities Study, Improving DOD Strategic Planning, Resourcing and Execution to Satisfy Joint Capabilities. Department of Defense. Washington, D.C.: January 2004. [hyperlink, http://www.acq.osd.mil/actd/articles/JointDefenseCapabilitiesStudyFinalR eport_January2004.pdf]. Related GAO Products: Defense Acquisitions: Fundamental Changes Are Needed to Improve Weapon Program Outcomes. [hyperlink, http://www.gao.gov/products/GAO-08-1159T]. Washington, D.C.: September 25, 2008. Defense Acquisitions: A Knowledge-Based Funding Approach Could Improve Major Weapon System Program Outcomes. [hyperlink, http://www.gao.gov/products/GAO-08-619]. Washington, D.C.: July 2, 2008. Defense Acquisitions: Better Weapon Program Outcomes Require Discipline, Accountability, and Fundamental Changes in the Acquisition Environment. [hyperlink, http://www.gao.gov/products/GAO-08-782T]. Washington, D.C.: June 3, 2008. Defense Acquisitions: Results of Annual Assessment of DOD Weapon Programs. [hyperlink, http://www.gao.gov/products/GAO-08-674T]. Washington, D.C.: April 29, 2008. Defense Acquisitions: Assessments of Selected Weapons Programs. [hyperlink, http://www.gao.gov/products/GAO-08-467SP]. Washington, D.C.: March 31, 2008. Defense Management: DOD Needs to Reexamine Its Extensive Reliance on Contractors and continue to Improve Management and Oversight. [hyperlink, http://www.gao.gov/products/GAO-08-572T]. Washington, D.C.: March 11, 2008. Defense Contracting: Additional Conflict of Interest Safeguards Needed for Certain DOD Contractor Employees. [hyperlink, http://www.gao.gov/products/GAO-08-169]. Washington, D.C.: March 7, 2008. Best Practices: Increased Focus Requirements and Oversight Needed to Improve DOD's Acquisition Environment and Weapon System Quality. [hyperlink, http://www.gao.gov/products/GAO-08-294]. Washington, D.C.: February 1, 2008. Defense Acquisitions: Department of Defense Actions on Program Manager Empowerment and Accountability. [hyperlink, http://www.gao.gov/products/GAO-08-62R]. Washington, D.C.: November 9, 2007. Best Practices: An Integrated Portfolio Management Approach to Weapon System Investments Could Improve DOD's Acquisition Outcomes. [hyperlink, http://www.gao.gov/products/GAO-07-388]. Washington, D.C.: March 30, 2007. DOD Acquisition Outcomes: A Case for Change. [hyperlink, http://www.gao.gov/products/GAO-06-257T]. Washington, D.C.: November 15, 2006. Best Practices: Stronger Practices Needed to Improve DOD Technology Transition Processes. [hyperlink, http://www.gao.gov/products/GAO-06-883]. Washington, D.C.: September 14, 2006. Best Practices: Better Support of Weapon System Program Managers Needed to Improve Outcomes. [hyperlink, http://www.gao.gov/products/GAO-06-110]. Washington, D.C.: November 30, 2005. [End of section] Footnotes: [1] Pub. L. No. 110-181, § 813 (2008). [2] In assessing the seven potential changes, we defined feasible changes as those changes that are reasonable to implement, and advisable changes as those that could improve MDAP outcomes. [3] DOD issued updated guidance (DOD Instruction 5000.02) concerning the operation of its defense acquisition system in December 2008. We conducted our interviews with the experts prior to DOD issuing the updated guidance. We believe there is nothing in the updated guidance that would change the opinions provided to us by the experts or the results of our engagement. [4] Pub L. No. 109-163, § 801 (2006), Pub. L. No. 109-364, § 805 (2007), Pub. L. No. 110-181, § 943 (2008). [5] Federal statutes, located primarily in title 10 of the United States Code, along with the Federal Acquisition Regulation, the Defense Federal Acquisition Regulation Supplement, defense regulation and guidance provide the framework for the defense acquisition system. [6] JCIDS falls under the oversight of the Chairman, Joint Chiefs of Staff and is defined in CJCSI 3170.01F. PPBES falls under the oversight of the Deputy Secretary of Defense and is defined in DOD Financial Management Regulation 7045.14-R. DAS falls under the oversight of the USD (AT&L) and is defined in DOD Directive 5000.01 and DOD Instruction 5000.02. [7] According to DOD, the MDA is the designated individual who has overall responsibility for an investment. This person has the authority to approve an investment's progression in the acquisition process and is responsible for reporting cost, schedule, and performance results. For example, the milestone decision authority for a major defense acquisition program, when not delegated to the component level, is the USD (AT&L). [8] The DAE is the USD (AT&L) who has responsibility for supervising the DAS. The DAE takes precedence on all acquisition matters after the Secretary and the Deputy Secretary of Defense. In the military departments, the officials delegated as CAEs (also called Service Acquisition Executives (SAE)) are respectively, the Assistant Secretary of the Army (Acquisition, Logistics, and Technology) (ASA (AL&T)), the Assistant Secretary of the Navy (Research, Development and Acquisition) (ASN (RD&A)), and the Assistant Secretary of the Air Force (Acquisition) (ASAF (A)). The CAEs are responsible for all acquisition functions within their components. [9] DOD, 2006 Quadrennial Defense Review Report (Washington, D.C., February 2006); CSIS, Beyond Goldwater-Nichols: U.S. Government and Defense Reform for a New Strategic Era, vol. 2 (Washington, D.C., July 2005); DSB, Defense Science Board Summer Study on Transformation: A Progress Assessment, vol. 1. (Washington, D.C., February 2006); DOD, Defense Acquisition Performance Assessment Project Report (Washington, D.C., January 2006); DOD, A Quest for Excellence: Final Report to the President by the President's Blue Ribbon Commission on Defense Management (Washington, D.C., June 1986). [10] See also the Goldwater-Nichols Department of Defense Reorganization Act of 1986, Pub. L. No. 99-433. [11] Pub. L. No. 109-163 § 801 (2006); Pub. L. No. 109-364 § 805 (2007), and Pub. L. No. 110-181 § 943 (2008). [12] The IPL defines the COCOM's highest priority capability gaps for the near term, including shortfalls that may adversely affect COCOM missions. These lists are prepared by the COCOMs and submitted to the Joint Chiefs of Staff and then to the Joint Requirements Oversight Council (JROC). After JROC reviews the IPL and decides to endorse the requirements, the approved document is given to the Vice Chairman of the Joint Chiefs and finally the Chairman of the Joint Chiefs for final review. The document is analyzed against established programs within the services to see if needs can be met through existing programs. If a capability is needed that does not have a corresponding program in development, then the Joint Staff will determine the most appropriate organization to further develop the requirement. [13] GAO, Defense Acquisitions: DOD's Requirements Determination Process Has Not Been Effective in Prioritizing Joint Capabilities, [hyperlink, http://www.gao.gov/products/GAO-08-1060]. (Washington, D.C.: Sept. 25, 2008). [14] The Joint Staff, Joint Requirements Oversight Council. JROCM 130- 08, June 20, 2008. Assignment of Joint Potential Designators and Coordination by Combatant Commands on Capabilities Documents. [15] JCA provide a common medium to discuss and describe capabilities across many DOD activities and processes. For example, Battle Space Awareness is the Joint Capability Area that encompasses the ability to understand dispositions and intentions as well as characteristics and conditions of the operational environment that bear on national and military decision making. [16] Concept decision (now the Material Development Decision under the updated 5000.02 instruction) is the first decision point of the DAS. It authorizes entry into the material solution analysis phase. [17] The Conference Report for the National Defense Authorization Act for Fiscal Year 2008 proposes seven milestones. H.R. Conf. Rep. No. 110- 477, at 951 (2007). [18] According to the GAO Cost Assessment Guide, rebaselining should occur rarely, as infrequently as once in the life of a program or project. Schedule rebaselining should occur only when a schedule variance is significant enough to limit its utility as a predictor of future schedule performance. GAO, Cost Assessment Guide: Best Practices for Estimating and Managing Program Costs, Exposure Draft, [hyperlink, http://www.gao.gov/products/GAO-07-1134SP] (Washington, D.C.: July 2007), p. 251. [19] IDA, Acquisition Initiatives Review, Phase II, (Alexandria, VA.: February 2006). [20] According to DOD, a time-certain development period should be tailored to the characteristics of the weapon system under development. For example, 6 years to develop and produce major acquisitions, such as a nuclear submarine or an aircraft carrier, may not be realistic. [21] CJCSI 3170.01F, May 1, 2007, requires the Capability Development Document (CDD) and Capability Production Document (CPD) to include the definition of IOC and the required IOC date. The CDD and CPD are approved and validated by the JROC, and the Acquisition Program Baseline (APB) is approved by the MDA. DODI 5000.02, December 8, 2008, requires the APB to include the definition of system IOC and the IOC date. However, no time limitations are specified in either instruction. [22] GAO, Tactical Aircraft: DOD Needs a Joint and Integrated Investment Strategy, [hyperlink, http://www.gao.gov/products/GAO-07-415] (Washington, D.C.: Apr. 2, 2007). [23] DOD Directive 5000.04-Cost Analysis Improvement Group, dated August 16, 2006, describes the responsibilities of the CAIG. [24] According to DOD, the CAIG prepares a signature report on each of its estimates, and provides this report directly to the USD (AT&L). The CAIG independent cost estimates are prepared in accordance with the existing statutory requirements 10 USC § 2434, and it is presented for consideration at the Defense Acquisition Board milestone review meetings. [25] DOT&E is the principal staff assistant and senior advisor to the Secretary of Defense on operational test and evaluation (OT&E) in DOD. DOT&E's responsibilities include issuing DOD OT&E policy and procedures; reviewing and analyzing the results of OT&E conducted for each MDAP; providing independent assessments to Secretary of Defense, the USD (AT&L), and Congress; making budgetary and financial recommendations to the Secretary of Defense regarding OT&E; and providing oversight to ensure OT&E for MDAPs is adequate to confirm operational effectiveness and suitability of the defense system in combat use. [26] GAO, Defense Acquisitions: A Knowledge-Based Funding Approach Could Improve Major Weapon System Program Outcomes, [hyperlink, http://www.gao.gov/products/GAO-08-619] (Washington, D.C.: July 2, 2008). [27] GAO, Defense Acquisitions: Better Weapon Program Outcomes Require Discipline, Accountability, and Fundamental Changes in the Acquisition Environment, [hyperlink, http://www.gao.gov/products/GAO-08-782T] (Washington, D.C.: June 3, 2008). [28] GAO, Space Acquisitions: DOD Needs to Take More Action to Address Unrealistic Initial Cost Estimates of Space Systems, [hyperlink, http://www.gao.gov/products/GAO-07-96] (Washington, D.C.: November 17, 2006). [29] [hyperlink, http://www.gao.gov/products/GAO-08-619]. [30] GAO, Test and Evaluation: Impact of DOD's Office of the Director of Operational Test and Evaluation, NSIAD-98-22 (Washington, D.C.: October 24, 1997). [31] 10 USC § 2366A and § 2366B have certification criteria for both milestone A and B reviews. [32] USD (AT&L) Memorandum, Program Management Tenure and Accountability, May 25, 2007. [33] GAO, Defense Acquisitions: Department of Defense Actions on Program Manager Empowerment and Accountability, [hyperlink, http://www.gao.gov/products/GAO-08-62R] (Washington, D.C.: March 11, 2008). [34] Goldwater Nichols Department of Defense Reorganization Act of 1986, Pub. L. No. 99-433. [35] Pub. L. No. 110-181. [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s Web site, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: