This is the accessible text file for GAO report number GAO-10-439 entitled 'Best Practices: DOD Can Achieve Better Outcomes by Standardizing the Way Manufacturing Risks Are Managed' which was released on April 22, 2010. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Report to Congressional Requesters: United States Government Accountability Office: GAO: April 2010: Best Practices: DOD Can Achieve Better Outcomes by Standardizing the Way Manufacturing Risks Are Managed: GAO-10-439: GAO Highlights: Highlights of GAO-10-439, a report to congressional requesters. Why GAO Did This Study: Cost growth and schedule delays are prevalent problems in acquiring defense weapon systems. Manufacturing systems has proven difficult, particularly as programs transition to production. In December 2008, the Department of Defense (DOD) issued an updated version of its acquisition policy that reflects earlier consideration of manufacturing risks. A joint defense and industry group developed manufacturing readiness levels (MRL) to support assessments of manufacturing risks. Use of MRLs on all weapon acquisition programs has been proposed. In response to a congressional request, this report assesses the manufacturing problems faced by DOD, how MRLs can address manufacturing problems, how MRLs compare to manufacturing best practices of leading commercial firms, and challenges and barriers to implementing MRLs at DOD. In conducting our work, we contacted DOD, military services, and contractors; held interviews with leading commercial firms; reviewed program documents and policy proposals; and spoke with manufacturing experts. What GAO Found: DOD faces problems in manufacturing weapon systems—systems cost far more and take much longer to build than estimated. Billions of dollars in cost growth occur as programs transition from development to production, and unit-cost increases are common after production begins. Several factors contribute to these problems including inattention to manufacturing during planning and design, poor supplier management, and a deficit in manufacturing knowledge among the acquisition workforce. Essentially, programs did not identify and resolve manufacturing risks early in development, but carried risks into production where they emerged as significant problems. MRLs have been proposed as new criteria for improving the way DOD identifies and manages manufacturing risks and readiness. Introduced to the defense community in 2005, MRLs were developed from an extensive body of manufacturing knowledge that includes defense, industry, and academic sources. An analysis of DOD’s technical reviews that assesses how programs are progressing show that MRLs address many gaps in core manufacturing-related areas, particularly during the early acquisition phases. Several Army and Air Force centers that piloted MRLs report these metrics contributed to substantial cost benefits on a variety of technologies and major defense acquisition programs. To develop and manufacture products, the commercial firms we visited use a disciplined, gated process that emphasizes manufacturing criteria early in development. The practices they employ focus on gathering sufficient knowledge about the producibility of their products to lower risks, and include stringent manufacturing readiness criteria to measure whether the product is sufficiently mature to move forward in development. These criteria are similar to DOD’s proposed MRLs in that commercial firms: * assess producibility at each gate using clearly defined manufacturing criteria to gain knowledge about manufacturing early, * demonstrate manufacturing processes in a production-relevant environment, and, * emphasize relationships with critical suppliers. However, a key difference is that commercial firms, prior to starting production, require their manufacturing processes to be in control— that is, critical processes are repeatable, sustainable, and consistently producing parts within the quality standards. DOD’s proposed MRL criteria do not require that processes be in control until later. Acceptance of MRLs has grown among some industry and DOD components. Yet, DOD has been slow to adopt a policy that would require MRLs across DOD. Concerns raised by the military services have centered on when and how the MRL assessments would be used. While a joint DOD and industry group has sought to address concerns and disseminate information on benefits, a consensus has not been reached. If adopted, DOD will need to address gaps in workforce knowledge, given the decrease in the number of staff in the production and manufacturing career fields. What GAO Recommends: GAO recommends that the Secretary of Defense require the use of MRLs across DOD programs, strengthen the MRL criteria (process control) for production start, assess the need for tools, and assess the manufacturing workforce to address knowledge gaps. DOD partially concurred with the first recommendation, and concurred with the other three. View the full [hyperlink, http://www.gao.gov/products/GAO-10-439]. or key components. For more information, contact Michael Sullivan at (202) 512-4841 or sullivanm@gao.gov. [End of section] Contents: Letter: Background: Manufacturing Problems Are Attributed to Several Factors during the Planning and Design Phases of Selected DOD Weapons Programs: MRLs Have Been Proposed to Improve the Way DOD Identifies and Manages Manufacturing Risk and Readiness: DOD's Proposed MRLs Embody Many Best Practices of Leading Commercial Firms: MRLs Are Hampered by Lack of an Agencywide Policy and Manufacturing Workforce Concerns: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Scope and Methodology: Appendix II: Manufacturing Readiness Level (MRL) Definitions: Appendix III: Manufacturing Readiness Level (MRL) Threads and Subthreads (Risk Areas): Appendix IV: Comments from the Department of Defense: Appendix V: GAO Contact and Staff Acknowledgments: Related GAO Products: Tables: Table 1: Basic Manufacturing Readiness Level Definitions: Table 2: Basic Manufacturing Threads (Risk Areas) for MRL 1-10: Table 3: Many Manufacturing Criteria Used by Leading Commercial Firms Are Similar to DOD's MRLs: Table 4: Percent of Manufacturing Workforce Decrease from 2001 to 2007: Figures: Figure 1: Distribution of Average Procurement Unit-Cost Growth after Production Decision for Major Defense Acquisition Programs: Figure 2: Contributing Factors to Manufacturing Problems for Four DOD Case-Study Programs: Figure 3: Relationship of MRLs to System Milestones and Technology Readiness Levels (TRL): Figure 4: GE CT Scanner Using Advanced Scintillator Material: Figure 5: Honeywell Uses Three Producibility Models and MRL Workshop: Figure 6: GE Aviation's Turbine Airfoils Lean Lab Proves Out Production Processes: Figure 7: Leading Commercial Firms Use Statistical Controls to Ensure Quality Products: Abbreviations: CT: Computed Tomography: DOD: Department of Defense: MRL: Manufacturing Readiness Level: [End of section] United States Government Accountability Office: Washington, DC 20548: April 22, 2010: The Honorable Bill Nelson: Chairman: Subcommittee on Emerging Threats and Capabilities: Committee on Armed Services: United States Senate: The Honorable Jack Reed: United States Senate: The Department of Defense (DOD) has a well-documented history of taking much longer and spending much more than originally planned to develop and acquire its weapons systems. In particular, as systems transition from development to production, programs experience significant manufacturing problems. While DOD has made some progress over the last two decades in addressing the problem--including policy changes and advocating the use of best practices for product development--GAO's recent weapon system reviews show that manufacturing problems, among others, continue to hinder acquisition cost, schedule, and performance outcomes. It is essential to find better ways of doing business and, in particular, to make sure systems are manufactured on time and cost- effectively. To this end, leading commercial companies have achieved more predictable outcomes from their manufacturing efforts because they understand producibility--the relative ease of producing designs of an item, product, or system economically with available production techniques--and identify manufacturing risks early and manage them effectively throughout a product's development life cycle. On December 8, 2008, DOD issued a revised version of its policy instruction on operation of the defense acquisition system that, among other things, recognizes the need to consider manufacturing risks earlier in the acquisition life cycle and assesses risks prior to key decision points. In response to the need for the department to better design and produce more affordable weapon systems, and to give decision makers and managers better visibility into their program risks, a joint defense and industry working group was established in 2004 to develop manufacturing readiness levels (MRL), a measurement scale designed to improve the management and communication of manufacturing risk and readiness throughout acquisitions. Similar to technology metrics that measure the readiness of a technology, MRLs are new manufacturing criteria that measures the manufacturing maturity or readiness of a given technology, manufacturing process, system, or element of a weapon system at various phases of the acquisition life cycle. In response to a request from the Senate Subcommittee on Emerging Threats and Capabilities and Senator Reed, we reviewed DOD's efforts to adopt MRLs. This report addresses (1) the manufacturing problems experienced by selected DOD programs, (2) how MRLs can address DOD's manufacturing problems, (3) how proposed MRLs compare to manufacturing best practices of leading commercial companies, and (4) the challenges and barriers to implementing MRLs at DOD. To meet these objectives, we compared the manufacturing practices of DOD and its large prime contractors with those of leading commercial companies. We performed an aggregate analysis of DOD programs from our annual weapons assessment.[Footnote 1] We also evaluated four major defense weapon systems in production with known cost and schedule problems to gain in-depth insights as to the nature and causes of problems. We also evaluated two defense systems known to be producing systems within cost and schedule goals and compared their practices to those employed by commercial firms. We examined program documentation and policy proposals, and held discussions with manufacturing and systems-engineering officials from DOD program offices, prime contractors, and the Defense Contract Management Agency. We also reviewed lessons learned from DOD programs that pilot-tested MRLs. We met with officials from the Office of the Secretary of Defense, Air Force, Army, and Navy, Missile Defense Agency, Joint Defense Manufacturing Technology Panel, Defense Acquisition University, National Center for Advanced Technologies, and National Defense Industrial Association to discuss manufacturing topics and MRLs. On manufacturing workforce issues, we interviewed officials responsible for planning activities within each of the military services and the Defense Management Contract Agency. We compared manufacturing and production considerations in the prior version of DOD's policy instruction on operation of the defense acquisition system[Footnote 2] to those in the current version of the policy instruction.[Footnote 3] To identify manufacturing best practices of leading commercial companies, we interviewed and obtained documentation from manufacturing, quality, and supplier personnel at five companies, and reported on four companies: GE Aviation, an aerospace company; GE Healthcare, a producer of healthcare products and services; Honeywell Aerospace, a provider of aircraft integrated avionics, engines, systems, and services; Siemens Mobility, a producer of light rail cars. We selected companies that manufacture complex products and have won awards for quality manufacturing. Appendix I includes additional details about our scope and methodology. We conducted this performance audit from January 2009 to February 2010 in accordance with generally accepted government auditing standards. These standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We are making recommendations to the Secretary of Defense to require an assessment of the manufacturing readiness across DOD programs using MRL criteria, examine strengthening the MRL criteria related to process capability and control, assess analytical model needs and tools to support MRL assessments, and assess the manufacturing workforce knowledge and skills base and develop a plan to address DOD's current and future workforce knowledge gaps. In commenting on a draft of this report, DOD partially concurred with the first recommendation, and concurred with the other recommendations. Background: In recognition of the lack of manufacturing knowledge at key decision points and the need to develop more affordable weapon systems, DOD made recent changes to its policy. In 2008, the department made constructive changes to its policy instruction on operation of the defense acquisition system. It also developed MRLs as a measure that could strengthen the way the department manages and develops manufacturing-intensive systems. In 2004, the Joint Defense Manufacturing Technology Panel[Footnote 4] sponsored a joint defense and industry working group to design and develop MRLs for programs across DOD. In May 2005, MRLs were first introduced to the defense community in DOD's Technology Readiness Assessment Deskbook for science and technology and acquisition managers to consider. As new manufacturing readiness criteria, MRLs are a measurement scale designed to provide a common metric and vocabulary for assessing manufacturing maturity and risk. MRL assessments identify the risks and manufacturing readiness of a particular technology, manufacturing process, weapon system, subsystem, or element of a legacy program at key milestones throughout the acquisition life cycle. There are 10 basic MRLs designed to be roughly congruent with comparable levels of technology readiness levels for ease of use and understanding. Table 1 shows the MRLs and basic definitions (see appendix II for the detailed MRL definitions). Table 1: Basic Manufacturing Readiness Level Definitions: MRL: 1; Description: Basic manufacturing implications identified. MRL: 2; Description: Manufacturing concepts identified. MRL: 3; Description: Manufacturing proof of concept developed. MRL: 4; Description: Capability to produce the technology in a laboratory environment. MRL: 5; Description: Capability to produce prototype components in a production-relevant environment. MRL: 6; Description: Capability to produce a prototype system or subsystem in a production-relevant environment. MRL: 7; Description: Capability to produce systems, subsystems, or components in a production-representative environment. MRL: 8; Description: Pilot line capability demonstrated; ready to begin low- rate initial production. MRL: 9; Description: Low-rate production demonstrated; capability in place to begin full-rate production. MRL: 10; Description: Full-rate production demonstrated, and lean production practices in place. Source: Joint Defense Manufacturing Technology Panel. [End of table] The working group also developed a set of elements called "threads" to provide acquisition managers and those conducting assessments an understanding of the manufacturing risk areas (see table 2). For these threads, desired progress is defined for each MRL, to provide an understanding of risks as readiness levels increase from one MRL to the next. Conceptually, these threads are manufacturing elements that are essential to programs as they plan, prepare for, and manage the activities necessary to develop a product. For example, the materials thread requires an assessment of potential supplier capability by MRL 3 and an assessment of critical first-tier suppliers by MRL 7. Likewise, the manufacturing personnel thread calls for identifying new manufacturing skills by MRL 3 and identifying manufacturing workforce requirements for the pilot line by MRL 7. Table 2: Basic Manufacturing Threads (Risk Areas) for MRL 1-10: Thread (risk areas): Technology and the Industrial Base; Description: Requires an analysis of the capability of the national technology and industrial base to support the design, development, production, operation, uninterrupted maintenance support of the system and eventual disposal (environmental impacts). Thread (risk areas): Design; Description: Requires an understanding of the maturity and stability of the evolving system design and any related impact on manufacturing readiness. Thread (risk areas): Cost and Funding; Description: Requires an analysis of the adequacy of funding to achieve target manufacturing maturity levels. Examines the risk associated with reaching manufacturing cost targets. Thread (risk areas): Materials; Description: Requires an analysis of the risks associated with materials (including basic/raw materials, components, semi-finished parts, and subassemblies). Thread (risk areas): Process Capability and Control; Description: Requires an analysis of the risks that the manufacturing processes are able to reflect the design intent (repeatability and affordability) of key characteristics. Thread (risk areas): Quality Management; Description: Requires an analysis of the risks and management efforts to control quality, and foster continuous improvement. Thread (risk areas): Manufacturing Personnel; Description: Requires an assessment of the required skills, availability, and required number of personnel to support the manufacturing effort. Thread (risk areas): Facilities; Description: Requires an analysis of the capabilities and capacity of key manufacturing facilities (prime, subcontractor, supplier, vendor, and maintenance/repair). Thread (risk areas): Manufacturing Management; Description: Requires an analysis of the orchestration of all elements needed to translate the design into an integrated and fielded system (meeting program goals for affordability and availability). Source: Joint Defense Manufacturing Technology Panel. [End of table] As shown, each basic thread (risk area) has a description and general requirements for assessing risks for each thread. The working group further decomposed these MRL threads into subthreads to provide users a detailed understanding of the various kinds of manufacturing risks. See appendix III for a detailed breakdown of these threads (risk areas) for each MRL. DOD's Long-standing History of Manufacturing Problems: GAO has conducted an extensive body of work that highlights many of the manufacturing-related problems that both DOD and its prime contractors have faced. In many respects, DOD has recognized the nature of these problems throughout the years and has taken a number of proactive steps to address them. GAO's work has drawn on lessons learned and best practices to recommend ways for DOD to improve the way it develops and manufactures its weapon systems. Examples from our reports include the following: * In 1996, GAO reported the practices that world-class commercial organizations had adopted to more efficiently produce quality products, to improve DOD's quality assurance program.[Footnote 5] DOD was spending $1.5 billion extra per year on military-unique quality assurance requirements for major acquisitions and billions more on cost and schedule overruns to correct problems. GAO concluded that repeated unstable designs, poor process controls, and poor transition to production caused the manufacturing quality problems. While DOD had taken some actions, its culture was cited as the biggest reason for slow adoption and unimplemented recommendations. * In 1998, GAO reported on best commercial practices to offer ways to improve the process DOD uses to manage suppliers engaged in developing and producing major weapon systems.[Footnote 6] In assessing defense contractors and two case studies of munitions programs, the report concluded that suppliers were critical in the amount of technological innovation they contribute to the final product. * In 2002, GAO reported on how best practices could offer improvements to the way DOD develops new weapon systems, primarily the design and manufacturing aspects of the acquisition process.[Footnote 7] DOD's record showed a history of taking longer and spending more than planned to develop and acquire weapon systems, which reduced its buying power. The report identified and recommended best practices for capturing and using design and manufacturing knowledge early and new development processes that included high-level decision points and knowledge-based exit criteria before key decisions on production are made. Essentially, one of the high-level decision points has become what GAO commonly refers to as Knowledge Point 3--the point when a program has demonstrated the manufacturing processes are mature. The report also recommended a best practice that includes a standard called the Process Capability Index (Cpk), a process performance measurement that quantifies how closely a product is running to its specification limits. The index indicates how well the processes statistical performance meets its control limit requirement. * In 2008, GAO reported on how DOD and its defense contractors can improve the quality of major weapon systems.[Footnote 8] We reported that if DOD continued to employ the same acquisition practices as it has in the past, the cost of designing and developing its systems could continue to exceed estimates by billions of dollars. Quality problems were identified as the cause for cost overruns, schedule delays, and reduced weapon-system availability. Like DOD prime contractors, leading commercial firms rely on many practices related to systems engineering, manufacturing, and supplier quality, but they were more disciplined, and had institutionalized processes to ensure quality. * Since 2003, GAO has issued a series of annual assessment reports on selected weapons programs, increasing from 77 to 96 programs reviewed. [Footnote 9] At $296 billion, the cumulative cost growth for DOD programs reported in 2009 was found to be higher than it had been five years earlier, and the overall performance of weapon system programs was still poor. Although the cost growth and the 22-month average delay in delivering initial capabilities was not attributed to manufacturing alone, the lack of production maturity was cited as one of three key knowledge areas contributing to the department's cost growth, schedule delay, and performance problems. Revised Policy Incorporates Manufacturing Best Practices: DOD's December 2008 revision to its policy instruction on operation of the defense acquisition system[Footnote 10] incorporates a number of the best practices we identified in our previous work. The instruction covers the entire life cycle and considers manufacturing risks earlier in the acquisition life-cycle framework. In a November 2003 report on DOD's May 2003 revision to its policy, we reported that much of the revised policy agrees with GAO's extensive body of work and that of successful commercial firms. While we assessed DOD's revised policy as providing a good framework for capturing knowledge about critical technologies, product design and manufacturing processes, we reported in 2006 that acquisition officials were not effectively implementing the acquisition policy's knowledge-based process.[Footnote 11] We reported that the effective implementation of policy was limited by the absence of effective controls that require compliance and specific criteria for clearly demonstrating that acceptable levels of knowledge about technology, design, and manufacturing have been attained at critical junctures before making further investments in a program. We concluded that without specific criteria--or standards against which a judgment or decision is quantifiably based--decision makers are permitted to make decisions on the basis of subjective judgment. The December 2008 revised policy instruction establishes target maturity criteria for measuring risks associated with manufacturing processes at milestone decision points.[Footnote 12] During the material solutions phase, prior to milestone A, the 2008 policy instruction requires the analysis of alternatives to assess "manufacturing feasibility." During the technology development phase, prior to milestone B, the instruction states the following: * Prototype systems or appropriate component-level prototyping shall be employed to "evaluate manufacturing processes." * A successful preliminary design review will "identify remaining design, integration, and manufacturing risks." * A program may exit the technology development phase when "the technology and manufacturing processes for that program or increment have been assessed and demonstrated in a relevant environment" and "manufacturing risks have been identified." After milestone B, one of the purposes of the engineering and manufacturing development phase is to "develop an affordable and executable manufacturing process." The instruction says that: "the maturity of critical manufacturing processes" is to be described in a post-critical design review assessment; system capability and manufacturing process demonstration shall show "that system production can be supported by demonstrated manufacturing processes;" and the system capability and manufacturing process demonstration effort shall end, among other things, when "manufacturing processes have been effectively demonstrated in a pilot line environment, prior to milestone C." Finally, at milestone C, the instruction establishes two entrance criteria for the production and deployment phase, which include "no significant manufacturing risks" and "manufacturing processes [are] under control (if Milestone C is full-rate production)." Low-rate initial production follows in order to ensure an "adequate and efficient manufacturing capability." In order to receive full-rate production approval, the following must be shown: 1. "demonstrated control of the manufacturing process," 2. "the collection of statistical process control data," and: 3. "demonstrated control and capability of other critical processes." Even with the updated policy instruction in place that includes guidance for most knowledge-based practices, inconsistent implementation has hindered DOD's past efforts to reform its acquisition practices. For example, we reported in 2006 that DOD was not effectively implementing the knowledge-based approach process and evolutionary approach emphasized in its policy.[Footnote 13] While the policy outlined specific knowledge-based process of concept refinement and technology development to help ensure a sound business case is developed before committing to a new development program, we found that almost 80 percent of the programs we reviewed were permitted to bypass this process. Manufacturing Problems Are Attributed to Several Factors during the Planning and Design Phases of Selected DOD Weapons Programs: Defense acquisition programs continue to have problems manufacturing weapon systems. As a result, systems cost far more and take far longer to produce than estimated. Many programs authorized to enter production experienced billions of dollars in cost growth after the authorization--nearly two-thirds of those programs reported increases in average procurement unit costs. Several factors contribute to these issues during the planning and design phases. These include the inattention to manufacturing during planning and design, poor supplier management, and lack of a knowledgeable manufacturing workforce. Essentially, some of these programs moved into production without considering manufacturing risks earlier in development. This hindered managers from later managing those risks until they became problematic, and also led to subsequent problems with supplier management, such as prime contractors conducting little oversight of suppliers. Some programs also had an inadequate workforce--in terms of insufficient knowledge and numbers--to effectively manage and oversee defense manufacturing efforts. Manufacturing Contributed to Growth in Cost and Delays in Schedule: Defense acquisition programs continue to be troubled by unstable requirements, immature technology, and a lack of manufacturing knowledge early in design, resulting in more costly products that take longer to produce. Our 2009 annual assessment shows that total research and development costs were 42 percent higher than originally estimated. These higher costs reflect in part the learning that takes place as manufacturing processes are established and used to produce the first prototypes. Even programs that have been authorized to begin production have experienced substantial cost growth after the production decision. Production performance can be measured by examining the cost growth as expressed in changes to average procurement unit cost. This represents the value DOD gets for the procurement dollars invested in a certain program and shows the net effect of procurement cost growth and quantity changes. Figure 1 shows the levels of average procurement unit-cost growth for selected major defense acquisition programs. [Footnote 14] Figure 1: Distribution of Average Procurement Unit-Cost Growth after a Production Decision for Major Defense Acquisition Programs: [Refer to PDF for image: pie-chart] Programs with less than 5% growth: 39%; Programs with 5% to 10% growth: 29%; Programs with 11% to 15% growth: 16%; Programs with more than 15% growth: 10%; Source: GAO analysis of DOD data. Note: Data include all major defense acquisition programs that entered production in fiscal year 2000 or later. [End of figure] As indicated in figure 1, nearly two-thirds of programs that entered production after 2000 reported more than a 5 percent increase in average unit cost growth, while 32 percent of programs reported average unit cost growth that ranged from 11 percent to more than 15 percent. One program reported a 25 percent increase in average procurement unit cost. Further, 42 percent of those programs experienced production cost increases when procured quantities decreased or remained the same. For example, the Black Hawk helicopter's 2007 production estimate had no increase in quantities since 2005, yet its production cost increased $2.3 billion, and average procurement unit cost rose by 13 percent. The Joint Air-to- Surface Standoff Missile had an 8 percent quantity decrease since the 2004 production decision; but the production costs increased by $561 million and average procurement unit cost increased by 25 percent. As for schedule growth, DOD has continued to experience delays in delivering new or modified weapon systems to the warfighter. Over 50 percent of current programs in production have encountered some form of delay after the production decision, when manufacturing processes should be in control. Consequently, warfighters often must operate costly legacy systems longer than expected, find alternatives to fill capability gaps, or go without the capability altogether. The four DOD weapon systems we selected for in-depth review with known cost, schedule, and performance problems reported several key factors that contributed to manufacturing problems. These include the inattention to manufacturing during planning and design, poor planning for supplier management, and lack of a knowledgeable manufacturing workforce. Capturing critical manufacturing knowledge during the planning and design phases before entering production helps to ensure that a weapon system will work as intended and can be manufactured efficiently to meet cost, schedule, and quality targets. The programs in our review often lacked manufacturing knowledge at key decision points, which led to cost growth and schedule delays. For example, the Joint Air-to-Surface Standoff Missile program--an autonomous, air-to- ground missile designed to destroy high-value targets--experienced a critical unit-cost breach due to missile reliability problems not being addressed early in the design phase.[Footnote 15] Also, the Electromagnetic Aircraft Launch System--a new catapult technology being developed for the Navy's newest class of aircraft carriers--had experienced problems manufacturing compatible materials, which resulted in cost growth and schedule delays and was the focus of recent congressional interest. Figure 2 summarizes contributing factors for manufacturing problems experienced by the four DOD weapon systems. Figure 2: Contributing Factors to Manufacturing Problems for Four DOD Case-Study Programs: [Refer to PDF for image: illustrated table, containing photograph for each program] DOD Program: Exoatmospheric Kill Vehicle; Source of manufacturing problems: Inattention to manufacturing during planning and design: [Check]; Source of manufacturing problems: Poor supplier management planning: [Check]; Source of manufacturing problems: Lack of workforce knowledge and skills: [Check]; General problems: * Immature technologies caused development problems; * Cost and schedule problems increased total cost of the interceptor. DOD Program: Electromagnetic Aircraft Launch System; Source of manufacturing problems: Inattention to manufacturing during planning and design: [Check]; Source of manufacturing problems: Poor supplier management planning: [Empty]; Source of manufacturing problems: Lack of workforce knowledge and skills: [Check]; General problems: * Development resulted in cost growth and schedule delays. DOD Program: H-1 Helicopter Upgrade Program; Source of manufacturing problems: Inattention to manufacturing during planning and design: [Check]; Source of manufacturing problems: Poor supplier management planning: [Check]; Source of manufacturing problems: Lack of workforce knowledge and skills: [Check]; General problems: * Decision to remanufacture increased costs (utility helicopter configuration); * Systems engineering and configuration management challenges. DOD Program: Joint Air-to-Surface Standoff Missile; Source of manufacturing problems: Inattention to manufacturing during planning and design: [Check]; Source of manufacturing problems: Poor supplier management planning: [Check]; Source of manufacturing problems: Lack of workforce knowledge and skills: [Check]; General problems: * Increased costs and schedule delays; * Reliability problems. Source: GAO analysis of Army, Air Force, Navy, and Missile Defense Agency data. Images: Missile Defense Agency and Boeing public Web site per GMG program office (top); CVN-21 Program Office 050708-D-8455H-001 Washington, D.C. (July 8, 2005) U.S. Navy graphic (released) (second from top); USMC Light/Attack Helicopter (H-1) Program Office, PMA276. (third from top); Integrated Test 2 accomplished December 2006 (bottom). [End of figure] As indicated, most of the programs had more than one major problem related to manufacturing. These issues illustrate the major problems we discussed with defense and contractor officials, but do not encompass all the manufacturing problems experienced by the programs. For example, a recent Air Force study reports that manufacturing and quality assurance requirements are not included in the contracts to develop weapon systems, which could affect the contractor's approach to manufacturing. Officials from the Defense Contract Management Agency--a DOD component that works directly with defense suppliers to ensure that supplies and services are delivered on time, at projected cost, and meet performance requirements--also reported similar contract issues that could affect contractor performance on manufacturing. Manufacturing Was Overlooked during Early Development: Each of the four programs we examined did not give manufacturing strong consideration during the early planning and design phases. Programs were moved into production largely without considering manufacturing risks earlier in the acquisition process, as demonstrated by the experiences of the Exoatmospheric Kill Vehicle and the H-1 helicopter upgrade program. The Exoatmospheric Kill Vehicle was designed to intercept and destroy high-speed ballistic missile warheads in mid-flight, while the H-1 upgrade program converts the attack helicopter and the utility helicopter to the AH-1Z and UH-1Y configurations, respectively. The Exoatmospheric Kill Vehicle program was put on an accelerated development schedule in response to a directive to develop and deploy, at the earliest possible date, ballistic missile defense drawing on the best technologies available. According to the contractor, it bypassed some of its normal development-review processes to accelerate delivery of the vehicle, which also resulted in a high acceptance of manufacturing risks without sufficient identification and management of risk-mitigation plans. For example, the program went into production without completing qualification testing. In addition, the contractor continued to incorporate design changes while supplier production was ongoing, resulting in rework and disruption to the production line. Early lots of kill vehicles were built manually by engineers in the absence of automated production processes, which caused dissimilarities among vehicles in the fleet and will make refurbishments difficult.[Footnote 16] For several reasons, the H-1 helicopter upgrade program did not include manufacturing in the early phases of planning and also proceeded to production before its design was mature, according to the contractor. First, the program underestimated the complexity of updating and remanufacturing the aircraft without historical drawings. The emphasis was placed on minimizing development costs and resources were not available to assess manufacturing challenges early in the redesign process. Furthermore, the program started low-rate production before completing operational evaluation testing. As a result, the problems uncovered during testing had to be corrected on aircraft that were on the assembly line. Also, constant change orders and factory bottlenecks, among other problems, affected program costs and schedules. The schedule pressure allowed little opportunity to remedy the manufacturing problems, resulting in more complicated and expensive fixes. Ultimately the schedule slowed and the costs increased to the point that the program abandoned the remanufacturing upgrade and, instead, opted to purchase newly manufactured aircraft cabins for the UH-1Y configuration. Poor Planning Led to Supplier Problems: Inattention to manufacturing during planning and design led to subsequent problems with supplier management in two major defense acquisition programs we reviewed. Specifically, the prime contractors did not give adequate attention to managing their suppliers. For example, program officials for the Joint Air-to-Surface Standoff Missile told us that the responsibility for manufacturing processes and discipline shifted in the 1990s from the government to the defense contractors. The government started to rely on the prime contractor to ensure quality and reliability, particularly with subtier suppliers. In this case, the program office told us that the prime contractor for the missile program relied on the subtier suppliers to self-report their capabilities and did not engage in effective oversight of their work, which led to defective parts. The program office recently recruited experts in manufacturing to help the prime contractor address their supplier problems more effectively. In the Exoatmospheric Kill Vehicle program, supplier quality was inconsistent, resulting in unnecessary rework and uncovering problems late in production. For many suppliers, the kill vehicle program represents a small portion of their business, so the emphasis on quality was often lacking. Further, the program was initially procured as a capability based program, rather than requirements based program. Thus, the prime contractor did not impose requirements on the subcontractors to comply with stringent requirements for space programs. In turn, the subcontractors did not implement sufficient requirements which led to recurring quality issues. Lack of Manufacturing Knowledge Contributed to Problems: Some DOD programs and prime contractors had an inadequate defense manufacturing workforce--both in terms of numbers and experience--to effectively manage and oversee manufacturing efforts, which resulted in schedule delays or cost inefficiencies. The manufacturing workforce includes occupations such as specialists in quality assurance, business, manufacturing engineering, industrial engineering, and production control. In many cases, the programs lacked manufacturing expertise early in development, which hindered the program's ability to later manage manufacturing risks. For example, the contractor for the Electromagnetic Air Launch System did not have sufficient systems- engineering personnel involved in the design to help it transition from development to production. As a result, the program encountered schedule delays and cost increases. DOD conducted a program assessment review, which led the program office and contractor to increase systems engineering staff. For the Exoatmospheric Kill Vehicle program, the contractor's workforce and manufacturing processes could not readily undertake the rigors of production for a space-based capability, part of which must be manufactured in a clean room environment, and all of which commands rigorous processes and procedures due to highly technical designs. The contractor's hourly assembly personnel were trained to build tactical missiles on a high-rate production line and were not sufficiently trained in the quality-control standards required by clean-room manufacturing, such as carefully controlling foreign-object debris, specially maintaining the clean room, and using a partner in certain high-level tasks to ensure all steps are properly followed. These standards were not institutionalized, and the contractor eventually had to modify its facilities and production standards to correct the manufacturing problems. The facility had to be retooled and reconfigured late in development. The contractor also experienced high turnover in its workforce due to the increasing demands associated with working in a clean-room environment and working long hours. MRLs Have Been Proposed to Improve the Way DOD Identifies and Manages Manufacturing Risk and Readiness: The Joint Defense Manufacturing Technology Panel working group has proposed MRLs as new manufacturing readiness criteria that could improve weapon system outcomes by standardizing the way programs identify and manage manufacturing risks associated with developing and fielding advanced weapon systems. MRLs were first introduced to the defense community in DOD's 2005 Technology Readiness Assessment Deskbook as an important activity for science and technology and acquisition managers to consider. An analysis by the working group shows that MRLs address many of the manufacturing issues not covered by DOD's technical reviews, particularly reviews conducted in the early phases of acquisition. In their development, comprehensive efforts were undertaken to design and develop MRLs from DOD as well as industry resources. For example, the working group formulated MRLs from a manufacturing knowledge base of defense, industry, and academia to address two key areas of risk--immature product technologies and immature manufacturing capability. The working group also designed MRLs as a structured and disciplined approach for the way manufacturing risk and readiness is expected to be identified and assessed. The working group also developed a set of tools that include a deskbook, checklist, and a website to help managers and users apply MRLs and conduct assessments. In addition, the Army and Air Force report that their use of MRLs on pilot programs contributed to substantial cost benefits on a variety of programs, including major acquisition programs. MRLs Were Developed from Knowledge-Based Resources on Manufacturing: To develop MRLs, the working group conducted comprehensive sessions with industry participants to ensure the metrics and vocabulary for assessing manufacturing readiness would be an all-inclusive body of knowledge. Officials stated that a mature set of manufacturing knowledge resources already existed but it was scattered and not consistently applied in a disciplined way that aligned with the DOD acquisition life-cycle framework. In their formulation, MRLs were developed from an extensive body of manufacturing knowledge that included, but was not limited to, the following defense, industry, and academic sources: * DOD Instruction 5000.02, Operation of the Defense Acquisition System (Dec. 8, 2008), * Navy best-practices manual for using templates on design and manufacturing best practices, * Air Force manufacturing development guide, * military standards and specifications, and: * Malcolm Baldrige quality award criteria. Other standards and technical sources were obtained from the Institute of Electrical and Electronics Engineers, the International Standards Organization on quality management systems, automotive industry quality standards, and the supplier model from the Massachusetts Institute of Technology. Analysis Shows MRLs Address Manufacturing Gaps in DOD's Technical Reviews: An analysis conducted by the working group shows that MRLs address many of the manufacturing gaps identified in several of DOD's technical reviews[Footnote 17] that provide program oversight and determine how well programs are meeting expected goals, particularly the reviews conducted in the early acquisition phases. According to the working group, addressing these manufacturing gaps is fundamental to improving the way programs plan, design, and prepare for manufacturing. For example, the working group's analysis shows that DOD's current systems-engineering technical review checklist used for preliminary design reviews[Footnote 18] has only 27 of 759 total questions that deal with core manufacturing-related questions, whereas the MRL 6 assessment checklist for this juncture has 169 core manufacturing questions. More importantly, the technical review checklist did not address key manufacturing discipline in the areas of program management, systems engineering, requirements management, risk management, and program schedule. Similarly, the technical review checklist used for critical design reviews[Footnote 19] has only 22 of 824 total questions that deal with core manufacturing questions, whereas the MRL 7 assessment checklist for this juncture has 162 core questions. Core manufacturing disciplines were not addressed in the specific areas of management metrics, manufacturing planning, requirements management, system verification, and other areas. Finally, DOD's technical review checklist used for production readiness reviews[Footnote 20] has 194 of 613 total questions that deal with core manufacturing questions. While the MRL 8 assessment checklist has 14 fewer core questions on manufacturing at this juncture, the working group stated these core manufacturing questions are addressed earlier in the acquisition framework, which is reflective of commercial best practices where such manufacturing topics and discipline are addressed, in contrast to DOD's current practice. Draft Deskbook Explains MRL Application and Assessments: The draft MRL deskbook is a detailed instructional resource on how to apply MRLs and conduct assessments of manufacturing risk and readiness, such as how to structure and apply evaluations to a technology, component, manufacturing process, weapon system, or subsystem using the MRL definitions. It also demonstrates how assessments should be carried out at various phases by the managers of science and technology projects and technology demonstration projects intending to transition directly to the acquisition community, as well as acquisition program managers and the people involved in conducting assessments. According to the working group, MRLs can not only be used to improve how DOD manages and communicates manufacturing risk and readiness, but can also give decision makers and manager's better visibility into program risks. For example, a variety of manufacturing status and risk evaluations have been performed for years as part of defense acquisition programs in a variety of forms--for example, production readiness reviews, manufacturing management/production capability reviews, etc. However, these structured and managed reviews do not use a uniform metric to measure and communicate manufacturing risk and readiness. MRLs, when used in combination with technology readiness levels, are expected to address two key risk areas--immature product technologies and immature manufacturing capability. The draft deskbook says that it is common for manufacturing readiness to be paced by technology readiness or design stability, and that it is not until the product technology and product design are stable that manufacturing processes will be able to mature. MRLs can also be used to define manufacturing readiness and risk at the system or subsystem level. For these reasons, the MRL definitions were designed to include a target level of technology readiness as a prerequisite for each level of manufacturing readiness. Figure 3 shows the relationship of MRLs to system milestones and technology readiness levels in the defense acquisition life-cycle framework. Figure 3: Relationship of MRLs to System Milestones and Technology Readiness Levels (TRL): [Refer to PDF for image: illustrated table] Material Solution Analysis: MRL 1: Basic manufacturing implications identified; TRL 1: Basic principles observed. MRL 2: Manufacturing concepts identified; TRL 2: Concept formulated. MRL 3: Manufacturing proof of concept developed; TRL 3: Proof of concept. MRL 4: Capability to produce the technology in a laboratory environment; TRL 4: Breadboard in laboratory. Milestone A. Technology Development: MRL 5: Capability to produce prototype components in a production relevant environment; TRL 5: Breadboard in representative environment. MRL 6: Capability to produce a prototype system or subsystem in a production relevant environment; TRL 6: Prototype in representative environment. Milestone B. Engineering and Manufacturing Development: MRL 7: Capability to produce systems, subsystems or components in a production representative environment; TRL 7: Prototype in operational environment. MRL 8: Pilot line capability demonstrated; ready to begin low rate initial production; TRL 7: Prototype in operational environment. Production and Deployment: MRL 9: Low rate production demonstrated; capability in place to begin full rate production. Production cost targets are met; TRL 8: System qualification. MRL 10: Full rate production demonstrated and lean production practices in place. Production unit cost goals are met; TRL 9: Mission proven. Source: GAO analysis of DOD chart. Note: Alignment of MRLs and TRLs within the DOD acquisition framework are generalized and may not align exactly as illustrated. [End of figure] MRL Assessments Provide Basis for Identifying, Planning, and Managing Program Risks: MRL assessments are intended to leverage better manufacturing knowledge, enabling managers to be aware of problems or risks early in development, when they are easier to resolve and before significant investments are made. In turn, these risks can be addressed earlier in the life cycle when costs are lower. For example, the ability to transition technology smoothly and efficiently from the laboratories, onto the factory floor, and into the field is a critical enabler for evolutionary acquisition. Assessments can be applied to a technology, manufacturing process, weapon system, or subsystem using the definitions as a standard. As part of the assessment, a comparison is made between the actual MRLs and the target MRL levels. The difference between the two identifies the risks and forms the basis for assisting managers to develop a plan- -called a manufacturing maturation plan--to remove or reduce them. Risks should be identified throughout the life cycle and, when targets are not met, the plan updated to ensure the appropriate MRL will be achieved at the next decision point. The manufacturing maturation plan identifies manufacturing risks and provides a plan for mitigating each risk area throughout the duration of the technology or product- development program. The draft MRL deskbook says every assessment of manufacturing readiness should have an associated plan for areas where the MRL has not achieved its target level. The deskbook requires a manufacturing maturation plan to include the most essential items in planning for the maturity of an element of assessment that is below its target MRL. These include a statement of the problem that describes areas where manufacturing readiness falls short of the target MRLs, including key factors and driving issues, solution options and consequences of each option, and a maturation plan with a schedule and funding breakout. Other information should include the status of funding to execute the manufacturing plan and specific actions to be taken and by whom, and the MRL to be achieved and when it will be achieved. MRL Pilot Programs Show Positive Benefits: Army and Air Force programs have pilot-tested MRLs on science and technology and some major acquisition programs in an effort to increase the manufacturing readiness and maturity to higher levels appropriate to the phase of development. Both services performed MRL assessments on selected pilot programs to address manufacturing risks and assess technology transition. The Army reports numerous benefits from the use of MRLs such as manufacturing efficiencies, improved labor utilization, and cost benefits. Similarly, the Air Force has used MRLs to manage its manufacturing risks associated with new technologies, yielding tangible benefits. While MRLs cannot take full credit for all benefits derived in the pilot programs, officials noted they are a good way to manage, mitigate, and communicate--between science and technology, acquisition, the user, and the system developer--readiness and risks early and throughout the acquisition process to avoid major consequences from manufacturing-related problems. These programs provide insight on how the acquisition community can utilize MRLs within weapon system programs. Army: In 2004, the Army's Aviation and Missile Research, Development and Engineering Center began applying MRLs to various technologies in concept development, including those technologies transitioning to engineering and manufacturing development. Officials stated that without cost and manufacturing readiness planning, science and technology programs face certain barriers to transition, resulting in: (1) high unit production cost caused by a focus on technology without regard to affordability; and (2) manufacturing problems caused by design complexity resulting in a technology that is not feasible to manufacture. For example, the Army has applied MRLs to many programs, including warfighter-protection materials, Micro-Electro-Mechanical Systems, embedded sensors, and helicopter cabin structures. The warfighter-protection program--the next generation of helmets and body gear--reported that it was able to reduce scrap by 60 percent and reduced touch labor by 20 to 40 percent. On programs where cost benefits could be roughly calculated, the Army believes that MRLs, among other improvement initiatives, contributed to the $426 million in benefits on seven programs. MRLs were also used as a metric in the Technology Transition Agreement to communicate manufacturing maturity and facilitate a smooth transition to the acquisition community. Air Force: Air Force officials we met with discussed using MRLs to assess and identify gaps and understand risks in manufacturing maturity that would delay technology transition into an advanced systems development program or a fielded system upgrade. The Air Force has conducted several MRL assessments on advanced technology demonstrations and major defense acquisition programs, including the MQ-9 Reaper Unmanned Aircraft, Joint Strike Fighter, Advance Medium-Range Air-to-Air Missile, X-band thin radar array, and Sensor Hardening for Tactical Systems. Officials reported that the use of MRLs have contributed millions of dollars in cost avoidance, increased production rates, and has accelerated technology transition. For example, the Air Force reported realizing $65 million in savings by addressing problems with a costly manual drilling process. MRLs were used to raise new drilling technology from MRL 4 to MRL 9, achieving a unit-cost savings of $17,000 per aircraft from reduced tooling, manpower, floor space usage, and time. Because of MRL assessment's success on advanced technology programs, the Assistant Secretary of the Air Force for Acquisition directed the program office to perform MRL assessments on key MQ-9 Reaper manufacturing processes and technologies. The MQ-9 Reaper is an unmanned aerial vehicle designed to provide a ground attack capability during reconnaissance and surveillance missions. Officials stated that the MRL assessment results have (1) identified five areas that needed review prior to a milestone C production decision; (2) identified two risks to full-rate production--mitigations are in progress; and (3) provided evidence to support the contractor's ability to meet the production goal of two aircraft per month. To ensure that manufacturing requirements are enforced, officials have developed policy for programs managers to assess manufacturing readiness at key decision points. To support that policy, the Air Force has developed training for integrated product teams to execute the manufacturing readiness assessments. Also in August 2009, the Air Force Institute of Technology established a Manufacturing Readiness Assessment course to provide training for the assessments within the Air Force and is currently open to all services and industry. DOD's Proposed MRLs Embody Many Best Practices of Leading Commercial Firms: To successfully develop and manufacture their products, the commercial firms we visited used a disciplined, gated process that emphasized manufacturing criteria early and throughout the product's development. To measure manufacturing maturity, these firms developed processes that give manufacturing readiness and producibility primary importance throughout the product-development process, focusing on producing a product, not developing a technology. The goal is business profitability, and manufacturing maturity is important to this process from the earliest stages. The best practices they employed were focused on gathering a sufficient amount of knowledge about their products' producibility in order to lower manufacturing risks and included stringent manufacturing readiness criteria--to measure whether the product was mature enough to move forward in its development. In most respects, these criteria are similar to DOD's proposed MRLs. For example, as with MRLs, commercial firms: * assess producibility at each gate using clearly defined manufacturing readiness criteria, * gain knowledge about manufacturing early, * demonstrate manufacturing processes in a production-relevant environment, and: * emphasize the importance of effective supply-chain management. Essentially, commercial firms emphasize these criteria in order to maximize their understanding of manufacturing issues, to mitigate manufacturing risks that could affect business profitability or schedule goals for getting the product to market. DOD's MRLs were designed to mitigate similar manufacturing risks. However, the difference is that the commercial firms we visited required that their manufacturing processes be in control prior to low-rate production, whereas DOD's proposed MRL criteria do not require as early control of the manufacturing process. DOD's MRLs Are Similar to Manufacturing Criteria Used by Leading Firms: Leading commercial firms use manufacturing readiness criteria, similar to DOD's MRLs, to assess the producibility of a system, gathering knowledge about the producibility of a product and the maturity of the manufacturing process. These criteria are applied early, even before a product formally enters into development, to identify and manage manufacturing risks and gaps. Additional manufacturing readiness criteria are applied through all the stages of a product's development and production until the product is ready for commercial release. The firms we visited used manufacturing readiness criteria to measure both the readiness of the product or material to enter into development and to proceed through the necessary gates. Table 3 below shows examples of manufacturing readiness criteria that are common to both the MRLs and the commercial criteria, to illustrate their similarities. Both emphasized identifying risks and developing plans to mitigate these risks, setting realistic cost goals, and proving out manufacturing processes, material, and products. Table 3: Many Manufacturing Criteria Used by Leading Commercial Firms Are Similar to DOD's MRLs: MRL/phases: MRL 1-3; Pre-Concept Development (Invention Stage); Commercial manufacturing criteria and DOD MRLs: * Relevant materials and processes evaluated for manufacturability; * Cost models developed for new processes; * Critical manufacturing processes identified. MRL/phases: MRL 4; Concept Development; Commercial manufacturing criteria and DOD MRLs: * Risk-mitigation plans in place for management of manufacturing risks; * Key materials issues identified; * Manufacturing strategy developed and integrated with acquisition strategy. MRL/phases: MRL 5-6; Technology Development; Commercial manufacturing criteria and DOD MRLs: * Basic design requirements defined and all critical technology and components tested and evaluated; * Critical suppliers identified/supply chain in place; * Realistic cost targets are set; * Manufacturing processes and materials demonstrated in a production- relevant environment. MRL/phases: MRL 7; Product Development; Commercial manufacturing criteria and DOD MRLs: * Product requirements and features well-defined; * Pilot lines' yield-data gathered and assessed; * Manufacturing processes demonstrated in a production-representative environment. MRL/phases: MRL 8; Production (Preparation); Commercial manufacturing criteria and DOD MRLs: * Quality targets demonstrated on pilot line; * Manufacturing processes verified for low-rate production on pilot line; * Yield and rates required to begin low-rate production verified; * Manufacturing plan completed and all key manufacturing risks mitigated. Source: GAO analysis of DOD and commercial data. [End of table] Best Practice: Commercial Companies Emphasize Manufacturing Criteria Early and at Every Stage of the Product-Development Life Cycle: Each commercial firm we visited developed a disciplined framework for product development that assessed producibility at each gate using clearly defined manufacturing-maturity criteria that are similar in many respects to DOD's MRLs. These include assessments of all aspects of manufacturing technology and risk, supply-chain issues, production facilities and tooling, and materials. Throughout the product- development life cycle, these criteria were applied to determine entry or exit into the next phase and led to informed decisions about whether the product was ready to move forward in its development. Manufacturing risks--such as those found in new manufacturing technologies or production facilities, new or revolutionary materials or supply-chain issues--were assessed at each step. Deliverables, including risk-identification and mitigation plans, manufacturing plans, and funding and resource needs, were required at each gate in order to progress to the next product-development gate. Targets were developed for each gate, including cost, schedule, and yield goals, and the product team was responsible for either meeting these targets or having risk-mitigation plans in place if the targets had not been met. GE Aviation exemplifies this disciplined process, using a highly structured gated process with detailed checklists for entry and exit into each phase. Like DOD's MRLs, these checklists contain increasingly detailed criteria--as they move from product start to production--for evaluating manufacturing technologies, cost drivers, materials, and supply-chain issues. Structured teams are brought together, tools are identified for execution and control of the process, and scheduled reviews are conducted with defined deliverables and checklists for each milestone. At each milestone, a vigorous review of the plans for the product's development and manufacturing and risk-reduction efforts highlights issues before they become problems. The firm's goal is to have mature processes by production. To achieve this, it considers manufacturing readiness throughout. Each project's team is cross-functional and includes senior management, mid- management and the project team. This robust review process leverages expertise across GE Aviation, reduces risk, and highlights issues before they become problems. As with all the commercial firms we visited, GE Aviation requires strong management involvement at each gate, along with decision reviews to determine if enough knowledge is available and risk- mitigation plans are in place to proceed or if actions to address and mitigate manufacturing risks can show a viable way forward. This allows management to resolve problems rather than pass them on to the next phase. At project start, which corresponds to MRL 4, the senior leadership team and product leadership team generate the product idea and assess the need for the project. They provide linkage between the business strategy and the project and develop the high-level project strategy. They identify any new product material or manufacturing processes and begin to develop a risk-reduction strategy for these issues. By the time the product enters the preliminary design phase, senior leadership and project teams agree on the approach to the project. At this time, product directors must have a manufacturing plan in place in order to identify how they are going to achieve manufacturing readiness. Technical risks are identified in the manufacturing plan, as well as risk-abatement strategies for materials and manufacturing processes and supply-chain risks. The plan has to show how issues will be successfully addressed by the detailed design phase, when leadership, the project team, and customers agree on the product to be delivered. If agreement is reached, they freeze the project plan and a decision is made to fund or terminate the project. Multidisciplinary Team/Manufacturing Experts: In the commercial firms we visited, product-development teams were multidisciplinary, generally including management, manufacturing, quality, finance, suppliers, and engineering, with necessary skills available to assess manufacturing readiness. Leading firms recognize the value of having a knowledgeable, well-trained, and skilled manufacturing engineering workforce involved in these multidisciplinary teams from the beginning and throughout the process. When Honeywell reorganized its aerospace business in 2005, it created an advanced manufacturing engineering organization to focus on manufacturing concerns in the earliest phases of new product- development programs. This organization consists of engineers to support various manufacturing disciplines in Honeywell. An important part of this advanced engineering organization is its technology group, which consists of a select number of technology fellows with extensive expertise in key manufacturing disciplines that touch nearly all the products Honeywell produces. Honeywell retains highly skilled manufacturing expertise through this program and uses these experienced and knowledgeable manufacturing engineers to oversee each project's manufacturing assessments. Maturing Technology and Manufacturing Processes: Commercial firms focus on maturing and validating technology and manufacturing processes before these are associated with a product and before entry into the gated process. They keep invention and unproven technologies in the technology base until their producibility at the scale needed can be proven. As an example, GE Healthcare's Gemstone scintillator underwent years of laboratory development on a small scale until GE Healthcare was satisfied that this material was ready to be used on its computed tomography (CT) scanners. Scintillators work by converting the X-rays in the CT scanner into visible light. GE Healthcare had been manufacturing its own scintillators since the late 1980s, but it needed an improved one that worked faster, for better clarity of vision and to reduce the amount of exposure to radiation. In 2001, the firm began basic composition development at the laboratory scale and narrowed down the alternatives to find the material with the best properties for this use. Even at this early stage, several years before the material would enter into GE Healthcare's gated process, there was early engagement by the chemists with the manufacturing side. Before they decided on a solution, a determination was made that it could produce them with sufficient yield and quality: even if a material had the best optical qualities, it had to balance this with its producibility. GE Healthcare tested thousands of alternatives to determine what could meet its technical requirements and be producible in the quantities needed. The firm narrowed it down to a garnet-based, rare-earth minerals composite, and began producing it in small but increasing quantities. After narrowing the field to this garnet-based compound, GE Healthcare began to determine its suppliers and what equipment was needed. The firm then began building its first pilot plant to produce the material and the scintillators, 2 years before the scintillator entered the firm's gated process. Figure 4 shows a photo of a CT scanner that uses the scintillator technology. Figure 4: GE CT Scanner Using Advanced Scintillator Material: [Refer to PDF for image: photograph] GE Healthcare matured its CT Scanner material years before project start to validate the material's producibility. Source: Copyright © General Electric Company USA. All rights reserved. [End of figure] Best Practice: Commercial Firms Have Adopted DOD's MRLs or Are Employing Similar Criteria in Their Product-Development Process: Because leading commercial firms focus on producibility as a key element to successfully develop products, they use rigorous analysis methods to assess producibility and to identify and manage manufacturing risks and gaps. They apply these methods and tools early and throughout product development and use them to manage their product development on a daily basis. This commercial approach is a process in which quality is designed into a product and manufacturing processes are brought into statistical control to reduce defects, in contrast to practices employed by many defense contractors where problems are identified and corrected after a product is produced. Some firms were familiar with the DOD MRL proposal and had taken steps to use the concepts at their own companies. Honeywell, for example, determined that early decisions were responsible for many production issues and so they developed analytical tools and models that support evaluations of manufacturing and risk throughout the product- development life cycle. In 2005, Honeywell engineers began looking for a way to measure manufacturing readiness and producibility, since they realized that early program decisions were driving many production issues and that by the time a product entered engineering and manufacturing development, it was too late to efficiently affect these issues. Some of these issues include cost overruns, quality problems, low-yield issues, service and maintainability inefficiencies, and supply-chain problems. A literature search led them to DOD's MRLs and they realized that these could provide the type of metric needed for a quantitative assessment. Honeywell then evolved its own criteria from these MRLs, modified to meet Honeywell's needs and expanded to address concerns such as design, obsolescence, and testability issues. Their MRL Maturity Model assessment tool, which evolved from an early version of DOD's MRLs, is the main tool in the assessment and is built upon three enabling producibility analysis tools. The model provides an MRL score for the product "as is," which is then compared to the MRL score desired to exit the phase. This model gives the firm a systematic way to be sure all the information is considered and the right questions are asked by less-experienced engineers who support the program. This MRL tool was developed 5 years ago and has evolved in an iterative, continuous improvement process since then, based on feedback from its users. Figure 5 shows a simplified depiction of this MRL model and the three enabling tools. Figure 5: Honeywell Uses Three Producibility Models and MRL Workshop: [Refer to PDF for image: illustration] Manufacturing Complexity Model: * Identifies design attributes driving manufacturing complexity. Yield Prediction Model: * Quantifies anticipated yield of proposed design concepts. Design for Manufacturing Scorecard Analysis: * Quantifies the impact of design for manufacturing violations. MRL Workshop: * Review manufacturing maturity artifacts against evaluation criteria to gain consensus on ratings for each category. The above combine to yield: MRL Assessment Tool: * Built on inputs from three producibility analysis tools and MRL workshop to evaluate manufacturing maturity and identify gaps. MRL Maturity Model Output: * MRL score. Source: GAO analysis of Honeywell data. [End of figure] The output of this tool is an MRL assessment score that can identify gaps or risks. For example, spreadsheets show the MRL scoring at a glance for each of the elements evaluated, pinpointing the gaps; risk worksheets to quantify the risks; and action plans to close the gaps and mitigate these risks. It links to the firm's gated process, providing entry and exit criteria and feedback on how to meet these criteria. The important information obtained is not necessarily what MRL level the item is at currently, but rather the robustness of the gap-closure plan to get to the desired level for the next gate. The application of the MRL tool helps identify what these key gaps are and what steps are required to close them. The three enabling producibility tools that provide support for this assessment and early input on the producibility risks are: a Design for Manufacturing Model, a Product Complexity Model and a Yield Prediction Model: * Manufacturing Complexity Model: This model identifies the design features that are driving manufacturing complexity into the design and enables scenarios to be evaluated to see what actions can be taken to simplify the design. Higher-complexity designs generally cost more and are higher risk, so the goal is to identify alternative design solutions that minimize complexity, but still meet all the performance requirements. * Yield Prediction Model: Honeywell has also developed yield prediction models based on statistical principles that correlate opportunities for defects in a design to established process capability benchmarks. This approach is used to predict yield during early design activities based on knowledge of the manufacturing processes used and the complexity of the design. * Design for Manufacturing Scorecard analysis: The third Honeywell- developed tool is a design for manufacturing scorecard, which quantifies how well the design adheres to recommended best practices. The goal of using the tool is to provide feedback to the designers so that they see how their design decisions directly affect producibility and help pinpoint improvement areas early in the process. Honeywell then conducts an MRL workshop, with a team led by an engineer from its Advanced Manufacturing Engineering group that includes the program manager and various subject-matter experts. This team reviews the tools and the MRL criteria to gain consensus on ratings for each category. Honeywell's Manufacturing Maturity Model, with input from these enabling tools, is used to develop an MRL score for the product. These assessments provide early producibility evaluations essential to mitigating design-driven risks. Since many producibility issues are driven by early design architecture decisions, these tools provide a way to analyze these decisions early and make the necessary performance and producibility trades through "virtual prototyping" long before actual hardware is built. The MRL score provides the necessary framework to ask the questions that such an analysis needs to answer. After the MRL assessment is complete and the MRL scores and risk- mitigation plans are approved, the MRL analysis and risk mitigations are incorporated into the daily schedule of the program office. The office continually monitors the MRL levels, updating them and working toward its risk mitigation goals. Best Practice: Leading Firms Prove Out Manufacturing Tooling, Equipment, and Processes before Entry into Production: Companies we visited spent years prior to production developing and proving out their manufacturing processes, including building test articles on pilot production facilities to perfect these processes. This allowed them to perfect and validate these processes, eliminate waste and scale up gradually to the required manufacturing level. They reduce errors and inefficiencies with the purpose of retiring manufacturing risks. GE Aviation officials told us that certain advanced manufacturing technologies achieve significant cost savings by getting the costs lower earlier in the process and decreasing cycle time for faster implementation. An example of manufacturing techniques or processes that have made a big difference in costs, accuracy, and reliability include processes for drilling small shaped holes for turbine airfoils. GE Aviation's Turbine Airfoils Lean Lab provides a mock-up of a production facility or process, where such technologies and production processes can be tested to eliminate waste, scrap, and excess steps. They focus on one part family or process, such as the turbine airfoil shaped-hole manufacturing. The turbine airfoil is a part of the jet engine that generates power--it extracts horsepower from the high- temperature, high-speed combusted gasses. Turbine airfoil blades require hundreds of cooling holes that help maintain part integrity at elevated operating temperatures. Traditionally, round holes were used, but the technology has evolved to compound-angle-shaped holes, which improve cooling effectiveness and reduce engine stress. These type of holes cannot be economically produced by traditional methods and require improved manufacturing techniques. Advanced laser drilling was determined to be feasible, and GE Aviation decided to initiate the program through the Lean Lab to ensure manufacturing readiness of the process. GE Aviation officials compared their processes in this case to DOD's MRLs. Prior to entering their gated process, they began making investments in potential technologies, including tooling (MRL 1-3). As the gated process began, risks were identified and risk-abatement plans were put in place (MRL 4). GE Aviation then set up the Lean Lab to test the way the airfoil would actually be built. New processes were introduced that included new laser methods for hole drilling, improved robotic technology, machining, and grinding (MRL 5-6). The managers then ran the pilot production line for some time to manufacture these airfoils using actual production operators to be confident that the process would translate to the production line. Adjustments were made to improve efficiency and retested on the line until they were satisfied that they had worked out the best procedures. GE had tooling-design experts on the team at the Lean Lab to provide rapid part and tool manufacturing. Processes were brought into statistical control in order to take the complexity out of manufacturing, simplify the process, and reduce waste (MRL 7-8). They then dismantled the production line at the Lean Lab, took it to the manufacturing facility, and set it up exactly the same, with no variations allowed (MRL 9). This seamless introduction of the new manufacturing technology and the lean principles developed in the lab are expected to save many millions of dollars across GE Aviation, on production of this part family alone. Figure 6 shows a photo of GE Aviation's Lean Lab setup. Figure 6: GE Aviation's Turbine Airfoils Lean Lab Proves Out Production Processes: [Refer to PDF for image: photograph] GE Aviation's Turbine Airfoils Lean Lab provided a seamless way to introduce new manufacturing processes. Source: Copyright © General Electric Company USA. All rights reserved. [End of figure] GE Healthcare provides another example of proving out manufacturing processes prior to production in their development of the Gemstone scintillator for use on their CT Scanners. In 2003, the technology for this transitioned into the firm's formal gated process or product start-up, and it began a detailed and extensive development of the manufacturing process. The firm built a pilot plant for this purpose and began manufacturing the composite in increasing amounts. In this first pilot plant, it was able to process the materials in increased quantities from what it produced in the lab. GE Healthcare verified that it had the right technologies to minimize manufacturing risks. In the laboratory environment, the firm had already answered the question "Can this composite be made with the desired properties?" and now asked "Can it be made with sufficient yield and quality to be manufactured in the desired amounts?" This early engagement with manufacturing enabled the firm to develop the process and reduce errors and inefficiencies with the purpose of reducing manufacturing risks. GE Healthcare then built a second pilot production plant that further increased the amount produced above that of the first pilot plant. The firm continued its focus on gaining knowledge early, but on a larger scale: building the pilot plants was important to perfecting the process and gaining knowledge about the material's producibility. At this stage, which coincides with MRL 8, it eliminated most of the technical risks involved in manufacturing the material. The firm then began to build its full-scale facility, which was ready 18 months before product launch. When the full-scale production facility was completed, further scale- up of the material's manufacturing became the focus. Changes to the design were made as needed to facilitate this. Any remaining manufacturing risks were eliminated prior to entry into the next stage, the product-validation stage. The Food and Drug Administration requires validation of finished medical devices. GE Healthcare told us that this means that all the equipment, processes, procedures, and factory workers are the same as will be used in actual production. Through use of the pilot plants to perfect the manufacturing of the scintillator material, GE Healthcare was able to produce production- representative material to satisfy this requirement. Best Practice: Commercial Firms Work Closely with Suppliers, Who Must Meet High Quality Standards for Parts and Supplies: Commercial firms focus on developing strong relationships with their suppliers to ensure quality parts are provided in a timely manner. This begins with rigorous supplier-selection criteria to create a strong supplier base to provide quality parts. Similarly, DOD's MRL supply-chain thread focuses on supplier capability throughout the acquisition life cycle, from as early as pre-milestone A (MRL 3), where initial assessment of the supply chain begins, through MRL 5, where supply-chain sources have been identified, and continuing to MRL 8, where the supply-chain should be stable and adequate to support low- rate production. Commercial firms generally have long-term relationships with these suppliers and can identify the supplier that is the best source of material or parts early, well before production begins. Leading commercial firms apply the same standards to these suppliers as they apply to their own manufacturing processes, such as ISO 9000[Footnote 21] or other quality standards. Throughout product development and production, they establish effective communications with their suppliers so they can continually assess their performance. These firms work closely with their suppliers to retain these beneficial relationships, providing training where necessary and assistance if manufacturing problems arise. GE Healthcare suppliers have to be validated before production begins, but qualifying them starts in the design phase. Suppliers are expected to meet the ISO 9000 standards and the Food and Drug Administration's medical devices standards, but GE Healthcare's own standards are more stringent that those. The supplier-qualification process ensures that suppliers meet GE Healthcare's requirements, have a quality system that provides the appropriate controls for the part provided and meet regulations and requirements of multiple agencies, such as the Food and Drug Administration. Once a supplier is qualified, it becomes an approved supplier. GE Healthcare also audits most of its suppliers and looks for issues such as lapsed ISO 9000 certification or a failed review. If it finds these things, GE Healthcare will ask the supplier for a plan to correct the deficiency and reaudit the supplier. GE Healthcare does annual risk assessments on the suppliers, based on data gathered during these audits, with sole-source or single-source suppliers being a high risk. If a supplier falls out of qualified status, GE Healthcare will do more frequent assessments. It constantly monitors the suppliers for quality. It helps the supplier get to the quality needed, but quality goals must be met. Siemens is a global company that employs about 70,000 people in the United States. We visited Siemens Mobility Division, which builds light rail cars for public transit. Siemens places special emphasis on its supplier relationships, since it knows its suppliers can contract to other rail-car builders, as there is competition for suppliers in this market. If it has a good relationship with its suppliers, it can continue to benefit from the relationships with high-quality suppliers. Once it qualifies a supplier, it takes the responsibility for keeping the supplier qualified, providing technical assistance if necessary to keep the supplier in its pipeline. Even as early as the bid phase of the contract, Siemens knows who it will need as suppliers and if any particular supplier is new or challenged in some respect. Siemens applies a three-step supplier-qualification process to its suppliers. This starts with a supplier self-assessment. The firm's supplier-qualification personnel then visit the supplier's plant and evaluate the supplier on the same self-assessment form, to determine if the supplier will make it to the vendor-qualification list. Once a supplier is on the approved vendor-qualification list, Siemens does risk ratings for these vendors to be sure it can keep them on the qualified-vendor list. The firm updates these assessments if the vendor situation changes, rating the vendor at low risk if it is fully qualified and working with it if some aspects are not qualified. Siemens takes responsibility for keeping the approved suppliers qualified, since finding and qualifying new vendors can be time- consuming and risky. It tries not to overload any one supplier, because some of their suppliers are small or specialty operations, so it keeps a pool of qualified suppliers for as many parts or materials as it can. Commercial Firms Require That Manufacturing Processes Be in Control Earlier Than DOD's MRLs: Although the firms we visited used manufacturing readiness criteria similar to DOD's proposed MRLs, one important difference we observed is that the commercial best practice is to have manufacturing processes in control prior to the production decision, while DOD's MRLs require manufacturing processes and procedures to be established and controlled during MRL 9, which occurs after the milestone C production decision, which authorizes a program to enter low-rate initial production, or equivalent.[Footnote 22] Although DOD's MRLs incorporate many of the commercial manufacturing best practices into their manufacturing design and implementation criteria, the process controls criteria would be met too late in the process to achieve their full effect. DOD's MRL matrix states that low-rate production yield and rate targets should be achieved at MRL 9, after the production decision has been made. The commercial firms we talked to emphasized that production processes must be in control before this decision is made. They realize that they are unable to make predictions about production performance until the process is stable and defects are predictable. Not achieving process control could result in low quality, extensive rework and waste, and not meeting cost and schedule targets. Firms established pilot lines to prove out production material, processes, and tooling, and worked to get processes under control before the system could move from the pilot line to production line. Figure 7 shows a depiction of the commercial manufacturing process approach. Figure 7: Leading Commercial Firms Use Statistical Controls to Ensure Quality Products: [Refer to PDF for image: illustration] Input: * Raw material, components, sub-components. Controllable inputs: * Temperature. Uncontrollable inputs: * Material properties. The above combine to form: Process. From Process: Quality characteristics. Lead to: Measure Process Control: Production decision. Source: GAO analysis of commercial firm data. [End of figure] The companies we visited used various approaches to build process capability and provide timely information on whether manufactured components, subsystems, or systems meet design specification. For example, GE Aviation uses a statistical measurement, called Z sigma level, to determine whether its processes have been brought under control or if variations in its manufacturing process could affect the quality of the product. The product is not moved into production until the firm is satisfied that these processes are in control. Similarly, GE Healthcare's milestone process requires that a set of quality targets are part of the program and that those quality targets are met. Measures of process control vary from company to company, such as using yield or scrap and rework rates or sigma levels, but each looks carefully at those measures to ensure they carried no product-quality risk and uses this information to determine if the product is ready to be manufactured. Two Successful DOD Programs Used Criteria Similar to Commercial Firms: Two DOD programs, the Army's Lakota aircraft and the Missile Defense Agency's Standard Missile 3 Block 1A, that had successful manufacturing outcomes employed some of the same practices as leading commercial firms. Both used a type of manufacturing readiness criteria to evaluate whether the programs were ready to enter into production and both programs focused on manufacturability as a key indicator of program success, using well-developed technology and a conservative approach in design and development. The Lakota aircraft, a light utility helicopter that conducts noncombat missions, was a mature aircraft design when the Army entered into the contract with the European Aeronautic Defence and Space Company to purchase this commercially available helicopter. The program shows how careful attention to manufacturing readiness can reduce program risks. According to program office officials, the contractor was chosen in part because of its manufacturing track record, and it completed extensive planning, both internally and with its supplier base, to ensure on-time and reliable deliveries. Production planning and preparation were accomplished, including assessments of the manufacturing processes, capabilities, and facilities. These assessments determined that the program was low risk and ready for full-rate production. The Lakota is currently in full- rate production and has met its cost and schedule targets. The Standard Missile 3 is a ship-based, antiballistic missile used by the Aegis ballistic missile defense system. Similar to the Lakota, the system met its cost and schedule goals by using an incremental, low- risk approach. Like the commercial firms we visited, the program built knowledge through the use of a type of manufacturing readiness criteria, which allowed the early identification of risk and implementation of mitigation strategies. The Standard Missile 3 Block IA was also on target for manufacturing cost and schedule and reported a lower cost per unit than was originally estimated on its production buys. As in the successful commercial firms we visited, manufacturing issues were considered very early in the design phase, leading to minimal changes in the program from flight test to production. MRLs Are Hampered by Lack of an Agencywide Policy and Manufacturing Workforce Concerns: While acceptance of MRLs is growing within DOD and the defense industry, the services' leadership appears to be resistant, and adoption efforts have been slow. For example, obtaining agreement on a policy that would institutionalize MRLs defensewide has proven difficult. Concerns raised by the military-service policymakers have centered on when and how the MRL assessments would be used. Officials responsible for the draft policy have promoted MRLs as an initiative that can address the manufacturing element in the design and production of weapon systems, citing commercial best practices that employ similar methods, and benefits derived from pilot programs. While extensive efforts have been made to promote the benefits of MRLs in support of a revised draft policy, it has taken nearly 2 years to allay concerns and it has not yet been approved. DOD is likely to face serious challenges even if an agreement is reached to approve the policy, however, because the number of DOD's production and manufacturing career-field employees has diminished, particularly within the Air Force. Although the services are at the beginning stages of revitalizing their production and manufacturing workforce, DOD currently does not have adequate in-house expertise with the requisite knowledge to assess manufacturing throughout DOD. Essentially, the military services and Defense Contract Management Agency have identified knowledge and manpower gaps in their manufacturing workforce and believe that any initiative deploying MRLs defensewide could be hampered as a result. Draft Policy to Institutionalize MRLs Has Proven Difficult, but the DOD Community Is Starting to See Its Value: While acceptance of MRLs is growing within DOD and the defense industry, the Army's, Navy's, and Air Force's leadership appears to be resistant and adoption efforts have been slow. For example, a July 2008 draft MRL policy memorandum garnered disagreement among the military-service policymakers. The military services' leadership agreed that MRLs provide value in the early acquisition phases but disagreed with the policy's intent to formalize the process. For example, the MRL policy memorandum stated that on the basis of analyses by GAO and the Defense Science Board--as well as positive results on two Air Force pilot programs--that acquisition category I programs be assessed using the MRL scale. In particular, the draft policy included provisions that would require programs at milestone B to be assessed at MRL 6 or beyond for all critical technologies; programs at milestone C to be assessed at MRL 8 for all critical technologies; procedures to be coordinated for including assessments of manufacturing readiness in addition to technology readiness assessments at milestone B and C; and incorporation of guidance into training materials and guidebooks on best practices for addressing manufacturing from the earliest stages of development through production and sustainment. In response to the draft policy, each of the military services issued memorandums in July 2008 to the Under Secretary of Defense (Acquisition, Technology and Logistics) or the Director, Defense Research and Engineering, stating they support MRLs and their use earlier in the acquisition process but they saw limited value in doing formal assessments prior to milestone C. In general, the services had concerns on when and how MRL assessments would be used. More specifically, their concerns included the following: * Evaluation results that could be used as the basis for go/no go decisions. * A growing number of assessments being levied on acquisition programs. * Resources required to prove out multiple production lines in a competitive prototyping environment during the technology-development phase. Since 2008, officials responsible for the draft policy memorandum have been working to address concerns raised by the services. According to the working group, most concerns pointed to a need to clarify how the information is intended to be used by decision makers at key milestones, particularly at the earlier milestones. According to the working group officials we interviewed, the intent is to inform decision makers with critical information--such as manufacturing risk and readiness measures, as appropriate to the phase of acquisition--so that knowledge-based decisions can be made earlier in the process to influence better outcomes in terms of cost and schedule in the later acquisition phases. Moreover, they cite that similar methods are employed by leading commercial firms as a best practice, plus the fact that MRL pilot programs have already demonstrated significant benefits. The revised MRL draft policy has not yet been approved. Officials familiar with the status of the draft policy stated that the leadership at one of the military services is still opposed to the idea of standardizing MRLs across DOD, and efforts to get approval have not yet occurred within the Office of the Director, Defense Research and Engineering. DOD experienced similar problems introducing technology readiness levels. There was opposition to the use of technology readiness levels, but they became a standard for programs to follow, and the standard that technologies should be demonstrated in a relevant environment became a statutory requirement for all major acquisition programs seeking to enter system development.[Footnote 23] Programs report benefits from using technology readiness levels. Some officials believe that MRLs could significantly reduce cost growth. For example, the Army and Air Force have reported MRLs were a factor that contributed to benefits of hundreds of millions of dollars in reduced program costs, improved schedule, and better performance of products. MRL Acceptance Is Growing within DOD and Defense Industry: A number of Army, Air Force, and Missile Defense Agency programs--as well as defense contractors--have embraced MRLs as the method for assessing manufacturing maturity, risk, and readiness. For example, some Army commands have opted to use them on their science and technology efforts that have manufacturing elements, and have developed a formal process for identifying them. Similarly, two of three Air Force product centers under the materiel command--the Aeronautical Systems Center and the Air Armament Center--have recently issued local policy that mandate the use of MRLs. For example, in a policy memorandum by the Aeronautical Systems Center, dated October 13, 2009, all programs are now required to have manufacturing readiness assessments using MRLs, prior to each major milestone review. The memorandum acknowledged that the transition to production has historically been challenging for many programs and that manufacturing assessments are a key tool to ensure that programs are ready to begin production. The Missile Defense Agency has included MRLs as part of their assessment criteria. In addition, senior missile defense manufacturing personnel have developed and conducted training on how to conduct these assessments. Similarly, a number of defense contractors have implemented MRLs as a discipline for identifying, managing, and communicating manufacturing risk and readiness. These contractors report a number of benefits using the MRLs, including reductions in program costs and improved production schedule. For example, in 2006, Raytheon participated in pilot MRL program assessments involving the Advanced Medium-Range Air- to-Air Missile and a portfolio of other programs and concluded the approach makes good business sense to lower risk. Raytheon claimed cost reductions of 30 percent or more could be achieved by using MRLs. Raytheon officials state that the combination of technology and manufacturing assessment processes changes the culture by driving a collaborative partnership between programs, design, and manufacturing engineering earlier in the product-development life cycle where maturity efforts can have the greatest effect on improving program affordability and predictability. As a result, Raytheon is deploying MRLs as a standard across the organization. Lockheed Martin is exploring ways to integrate MRLs within its existing review processes. As previously discussed, Honeywell adopted MRLs for use on both its defense and commercial products, and developed several models as an analysis-based approach to quantify their producibility risks. Manufacturing Workforce Knowledge and Manpower Gaps May Impede Implementation of MRLs: The services are in the beginning stages of revitalizing their manufacturing workforce, largely in response to a February 2006 Defense Science Board task force report on "The Manufacturing Technology Program: A Key to Affordably Equipping the Future Force." The report acknowledged that both the manufacturing expertise in the workforce and program funding have declined, thus eliminating much of the engineering and manufacturing talent across DOD and the industrial base. The report concluded that what was once a promising career field in the military services--with promotion paths, training, and professional development--has been systematically eliminated over the past few decades. Table 4 shows the decrease in the manufacturing career field across DOD from 2001 to 2007. Table 4: Percent of Manufacturing Workforce Decrease from 2001 to 2007: Army: 2001: 2,427; 2002: 2,333; 2003: 2,215; 2004: 2,226; 2005: 2,287; 2006: 2,193; 2007: 2,083; Percent reduction: 14*. Navy: 2001: 1,997; 2002: 2,297; 2003: 2,259; 2004: 2,232; 2005: 2,032; 2006: 2,000; 2007: 1,960; Percent reduction: <1%. Air Force: 2001: 518; 2002: 499; 2003: 409; 2004: 408; 2005: 407; 2006: 334; 2007: 326; Percent reduction: 37%. Source: DOD. Note: Data include military and civilian personnel. [End of table] As indicated, DOD's manufacturing career workforce trends show an overall decline, with the Army and Air Force having had the biggest declines at 14 percent and 37 percent, respectively. According to a DCMA official, the agency experienced about a 30 percent decrease during the same timeframe. An Army official responsible for workforce planning activities noted, however, there are no positions designated specifically for manufacturing, which make it difficult to determine the true career workforce numbers in this category. Fewer experts mean that fewer people at both the working level and in leadership positions understand the processes involved in developing and manufacturing defense systems and their importance in producing high- quality and reliable systems. Further, fewer people are capable of conducting production-readiness reviews, evaluating industry's work on programs, and staying abreast of industry research and development. According to a recent study, of major concern is that recent estimates show 30 percent of the civilian manufacturing workforce--classified as production, quality, and manufacturing--are eligible for full retirement, and approximately 26 percent will become eligible for full retirement over the next 4 years. This means DOD will soon have an exodus of its manufacturing workforce and, accordingly, must plan for this eventuality. Although the services are at the beginning stages of revitalizing their production and manufacturing workforce, program officials believe they currently do not have the in-house expertise with the requisite knowledge to assess manufacturing, if MRLs were to be mandated and deployed across DOD. For example, in interviews with career planning officials at the military services, most report that they have workforce challenges in manufacturing knowledge gaps or insufficient number of personnel to conduct the work, or both. The Defense Contract Management Agency reported similar manufacturing knowledge gaps due to a lack of focus in this area, but it now has new leadership in place and is establishing plans to address these deficiencies. Essentially, these knowledge deficiencies affect many areas, such as policy support for programs, the ability to develop an effective strategic plan and investment strategy for manufacturing technology, the ability to implement MRLs and conduct assessments, and the ability to effectively and affordably acquire high-quality weapon systems. Conclusions: MRLs, resourced and used effectively, offer the potential for DOD to achieve substantial savings and efficiencies in developing and acquiring weapon systems. MRLs have been shown to work in reducing the cost and time for developing technologies and producing systems. Moreover, they have been shown to work on individual programs, and some Army commands and Air Force centers have adopted them. They are consistent with commercial best practices and have even been adopted by some defense firms. Yet, they have not been adopted DOD-wide. MRLs are being met with resistance similar to that experienced by technology readiness levels when they were first introduced. However, technology readiness levels are now widely accepted and used across DOD. While MRLs represent a common body of knowledge and reflect many of the practices used by leading commercial companies, there is room for improvement. Criteria used for getting manufacturing processes under control are still not specific enough, allowing demonstration of controls to occur too late in the process--after the milestone C decision authorizing low-rate initial production--whereas commercial firms require that critical processes be in control earlier. While MRLs represent positive change, unless these criteria are strengthened at the time a production decision is made, DOD will have missed an opportunity to reduce the risk of continued cost growth on acquisition programs. Moreover, use of MRLs would be enhanced by the development of analytical tools, such as those used by Honeywell, to support MRL assessments. A serious concern is that DOD's in-house manufacturing workforce has been diminishing for decades and that, therefore, could hamper successful implementation of MRLs. Unless DOD develops long-range plans to build its in-house manufacturing workforce, it may not be able to realize the full potential of integrating manufacturing readiness levels into its processes. Recommendations for Executive Action: To ensure that DOD is taking steps to strengthen and improve the producibility and manufacturing readiness of technologies, weapon systems, subsystems, or manufacturing processes, we recommend that the Secretary of Defense do the following: * Require the assessment of manufacturing readiness across DOD programs using consistent MRL criteria as basis for measuring, assessing, reporting, and communicating manufacturing readiness and risk on science and technology transition projects and acquisition programs. * Direct the Office of the Director, Defense Research and Engineering to examine strengthening the MRL criteria related to the process capability and control of critical components and/or interfaces prior to milestone C, or equivalent, for low-rate initial production decision. * Direct the Office of the Director, Defense Research and Engineering to assess the need for analytical models and tools to support MRL assessments. * Assess the adequacy of the manufacturing workforce knowledge and skills base across the military services and defense agencies and develop a plan to address current and future workforce gaps. Agency Comments and Our Evaluation: DOD provided us written comments on a draft of this report. DOD partially concurred with our recommendation to require the assessment of manufacturing readiness across DOD programs using MRL criteria, and concurred with our other recommendations. Their comments can be found in appendix IV of this report. In its comments, DOD partially concurred with the recommendation that DOD programs be required to assess manufacturing readiness using consistent MRL criteria as the basis for measuring, assessing, reporting, and communicating manufacturing readiness and risk on science and technology transition projects and acquisition programs. DOD cites the Department of Defense Instruction 5000.02 as reflecting on manufacturing throughout the acquisition life cycle and, specifically, establishing a framework to continually assess and mitigate manufacturing risks. In its remarks, DOD states that the manufacturing readiness criteria will be tailored to programs and embedded into reviews and assessment templates, including systems engineering reviews, preliminary design reviews, and critical design reviews as well as acquisition phase exit criteria. While we are encouraged by DOD's plans to incorporate manufacturing readiness criteria into various assessments, we are concerned about the absence of any reference to MRLs, which identify specific benchmarks for each acquisition phase. It is unclear from DOD's comments whether it intends to use a common definition of manufacturing readiness as acquisition phase exit criteria or whether the exit criteria will be decided on a case-by-case basis. While tailoring to individual programs is appropriate, tailoring must take place in the context of well-understood criteria for moving from phase to phase. A hallmark of the commercial programs we have looked at in this and other reviews is the reliance on disciplined processes for assessing readiness to proceed into more costly development and production phases. Firm criteria are needed to identify and address producibility and manufacturing risks on a timely basis, before they result in expensive production problems. We also received technical comments from DOD, which have been addressed in the report, as appropriate. We are sending copies of this report to the Secretary of Defense; the Secretary of the Army; Secretary of the Navy; Secretary of the Air Force; Director, Missile Defense Agency; Director, Defense Contract Management Agency; and Office of Management and Budget. In addition, the report will be made available at no charge on the GAO Web site at [hyperlink, http://www.gao.gov]. If you, or your staff, have any questions concerning this report, please contact me at (202) 512-4841. Contact points for our offices of Congressional Relations and Public Affairs may be found on the last page of this report. The major contributors are listed in appendix V. Signed by: Michael J. Sullivan: Director: Acquisition and Sourcing Management: [End of section] Appendix I: Scope and Methodology: This report compares the Department of Defense (DOD) and its large prime contractors' manufacturing practices with those of leading commercial companies--with a focus on improving the manufacturing of defense weapon systems. Specifically, we assessed (1) the manufacturing problems experienced by DOD, (2) how manufacturing readiness levels (MRLs) can address DOD's manufacturing problems, (3) how proposed MRLs compare to manufacturing best practices of leading commercial companies, and (4) the challenges and barriers to implementing MRLs at DOD. To identify the manufacturing problems experienced by DOD, we performed an aggregate analysis of DOD programs from our annual assessment database. We also conducted case studies of four programs with known cost and schedule problems to make observations on the types of problems DOD weapon systems may experience. The programs we reviewed, along with the prime contractors responsible for developing the systems, are: * Joint Air-to-Surface Standoff Missile, an air-to-surface missile funded by the Air Force and developed by Lockheed Martin; - Exoatmospheric Kill Vehicle, a ballistic-missile interceptor funded by the Missile Defense Agency and developed by Raytheon; - Electromagnetic Aircraft Launch System, a launch system for aircraft carriers funded by the Navy and developed by General Atomics; and: - H-1 helicopter upgrade, tactical utility and attack helicopters funded by the Navy and developed by Bell Helicopter. To evaluate the four DOD weapon programs, we examined program documentation, such as acquisition decision memos and production readiness reviews, and held discussions with manufacturing and systems engineering officials from DOD program offices, the prime contractors, and the Defense Contract Management Agency. Based on the information gathered through interviews conducted and documentation synthesized, we identified commonalities among the case studies. To determine how MRLs can address the manufacturing problems experienced by defense programs, we conducted interviews with officials from the Office of Secretary of Defense, Office of the Director, Defense Research & Engineering, Joint Defense Manufacturing Technology Panel working group, National Center for Advanced Technologies, National Defense Industrial Association, and Defense Acquisition University on their observations on MRLs. We also reviewed the MRL deskbook, matrix (risk areas), analyses, and training materials. We also conducted interviews with Army, Navy, and Air Force officials who were involved or familiar with the pilot tests of MRLs on various programs. The pilot programs we examined at the military services include the following: * Army--micro electro-mechanical systems inertial measurement unit, micro electro-mechanical systems safety arm, ferroelectric and micro electro-mechanical systems phase shifter, low-cost materials for improved protection, rotorcraft cabin floor structure, embedded sensors, and armor manufacturing; * Air Force--MQ-9 Reaper, F-35 Joint Strike Fighter induct inlet, high- durability hot exhaust structures, F-135 Pratt & Whitney propulsion system, sensor hardening for tactical systems, and X-Band thin array radar; and: * Navy--P-8A aircraft. To identify practices and criteria used by leading commercial companies that can be used to improve DOD's manufacturing process, we selected and visited five companies based on several criteria: companies that (1) make products that are comparable to DOD in terms of complexity, (2) are recognized as leaders in developing manufacturing readiness criteria, or (3) have won awards for their manufacturing best practices, or a mix of the above. We met with these companies to discuss their product-development and manufacturing practices and the steps that they take to mitigate manufacturing risks, ensure manufacturing readiness, and improve supplier quality. We met with these companies to discuss their product-development life cycle and the methods and metrics they use to measure manufacturing maturity and producibility; manufacturing risk management; supplier management; and the key factors in the company's successful manufacturing outcomes. We generalized much of the information due to the proprietary nature of the data relating to their manufacturing processes. Several companies provided data on specific processes or products that they agreed to allow us to include in this report. We reported on four of the five companies we visited. The five companies we visited include the following: * GE Aviation, a leading aerospace company, whose portfolio includes commercial engines and services, military engines and services, business and general aviation, engine components, and aviation systems. We met with manufacturing and quality officials in Cincinnati, Ohio, and discussed their manufacturing practices and manufacturing maturity metrics. We also toured their Lean Lab production facility and saw how these practices were applied. * GE Healthcare, which manufactures a range of products and services that includes medical imaging and information technologies and medical diagnostics. We met with manufacturing officials at their Milwaukee, Wisconsin, plant and discussed their manufacturing practices, including the development and manufacturing of their Gemstone scintillator for use on advanced CT scanners. * Honeywell Aerospace, a global provider of integrated avionics, engines, systems, and services for aircraft manufacturers, airlines, business and general aviation, and military and space operations. We met with manufacturing officials at their Phoenix, Arizona, facility and discussed their manufacturing maturity processes and the models and tools they used to assess this. * Siemens Mobility, a division of Siemens that develops and builds light rail cars for the North American market. We met with manufacturing and procurement officials at their Sacramento, California, manufacturing and assembly plant to discuss the manufacturing processes used in building their rail cars and their supplier management practices. * Toyota Motor Engineering and Manufacturing is responsible for Toyota's engineering design, development and manufacturing activities in North America. We met with officials in their production engineering division in Erlanger, Kentucky, and also in their Toyota Technical Center located in Ann Arbor, Michigan, and discussed their vehicle development process and their methods for assuring supplier quality. At each of the companies, we interviewed senior management officials knowledgeable about the manufacturing methods, techniques, and practices used throughout manufacturing and product development to ensure manufacturing maturity and producibility of their products. In particular, we discussed their (1) product development life cycle and the methods, metrics, and tools used to determine manufacturing maturity and producibility, (2) methods for identifying and mitigating risks in manufacturing a product, and (3) methods for supplier management to provide steady supply of quality parts. In addition, we compared the practices of commercial firms to two major defense weapon systems known to be producing systems within cost and schedule goals and with successful manufacturing outcomes. To evaluate these two programs, we examined program documentation and held discussions with program and contracting officials. The two systems we reviewed, along with the prime contractors responsible for developing the systems, are: * Lakota aircraft, a light utility helicopter that conducts noncombat missions, funded by the Army and developed by the European Aeronautic Defence and Space Company; and: * Standard Missile 3 Block 1A, a ship-based antiballistic missile, funded by the Missile Defense Agency and developed by Raytheon. To determine the challenges and barriers to MRL implementation efforts, we interviewed officials who were involved with the draft policy to standardize MRLs, as well as the military-service policy organizations that commented on the proposal. We also synthesized the information gathered at the various levels throughout the defense community to determine the issues surrounding MRLs as well as their merits. These DOD organizations include the Office of the Secretary of Defense, each of the military-service policy groups and program offices, the Office of the Director, Defense Research and Engineering, Systems and Software Engineering, and Joint Defense Manufacturing Technology Panel. To obtain an understanding of the workforce challenges in manufacturing, we reviewed selected documentation--such as Defense Science Board studies--and interviewed officials at each of the military services and Defense Contract Management Agency who were responsible for workforce planning activities and revitalization initiatives. We conducted this performance audit from January 2009 to February 2010 in accordance with generally accepted government auditing standards. These standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. [End of section] Appendix II: Manufacturing Readiness Level (MRL) Definitions: MRL 1--Basic Manufacturing Implications Identified: This is the lowest level of manufacturing readiness. The focus is to address manufacturing shortfalls and opportunities needed to achieve program objectives. Basic research (i.e., budget activity 6.1 funds) begins in the form of studies. MRL 2--Manufacturing Concepts Identified: This level is characterized by describing the application of new manufacturing concepts. Applied research (i.e., budget activity 6.2 funds) translates basic research into solutions for broadly defined military needs. Typically this level of readiness in the science and technology environment includes identification, paper studies, and analysis of material and process approaches. An understanding of manufacturing feasibility and risk is emerging. MRL 3--Manufacturing Proof of Concept Developed: This level begins the validation of the manufacturing concepts through analytical or laboratory experiments. This level of readiness is typical of technologies in the science and technology funding categories of Applied Research and Advanced Development (i.e., budget activity 6.3 funds). Materials or processes, or both, have been characterized for manufacturability and availability but further evaluation and demonstration is required. Experimental hardware models have been developed in a laboratory environment that may possess limited functionality. MRL 4--Capability to Produce the Technology in a Laboratory Environment: This level of readiness is typical for science and technology programs in the budget activity 6.2 and 6.3 categories and acts as exit criteria for the materiel solution analysis phase approaching a milestone A decision. Technologies should have matured to at least technology readiness level 4. This level indicates that the technologies are ready for the technology-development phase of acquisition. At this point, required investments, such as manufacturing technology development, have been identified. Processes to ensure manufacturability, producibility, and quality are in place and are sufficient to produce technology demonstrators. Manufacturing risks have been identified for prototype build, and mitigation plans are in place. Target cost objectives have been established and manufacturing cost drivers have been identified. Producibility assessments of design concepts have been completed. Key design performance parameters have been identified as well as any special tooling, facilities, material handling, and skills required. MRL 5--Capability to Produce Prototype Components in a Production- Relevant Environment: This level of maturity is typical of the midpoint in the technology- development phase of acquisition, or in the case of key technologies, near the midpoint of an advanced technology-demonstration project. Technologies should have matured to at least technology readiness level 5. The industrial base has been assessed to identify potential manufacturing sources. A manufacturing strategy has been refined and integrated with the risk-management plan. Identification of enabling/ critical technologies and components is complete. Prototype materials, tooling and test equipment, as well as personnel skills, have been demonstrated on components in a production-relevant environment, but many manufacturing processes and procedures are still in development. Manufacturing technology development efforts have been initiated or are ongoing. Producibility assessments of key technologies and components are ongoing. A cost model has been constructed to assess projected manufacturing cost. MRL 6--Capability to Produce a Prototype System or Subsystem in a Production-Relevant Environment: This MRL is associated with readiness for a milestone B decision to initiate an acquisition program by entering into the engineering and manufacturing development phase of acquisition. Technologies should have matured to at least technology readiness level 6. It is normally seen as the level of manufacturing readiness that denotes completion of science and technology development and acceptance into a preliminary system design. An initial manufacturing approach has been developed. The majority of manufacturing processes have been defined and characterized, but there are still significant engineering or design changes, or both, in the system itself. However, preliminary design of critical components has been completed and producibility assessments of key technologies are complete. Prototype materials, tooling and test equipment, as well as personnel skills have been demonstrated on systems or subsystems, or both, in a production- relevant environment. A cost analysis has been performed to assess projected manufacturing cost versus target cost objectives and the program has in place appropriate risk reduction to achieve cost requirements or establish a new baseline. This analysis should include design trades. Producibility considerations have shaped system- development plans. Industrial capabilities assessment for milestone B has been completed. Long-lead and key supply-chain elements have been identified. All subcontractors have been identified. MRL 7--Capability to Produce Systems, Subsystems, or Components in a Production-Representative Environment: This level of manufacturing readiness is typical for the midpoint of the engineering and manufacturing-development phase leading to the post-critical design review assessment. Technologies should be maturing to at least technology readiness level 7. System detailed design activity is underway. Material specifications have been approved and materials are available to meet the planned pilot-line build schedule. Manufacturing processes and procedures have been demonstrated in a production-representative environment. Detailed producibility trade studies and risk assessments are underway. The cost model has been updated with detailed designs, rolled up to system level, and tracked against allocated targets. Unit-cost reduction efforts have been prioritized and are underway. The supply chain and supplier quality assurance have been assessed and long-lead procurement plans are in place. Production tooling and test equipment design and development have been initiated. MRL 8--Pilot-Line Capability Demonstrated; Ready to Begin Low-Rate Initial Production: This level is associated with readiness for a milestone C decision, and entry into low-rate initial production. Technologies should have matured to at least technology readiness level 7. Detailed system design is essentially complete and sufficiently stable to enter low- rate production. All materials are available to meet the planned low- rate production schedule. Manufacturing and quality processes and procedures have been proven in a pilot-line environment and are under control and ready for low-rate production. Known producibility risks pose no significant challenges for low-rate production. The engineering cost model is driven by detailed design and has been validated with actual data. The Industrial Capability Assessment for milestone C has been completed and shows that the supply chain is established and stable. MRL 9--Low-Rate Production Demonstrated; Capability in Place to Begin Full-Rate Production: At this level, the system, component, or item has been previously produced, is in production, or has successfully achieved low-rate initial production. Technologies should have matured to at least technology readiness level 9. This level of readiness is normally associated with readiness for entry into full-rate production. All systems-engineering/design requirements should have been met such that there are minimal system changes. Major system design features are stable and have been proven in test and evaluation. Materials are available to meet planned rate production schedules. Manufacturing process capability in a low-rate production environment is at an appropriate quality level to meet design key-characteristic tolerances. Production risk monitoring is ongoing. Low-rate initial production cost targets have been met, with learning curves validated. The cost model has been developed for the full-rate production environment and reflects the effect of continuous improvement. MRL 10--Full-Rate Production Demonstrated and Lean Production Practices in Place: This is the highest level of production readiness. Technologies should have matured to at least technology readiness level 9. This level of manufacturing is normally associated with the production or sustainment phases of the acquisition life cycle. Engineering/design changes are few and generally limited to quality and cost improvements. System, components, or items are in full-rate production and meet all engineering, performance, quality, and reliability requirements. Manufacturing process capability is at the appropriate quality level. All materials, tooling, inspection and test equipment, facilities, and manpower are in place and have met full-rate production requirements. Rate production unit costs meet goals, and funding is sufficient for production at required rates. Lean practices are well established and continuous process improvements are ongoing. [End of section] Appendix III: Manufacturing Readiness Level (MRL) Threads and Subthreads (Risk Areas): Acquisition Phase: Thread: Technology and industrial base; Acquisition Phase: Subthread: Technology maturity: Technology transition to production; Pre-material solution analysis: MRL 1: Should be assessed at TRL 1: [Empty]; Pre-material solution analysis: MRL 2: Should be assessed at TRL 2: [Empty]; Pre-material solution analysis: MRL 3: Should be assessed at TRL 3: Potential sources identified for technology needs. Understand state of the art. Material solution analysis: MRL 4 (Milestone A): Should be assessed at TRL 4: Industrial base capabilities surveyed and known; gaps/risks identified for preferred concept, key technologies, components, and/or key processes; Technology development: MRL 5: Should be assessed at TRL 5: Industrial base assessment initiated to identify potential manufacturing sources. Sole/single/foreign source vendors have been identified and planning has begun to minimize risks; Technology development: MRL 6 (Milestone B): Should be assessed at TRL 6: Industrial Capabilities Assessment (ICA) for Milestone (MS) B has been completed. Indus-trial capability in place to support manufacturing of development articles. Plans to minimize sole/foreign sources complete. Need for sole/single/foreign sources justified. Potential alternative sources identified; Engineering and manufacturing development: MRL 7: Should be assessed at TRL 7: Industrial capability to support production has been analyzed. Sole/single/foreign sources stability is assessed/monitored. Developing potential alternate sources as necessary; Engineering and manufacturing development: MRL 8 (Milestone C): Should be assessed at TRL 7: ICA for MS C has been completed. Industrial capability is in place to support Low Rate Initial Production (LRIP). Sources are available, multi-sourcing where cost-effective or necessary to mitigate risk; Low rate initial product: MRL 9 (Milestone FRP): Should be assessed at TRL 8: Industrial capability is in place to support start of Full Rate Production (FRP); Full rate product: MRL 10: Should be assessed at TRL 9: Industrial capability supports FRP and is assessed to support modifications, upgrades, surge and other potential manufacturing requirements. Acquisition Phase: Thread: Acquisition Phase: Subthread: Technology maturity: Pre-material solution analysis: MRL 1: Should be assessed at TRL 1: Pre-material solution analysis: MRL 2: Should be assessed at TRL 2: Pre-material solution analysis: MRL 3: Should be assessed at TRL 3: Material solution analysis: MRL 4 (Milestone A): Should be assessed at TRL 4: Technology development: MRL 5: Should be assessed at TRL 5: Technology development: MRL 6 (Milestone B): Should be assessed at TRL 6: Engineering and manufacturing development: MRL 7: Should be assessed at TRL 7: Engineering and manufacturing development: MRL 8 (Milestone C): Should be assessed at TRL 7: Low rate initial product: MRL 9 (Milestone FRP): Should be assessed at TRL 8: Full rate product: MRL 10: Should be assessed at TRL 9: Acquisition Phase: Thread: Acquisition Phase: Subthread: Technology maturity: Pre-material solution analysis: MRL 1: Should be assessed at TRL 1: Pre-material solution analysis: MRL 2: Should be assessed at TRL 2: Pre-material solution analysis: MRL 3: Should be assessed at TRL 3: Material solution analysis: MRL 4 (Milestone A): Should be assessed at TRL 4: Technology development: MRL 5: Should be assessed at TRL 5: Technology development: MRL 6 (Milestone B): Should be assessed at TRL 6: Engineering and manufacturing development: MRL 7: Should be assessed at TRL 7: Engineering and manufacturing development: MRL 8 (Milestone C): Should be assessed at TRL 7: Low rate initial product: MRL 9 (Milestone FRP): Should be assessed at TRL 8: Full rate product: MRL 10: Should be assessed at TRL 9: Acquisition Phase: Thread: Acquisition Phase: Subthread: Technology maturity: Pre-material solution analysis: MRL 1: Should be assessed at TRL 1: Pre-material solution analysis: MRL 2: Should be assessed at TRL 2: Pre-material solution analysis: MRL 3: Should be assessed at TRL 3: Material solution analysis: MRL 4 (Milestone A): Should be assessed at TRL 4: Technology development: MRL 5: Should be assessed at TRL 5: Technology development: MRL 6 (Milestone B): Should be assessed at TRL 6: Engineering and manufacturing development: MRL 7: Should be assessed at TRL 7: Engineering and manufacturing development: MRL 8 (Milestone C): Should be assessed at TRL 7: Low rate initial product: MRL 9 (Milestone FRP): Should be assessed at TRL 8: Full rate product: MRL 10: Should be assessed at TRL 9: Acquisition Phase: Thread: Technology and industrial base; Acquisition Phase: Subthread: Technology maturity: Manufacturing technology development; Pre-material solution analysis: MRL 1: Should be assessed at TRL 1: [Empty]; Pre-material solution analysis: MRL 2: Should be assessed at TRL 2: New manufacturing concepts and potential solutions identified; Pre-material solution analysis: MRL 3: Should be assessed at TRL 3: Manufacturing technology concepts identified through experiments/models; Material solution analysis: MRL 4 (Milestone A): Should be assessed at TRL 4: Manufacturing Science and Advanced Manufacturing Technology requirements identified; Technology development: MRL 5: Should be assessed at TRL 5: Required manufacturing technology development efforts initiated as applicable; Technology development: MRL 6 (Milestone B): Should be assessed at TRL 6: Manufacturing technology efforts continuing. Required manufacturing technology development solutions demonstrated in a production-relevant environment; Engineering and manufacturing development: MRL 7: Should be assessed at TRL 7: Manufacturing technology efforts continuing. Required manufacturing technology development solutions demonstrated in a production-representative environment; Engineering and manufacturing development: MRL 8 (Milestone C): Should be assessed at TRL 7: Primary manufacturing technology efforts concluding and some improvement efforts continuing. Required manufacturing technology solutions validated on a pilot line; Low rate initial product: MRL 9 (Milestone FRP): Should be assessed at TRL 8: Manufacturing technology process improvement efforts initiated for FRP; Full rate product: MRL 10: Should be assessed at TRL 9: Manufacturing technology continuous process improvements ongoing. Acquisition Phase: Thread: Design; Acquisition Phase: Subthread: Technology maturity: Producibility program; Pre-material solution analysis: MRL 1: Should be assessed at TRL 1: [Empty]; Pre-material solution analysis: MRL 2: Should be assessed at TRL 2: [Empty]; Pre-material solution analysis: MRL 3: Should be assessed at TRL 3: Relevant materials/processes evaluated for manufacturability using experimental results; Material solution analysis: MRL 4 (Milestone A): Should be assessed at TRL 4: Initial producibility and manufacturability assessment of preferred systems concepts completed. Results considered in selection of preferred design concepts and reflected in Technology Development Strategy (TDS) key components/technologies; Technology development: MRL 5: Should be assessed at TRL 5: Producibility and manufacturability assessments of key technologies and components initiated as appropriate. On-going design trades consider manufacturing processes and industrial base capability constraints. Manufacturing processes assessed for capability to test and verify in production, and influence on operations & support; Technology development: MRL 6 (Milestone B): Should be assessed at TRL 6: Producibility assessments and producibility trade studies (performance vs. producibility) of key technologies/components completed. Results used to shape Acquisition Strategy (AS), Systems Engineering Plan (SEP), Manufacturing and Producibility plans, and planning for Engineering and Manufacturing Development (EMD) or technology insertion programs. Preliminary design choices assessed against manufacturing processes and industrial base capability constraints. Producibility enhancement efforts (e.g., Design for Manufacturing (DFX)) initiated; Engineering and manufacturing development: MRL 7: Should be assessed at TRL 7: Detailed producibility trade studies using knowledge of key design characteristics and related manufacturing process capability completed. Producibility enhancement efforts (e.g., DFX) ongoing for optimized integrated system. Manufacturing processes reassessed as needed for capability to test and verify potential influence on operations & support; Engineering and manufacturing development: MRL 8 (Milestone C): Should be assessed at TRL 7: Producibility improvements implemented on system. Known producibility issues have been resolved and pose no significant risk for LRIP; Low rate initial product: MRL 9 (Milestone FRP): Should be assessed at TRL 8: Prior producibility improvements analyzed for effectiveness during LRIP. Producibility issues/risks discovered in LRIP have been mitigated and pose no significant risk for FRP; Full rate product: MRL 10: Should be assessed at TRL 9: Design producibility improvements demonstrated in FRP. Process producibility improvements ongoing. All modifications, upgrades, Diminishing Manufacturing Sources and Material Shortages (DMSMS), and other changes assessed for producibility. Acquisition Phase: Thread: Design; Acquisition Phase: Subthread: Technology maturity: Design maturity; Pre-material solution analysis: MRL 1: Should be assessed at TRL 1: Manufacturing research opportunities identified; Pre-material solution analysis: MRL 2: Should be assessed at TRL 2: Applications defined. Broad performance goals identified that may drive manufacturing options; Pre-material solution analysis: MRL 3: Should be assessed at TRL 3: Top level performance requirements defined. Tradeoffs in design options assessed based on experiments. Product life cycle and technical requirements evaluated; Material solution analysis: MRL 4 (Milestone A): Should be assessed at TRL 4: SEP and Test and Evaluation (T&E) Strategy recognize the need for the establishment/validation of manufacturing capability and management of manufacturing risk for the product life cycle. Initial potential Key Performance Parameter (KPPs) identified for preferred systems concept. System characteristics and measures to support required capabilities identified. Form, fit, and function constraints identified, and manufacturing capabilities identified for preferred systems concepts; Technology development: MRL 5: Should be assessed at TRL 5: Lower level performance requirements sufficient to proceed to preliminary design. All enabling/critical technologies and components identified and product life cycle considered. Evaluation of design Critical Characteristic (KCs) initiated. Product data required for prototype component manufacturing released; Technology development: MRL 6 (Milestone B): Should be assessed at TRL 6: System allocated baseline established. System and subsystem preliminary design sufficient for EMD. All enabling/critical technologies/components have been demonstrated. Preliminary design KCs defined; Engineering and manufacturing development: MRL 7: Should be assessed at TRL 7: Product requirements and features are defined well enough to support Critical Design Review (CDR) even though design change traffic may be significant. All product data essential for component manufacturing has been released. Potential KC risk issues have been identified and mitigation plan is in place; Engineering and manufacturing development: MRL 8 (Milestone C): Should be assessed at TRL 7: Detailed design of product features and interfaces is complete. All product data essential for system manufacturing has been released. Design change traffic does not significantly impact LRIP. KCs are attainable based upon pilot line demonstrations; Low rate initial product: MRL 9 (Milestone FRP): Should be assessed at TRL 8: Major product design features and configuration are stable. System design has been validated through operational testing of LRIP items. Physical Configuration Audit (PCA) or equivalent complete as necessary. Design change traffic is limited. All KCs are controlled in LRIP to appropriate quality levels; Full rate product: MRL 10: Should be assessed at TRL 9: Product design is stable. Design changes are few and generally limited to those required for continuous improvement or in reaction to obsolescence. All KCs are controlled in FRP to appropriate quality levels. Acquisition Phase: Thread: Cost and funding; Acquisition Phase: Subthread: Technology maturity: Production cost knowledge (cost modeling); Pre-material solution analysis: MRL 1: Should be assessed at TRL 1: [Empty]; Pre-material solution analysis: MRL 2: Should be assessed at TRL 2: Cost model approach defined; Pre-material solution analysis: MRL 3: Should be assessed at TRL 3: Initial cost targets and risks identified. High level process chart model developed. Technology cost models developed for new process steps and materials based on experiments; Material solution analysis: MRL 4 (Milestone A): Should be assessed at TRL 4: Manufacturing, material and special requirement cost drivers identified. Detailed process chart cost models driven by process variables. Cost driver uncertainty quantified; Technology development: MRL 5: Should be assessed at TRL 5: Manufacturing, material and special requirement cost drivers identified. Detailed process chart cost models driven by process variables. Cost driver uncertainty quantified; Technology development: MRL 6 (Milestone B): Should be assessed at TRL 6: Cost model updated with design requirements, material specifications, tolerances, integrated master schedule, results of system/subsystem simulations and production relevant prototype demonstrations; Engineering and manufacturing development: MRL 7: Should be assessed at TRL 7: Cost model updated with the results of systems/sub-systems produced in a production-representative environment and with production plant layout and design and obsolescence solutions; Engineering and manufacturing development: MRL 8 (Milestone C): Should be assessed at TRL 7: Cost models updated with results of pilot line build; Low rate initial product: MRL 9 (Milestone FRP): Should be assessed at TRL 8: FRP cost model up-dated with result of LRIP build; Full rate product: MRL 10: Should be assessed at TRL 9: Cost model validated against actual FRP cost. Acquisition Phase: Thread: Cost and funding; Acquisition Phase: Subthread: Technology maturity: Cost analysis; Pre-material solution analysis: MRL 1: Should be assessed at TRL 1: Identify any manufacturing cost implications; Pre-material solution analysis: MRL 2: Should be assessed at TRL 2: Cost elements identified; Pre-material solution analysis: MRL 3: Should be assessed at TRL 3: Sensitivity analysis conducted to define cost drivers and production development strategy (i.e., lab to pilot to factory); Material solution analysis: MRL 4 (Milestone A): Should be assessed at TRL 4: Producibility cost risks assessed. Initial cost models support Analysis of Alternatives (AoA) and Alternative Systems Review (ASR); Technology development: MRL 5: Should be assessed at TRL 5: Costs analyzed using prototype component actuals to ensure target costs are achievable. Decisions regarding design choices, make/buy, capacity, process capability, sources, quality, KCs, yield/rate, and variability influenced by cost models; Technology development: MRL 6 (Milestone B): Should be assessed at TRL 6: Costs analyzed using prototype system/subsystem actuals to ensure target costs are achievable. Allocate cost targets to subsystems. Cost reduction and avoidance strategies developed; Engineering and manufacturing development: MRL 7: Should be assessed at TRL 7: Manufacturing costs rolled up to system/sub-system level and tracked against targets. Detailed trade studies and engineering change requests supported by cost estimates. Cost reduction and avoidance strategies underway; Engineering and manufacturing development: MRL 8 (Milestone C): Should be assessed at TRL 7: Costs analyzed using pilot line actuals to ensure target costs are achievable. Manufacturing cost analysis supports proposed changes to requirements or configuration. Cost reduction initiatives ongoing; Low rate initial product: MRL 9 (Milestone FRP): Should be assessed at TRL 8: LRIP cost goals met and learning curve analyzed with actual data. Cost reduction initiatives ongoing. Touch labor efficiency analyzed to meet production rates and elements of inefficiency are identified with plans in place for reduction; Full rate product: MRL 10: Should be assessed at TRL 9: FRP cost goals met. Cost reduction initiatives ongoing. Acquisition Phase: Thread: Cost and funding; Acquisition Phase: Subthread: Technology maturity: Manufacturing investment budget; Pre-material solution analysis: MRL 1: Should be assessed at TRL 1: Potential investments identified; Pre-material solution analysis: MRL 2: Should be assessed at TRL 2: Program/projects have reasonable budget estimates for reaching MRL 3 through experiment; Pre-material solution analysis: MRL 3: Should be assessed at TRL 3: Program/projects have reasonable budget estimates for reaching MRL 4 by MS A; Material solution analysis: MRL 4 (Milestone A): Should be assessed at TRL 4: Manufacturing technology initiatives identified to reduce costs. Program has reasonable budget estimate for reaching MRL 6 by MS B. Estimate includes capital investment for production-relevant equipment. All outstanding MRL 4 risk areas understood, with approved mitigation plans in place; Technology development: MRL 5: Should be assessed at TRL 5: Program has updated budget estimate for reaching MRL 6 by MS B. All outstanding MRL 5 risk areas understood, with approved mitigation plans in place; Technology development: MRL 6 (Milestone B): Should be assessed at TRL 6: Program has reasonable budget estimate for reaching MRL 8 by MS C. Estimate includes capital investment for production-representative equipment by CDR and pilot line equipment by MS C. All outstanding MRL 6 risk areas understood, with approved mitigation plans in place; Engineering and manufacturing development: MRL 7: Should be assessed at TRL 7: Program has updated budget estimate for reaching MRL 8 by MS C. All outstanding MRL 7 risk areas understood, with approved mitigation plans in place; Engineering and manufacturing development: MRL 8 (Milestone C): Should be assessed at TRL 7: Program has reasonable budget estimate for reaching MRL 9 by the FRP decision point. Estimate includes investment for LRIP and FRP. All outstanding MRL 8 risk areas understood, with approved mitigation plans in place; Low rate initial product: MRL 9 (Milestone FRP): Should be assessed at TRL 8: Program has reasonable budget estimate for FRP. All outstanding MRL 9 risk areas understood, with approved mitigation plans in place; Full rate product: MRL 10: Should be assessed at TRL 9: Production budgets sufficient for production at required rates and schedule to support funded program. Acquisition Phase: Thread: Materials (raw materials, components, sub- assemblies and subsystems); Acquisition Phase: Subthread: Technology maturity: Maturity; Pre-material solution analysis: MRL 1: Should be assessed at TRL 1: Material properties identified for research; Pre-material solution analysis: MRL 2: Should be assessed at TRL 2: Material properties and characteristics predicted. Material properties and characteristics predicted; Pre-material solution analysis: MRL 3: Should be assessed at TRL 3: Material properties validated and assessed for basic manufacturability using experiments; Material solution analysis: MRL 4 (Milestone A): Should be assessed at TRL 4: Survey determines that the projected material has been produced in a laboratory environment. Survey determines that the projected material has been produced in a laboratory environment; Technology development: MRL 5: Should be assessed at TRL 5: Materials have been manufactured or produced in a prototype environment (maybe in a similar application/program). Maturity efforts in place to address new material production risks for technology demonstration; Technology development: MRL 6 (Milestone B): Should be assessed at TRL 6: Material maturity verified through technology demonstration articles. Preliminary material specifications in place and material properties have been adequately characterized; Engineering and manufacturing development: MRL 7: Should be assessed at TRL 7: Material maturity sufficient for pilot line build. Material specifications approved. Material maturity sufficient for pilot line build. Material specifications approved Engineering and manufacturing development: MRL 8 (Milestone C): Should be assessed at TRL 7: Materials proven and validated during EMD as adequate to support LRIP. Material specification stable; Low rate initial product: MRL 9 (Milestone FRP): Should be assessed at TRL 8: Material is proven and controlled to specification in LRIP. Material is proven and controlled to specification in LRIP; Full rate product: MRL 10: Should be assessed at TRL 9: Material is proven and controlled to specification in FRP. Material is proven and controlled to specification in FRP. Acquisition Phase: Thread: Materials (raw materials, components, sub- assemblies and subsystems); Acquisition Phase: Subthread: Technology maturity: Availability; Pre-material solution analysis: MRL 1: Should be assessed at TRL 1: [Empty]; Pre-material solution analysis: MRL 2: Should be assessed at TRL 2: Material availability assessed; Pre-material solution analysis: MRL 3: Should be assessed at TRL 3: Material scale-up issues identified; Material solution analysis: MRL 4 (Milestone A): Should be assessed at TRL 4: Projected lead times have been identified for all difficult-to- obtain, difficult-to-process, or hazardous materials. Quantities and lead times estimated; Technology development: MRL 5: Should be assessed at TRL 5: Availability issues addressed for prototype build. Significant material risks identified for all materials. Planning has begun to address scale-up issues; Technology development: MRL 6 (Milestone B): Should be assessed at TRL 6: Availability issues addressed to meet EMD build. Long-lead items identified. Potential obsolescence issues identified; Engineering and manufacturing development: MRL 7: Should be assessed at TRL 7: Availability issues addressed to meet EMD builds. Long-lead procurement identified/planned for LRIP. Obsolescence plan in place; Engineering and manufacturing development: MRL 8 (Milestone C): Should be assessed at TRL 7: Long-lead procurement initiated for LRIP. Availability issues pose no significant risk for LRIP; Low rate initial product: MRL 9 (Milestone FRP): Should be assessed at TRL 8: Long-lead procurement initiated for FRP. Availability issues pose no significant risk for FRP; Full rate product: MRL 10: Should be assessed at TRL 9: Program is in FRP, with no significant material availability issues. Acquisition Phase: Thread: Materials (raw materials, components, sub- assemblies and subsystems); Acquisition Phase: Subthread: Technology maturity: Supply chain management; Pre-material solution analysis: MRL 1: Should be assessed at TRL 1: [eMPTY]; Pre-material solution analysis: MRL 2: Should be assessed at TRL 2: [eMPTY]; Pre-material solution analysis: MRL 3: Should be assessed at TRL 3: Initial assessment of potential supply chain capability; Material solution analysis: MRL 4 (Milestone A): Should be assessed at TRL 4: Survey completed for potential supply chain sources; Technology development: MRL 5: Should be assessed at TRL 5: Potential supply chain sources identified; Technology development: MRL 6 (Milestone B): Should be assessed at TRL 6: Supply chain plans in place (e.g. teaming agreements and so forth) leading to an EMD contract award; Engineering and manufacturing development: MRL 7: Should be assessed at TRL 7: Effective supply chain management process in place. Assessment of critical first tier supply chain completed; Engineering and manufacturing development: MRL 8 (Milestone C): Should be assessed at TRL 7: Supply chain adequate to support LRIP. Assessment of critical second and lower tier supply chain completed; Low rate initial product: MRL 9 (Milestone FRP): Should be assessed at TRL 8: Supply chain is stable and adequate to support FRP. Long-term agreements in place where practical; Full rate product: MRL 10: Should be assessed at TRL 9: Supply chain proven and supports FRP requirements. Acquisition Phase: Thread: Materials (raw materials, components, sub- assemblies and subsystems); Acquisition Phase: Subthread: Technology maturity: Special handling (i.e., Government Furnished Property (GFP), shelf life, security, hazardous materials, storage environment, and so forth); Pre-material solution analysis: MRL 1: Should be assessed at TRL 1: [Empty]; Pre-material solution analysis: MRL 2: Should be assessed at TRL 2: Initial evaluation of potential regulatory requirements and special handling concerns; Pre-material solution analysis: MRL 3: Should be assessed at TRL 3: List of hazardous materials identified. Special handling procedures applied in the lab. Special handling concerns assessed; Material solution analysis: MRL 4 (Milestone A): Should be assessed at TRL 4: List of hazardous materials updated. Special handling procedures applied in the lab. Special handling requirements identified; Technology development: MRL 5: Should be assessed at TRL 5: Special handling procedures applied in production-relevant environment. Special 9Ahandling requirement gaps identified. New special handling processes demonstrated in lab environment; Technology development: MRL 6 (Milestone B): Should be assessed at TRL 6: Special handling procedures applied in production-relevant environment. Plans to address special handling requirement gaps complete; Engineering and manufacturing development: MRL 7: Should be assessed at TRL 7: Special handling procedures applied in production representative environment. Special handling procedures developed and annotated on work instructions; Engineering and manufacturing development: MRL 8 (Milestone C): Should be assessed at TRL 7: Special handling procedures applied in pilot line environment. Special handling procedures demonstrated in EMD or technology insertion programs. Special handling issues pose no significant risk for LRIP. All work instructions contain special handling provisions, as required; Low rate initial product: MRL 9 (Milestone FRP): Should be assessed at TRL 8: Special handling procedures applied in LRIP environment. Special handling procedures demonstrated in LRIP. Special handling issues pose no significant risk for FRP; Full rate product: MRL 10: Should be assessed at TRL 9: Special handling procedures effectively implemented in FRP. Acquisition Phase: Thread: Process capability and control; Acquisition Phase: Subthread: Technology maturity: Modeling and simulation (product and process); Pre-material solution analysis: MRL 1: Should be assessed at TRL 1: [Empty] Pre-material solution analysis: MRL 2: Should be assessed at TRL 2: Initial models developed, if applicable; Pre-material solution analysis: MRL 3: Should be assessed at TRL 3: Identification of proposed manufacturing concepts or producibility needs based on high-level process flowchart models; Material solution analysis: MRL 4 (Milestone A): Should be assessed at TRL 4: Production modeling and simulation approaches for process or product are identified; Technology development: MRL 5: Should be assessed at TRL 5: Initial simulation models (product or process) developed at the component level; Technology development: MRL 6 (Milestone B): Should be assessed at TRL 6: Initial simulation models developed at the subsystem or system level; Engineering and manufacturing development: MRL 7: Should be assessed at TRL 7: Simulation models used to determine system constraints and identify improvement opportunities; Engineering and manufacturing development: MRL 8 (Milestone C): Should be assessed at TRL 7: Simulation models verified by pilot line build. Results used to improve process and determine that LRIP requirements can be met; Low rate initial product: MRL 9 (Milestone FRP): Should be assessed at TRL 8: Simulation model verified by LRIP build, assists in management of LRIP and determines that FRP requirements can be met; Full rate product: MRL 10: Should be assessed at TRL 9: Simulation model verified by FRP build. Production simulation models used as a tool to assist in management of FRP. Acquisition Phase: Thread: Process capability and control; Acquisition Phase: Subthread: Technology maturity: Manufacturing process maturity; Pre-material solution analysis: MRL 1: Should be assessed at TRL 1: [Empty]; Pre-material solution analysis: MRL 2: Should be assessed at TRL 2: Identification of material and/or process approaches; Pre-material solution analysis: MRL 3: Should be assessed at TRL 3: Document high-level manufacturing processes. Critical manufacturing processes identified through experimentation; Material solution analysis: MRL 4 (Milestone A): Should be assessed at TRL 4: Complete a survey to determine the current state of critical processes; Technology development: MRL 5: Should be assessed at TRL 5: Maturity has been assessed on similar processes in production. Process capability requirements have been identified for pilot line, LRIP and FRP; Technology development: MRL 6 (Milestone B): Should be assessed at TRL 6: Manufacturing processes demonstrated in production-relevant environment. Begin collecting or estimating process capability data from prototype build; Engineering and manufacturing development: MRL 7: Should be assessed at TRL 7: Manufacturing processes demonstrated in a production- representative environment. Continue collecting or estimating process capability data; Engineering and manufacturing development: MRL 8 (Milestone C): Should be assessed at TRL 7: Manufacturing processes verified for LRIP on a pilot line. Process capability data from pilot line meets target; Low rate initial product: MRL 9 (Milestone FRP): Should be assessed at TRL 8: Manufacturing processes are stable, adequately controlled, and capable and have achieved program LRIP objectives. Variability experiments conducted to show FRP impact and potential for continuous improvement; Full rate product: MRL 10: Should be assessed at TRL 9: Manufacturing processes are stable, adequately controlled, capable, and have achieved program FRP objectives. Acquisition Phase: Thread: Process capability and control; Acquisition Phase: Subthread: Technology maturity: Process yields and rates; Pre-material solution analysis: MRL 1: Should be assessed at TRL 1: [Empty] Pre-material solution analysis: MRL 2: Should be assessed at TRL 2: [Empty] Pre-material solution analysis: MRL 3: Should be assessed at TRL 3: Initial estimates of yields and rates based on experiments or state of the art; Material solution analysis: MRL 4 (Milestone A): Should be assessed at TRL 4: Yield and rates assessment on proposed/similar processes complete and applied within AoA; Technology development: MRL 5: Should be assessed at TRL 5: Target yields and rates established for pilot line, LRIP, and FRP. Yield and rate issues identified. Improvement plans developed/initiated; Technology development: MRL 6 (Milestone B): Should be assessed at TRL 6: Yields and rates from production-relevant environment evaluated against targets and the results feed improvement plan; Engineering and manufacturing development: MRL 7: Should be assessed at TRL 7: Yields and rates from production-representative environment evaluated against pilot line targets and the results feed improvement plans; Engineering and manufacturing development: MRL 8 (Milestone C): Should be assessed at TRL 7: Pilot line targets achieved. Yields and rates required to begin LRIP verified using pilot line articles. Improvement plans ongoing and updated; Low rate initial product: MRL 9 (Milestone FRP): Should be assessed at TRL 8: LRIP yield and rate targets achieved. Yield improvements ongoing; Full rate product: MRL 10: Should be assessed at TRL 9: FRP yield and rate targets achieved. Yield improvements ongoing. Acquisition Phase: Thread: Quality management; Acquisition Phase: Subthread: Technology maturity: Quality management, including supplier quality; Pre-material solution analysis: MRL 1: Should be assessed at TRL 1: [Empty]; Pre-material solution analysis: MRL 2: Should be assessed at TRL 2: [Empty]; Pre-material solution analysis: MRL 3: Should be assessed at TRL 3: [Empty]; Material solution analysis: MRL 4 (Milestone A): Should be assessed at TRL 4: Quality strategy identified as part of the TDS and included in SEP; Technology development: MRL 5: Should be assessed at TRL 5: Quality strategy updated to reflect KC identification activities; Technology development: MRL 6 (Milestone B): Should be assessed at TRL 6: Initial quality plan and quality management system is in place. Quality risks and metrics have been identified; Engineering and manufacturing development: MRL 7: Should be assessed at TRL 7: Quality targets established. Demonstrate ability to collect and analyze quality data (process and system) in the production- representative environment; Engineering and manufacturing development: MRL 8 (Milestone C): Should be assessed at TRL 7: Quality targets demonstrated on pilot line. Continuous quality improvement ongoing. Supplier products have completed qualification testing and first-article inspection. Supplier products pass acceptance testing at a rate adequate to begin LRIP; Low rate initial product: MRL 9 (Milestone FRP): Should be assessed at TRL 8: Quality targets verified on LRIP line. Continuous quality improvement ongoing. Supplier products pass acceptance testing at a rate adequate to transition to FRP; Full rate product: MRL 10: Should be assessed at TRL 9: Quality targets verified on FRP line. Continuous quality improvement ongoing. Acquisition Phase: Thread: Manufacturing personnel; Acquisition Phase: Subthread: Technology maturity: Manufacturing personnel; Pre-material solution analysis: MRL 1: Should be assessed at TRL 1: [Empty]; Pre-material solution analysis: MRL 2: Should be assessed at TRL 2: [Empty]; Pre-material solution analysis: MRL 3: Should be assessed at TRL 3: New manufacturing skills identified; Material solution analysis: MRL 4 (Milestone A): Should be assessed at TRL 4: Manufacturing skill sets identified and production workforce requirements (technical and operational) evaluated as part of AoA. Determine availability of process development workforce for the TDP; Technology development: MRL 5: Should be assessed at TRL 5: Skill sets identified and plans developed to meet prototype and production needs. Special skills certification and training requirements established; Technology development: MRL 6 (Milestone B): Should be assessed at TRL 6: Manufacturing workforce skills available for production in a relevant environment. Identify resources (quantities and skill sets) and develop initial plans to achieve requirements for pilot line and production; Engineering and manufacturing development: MRL 7: Should be assessed at TRL 7: Manufacturing workforce resource requirements identified for pilot line. Plans developed to achieve pilot line requirements. Plans updated to achieve LRIP workforce requirements. Pilot line workforce trained on representative environment; Engineering and manufacturing development: MRL 8 (Milestone C): Should be assessed at TRL 7: Manufacturing workforce resource requirements identified for LRIP. Plans developed to achieve LRIP requirements. Plans updated to achieve FRP workforce requirements. LRIP personnel trained on pilot line where possible; Low rate initial product: MRL 9 (Milestone FRP): Should be assessed at TRL 8: LRIP personnel requirements met. Implement plan to achieve FRP workforce requirements; Full rate product: MRL 10: Should be assessed at TRL 9: FRP personnel requirements met. Production workforce skill sets maintained due to attrition of workforce. Acquisition Phase: Thread: Facilities; Acquisition Phase: Subthread: Technology maturity: Tooling/STE/SIE; Pre-material solution analysis: MRL 1: Should be assessed at TRL 1: [Empty]; Pre-material solution analysis: MRL 2: Should be assessed at TRL 2: [Empty]; Pre-material solution analysis: MRL 3: Should be assessed at TRL 3: [Empty]; Material solution analysis: MRL 4 (Milestone A): Should be assessed at TRL 4: Tooling/STE/SIE requirements are considered as part of AoA; Technology development: MRL 5: Should be assessed at TRL 5: Identify tooling and Special Test Equipment/Special Inspection Equipment (STE/SIE) requirements and provide supporting rationale and schedule; Technology development: MRL 6 (Milestone B): Should be assessed at TRL 6: Prototype tooling and STE/SIE concepts demonstrated in production relevant environment. Production tooling and STE/SIE requirements developed; Engineering and manufacturing development: MRL 7: Should be assessed at TRL 7: Production tooling and STE/SIE design and development efforts underway. Manufacturing equipment maintenance strategy developed; Engineering and manufacturing development: MRL 8 (Milestone C): Should be assessed at TRL 7: All tooling, test, and inspection equipment proven on pilot line and requirements identified for LRIP. Manufacturing equipment maintenance demonstrated on pilot line; Low rate initial product: MRL 9 (Milestone FRP): Should be assessed at TRL 8: All tooling, test, and inspection equipment proven in LRIP and requirements identified for FRP. Manufacturing equipment maintenance schedule demonstrated; Full rate product: MRL 10: Should be assessed at TRL 9: Proven tooling, test, and inspection equipment in place to support maximum FRP. Planned equipment maintenance schedule achieved. Acquisition Phase: Thread: Facilities; Acquisition Phase: Subthread: Technology maturity: Facilities; Pre-material solution analysis: MRL 1: Should be assessed at TRL 1: [Empty]; Pre-material solution analysis: MRL 2: Should be assessed at TRL 2: [Empty]; Pre-material solution analysis: MRL 3: Should be assessed at TRL 3: Specialized facility requirements/needs identified; Material solution analysis: MRL 4 (Milestone A): Should be assessed at TRL 4: Availability of manufacturing facilities for prototype development and production valuated as part of AoA; Technology development: MRL 5: Should be assessed at TRL 5: Manufacturing facilities identified and plans developed to produce prototypes; Technology development: MRL 6 (Milestone B): Should be assessed at TRL 6: Manufacturing facilities identified and plans developed to produce pilot line build; Engineering and manufacturing development: MRL 7: Should be assessed at TRL 7: Manufacturing facilities identified and plans developed to produce LRIP build; Engineering and manufacturing development: MRL 8 (Milestone C): Should be assessed at TRL 7: Pilot line facilities demonstrated. Manufacturing facilities adequate to begin LRIP. Plans in place to support transition to FRP; Low rate initial product: MRL 9 (Milestone FRP): Should be assessed at TRL 8: Manufacturing facilities in place and demonstrated in LRIP. Capacity plans adequate to support FRP; Full rate product: MRL 10: Should be assessed at TRL 9: Production facilities in place and capacity demonstrated to meet maximum FRP requirements. Acquisition Phase: Thread: Manufacturing management; Acquisition Phase: Subthread: Technology maturity: Manufacturing planning and scheduling; Pre-material solution analysis: MRL 1: Should be assessed at TRL 1: [Empty]; Pre-material solution analysis: MRL 2: Should be assessed at TRL 2: [Empty]; Pre-material solution analysis: MRL 3: Should be assessed at TRL 3: [Empty]; Material solution analysis: MRL 4 (Milestone A): Should be assessed at TRL 4: Manufacturing strategy developed and integrated with acquisition strategy. Prototype schedule risk mitigation efforts incorporated into TDS; Technology development: MRL 5: Should be assessed at TRL 5: Manufacturing strategy refined based upon preferred concept. Prototype schedule risk mitigation efforts initiated; Technology development: MRL 6 (Milestone B): Should be assessed at TRL 6: Initial manufacturing approach developed. All system-design-related manufacturing events included in Integrated Master Plan/Integrated Master Schedule (IMP/IMS). Manufacturing risk mitigation approach for pilot line or technology insertion programs defined; Engineering and manufacturing development: MRL 7: Should be assessed at TRL 7: Initial manufacturing plan developed. Manufacturing planning required to achieve MRL 8 has been included in the IMP/IMS. Manufacturing risks integrated into risk mitigation plans. Develop initial work instructions. Effective production control system in place to support pilot line; Engineering and manufacturing development: MRL 8 (Milestone C): Should be assessed at TRL 7: Manufacturing plan updated for LRIP. All key manufacturing risks are identified and assessed with approved mitigation plans in place. Work instructions finalized. Effective production control system in place to support LRIP; Low rate initial product: MRL 9 (Milestone FRP): Should be assessed at TRL 8: Manufacturing plan updated for FRP. All manufacturing risks tracked and mitigated. Effective production control system in place to support FRP; Full rate product: MRL 10: Should be assessed at TRL 9: All manufacturing risks mitigated. Acquisition Phase: Thread: [Empty]; Acquisition Phase: Subthread: Technology maturity: Materials planning; Pre-material solution analysis: MRL 1: Should be assessed at TRL 1: [Empty]; Pre-material solution analysis: MRL 2: Should be assessed at TRL 2: [Empty]; Pre-material solution analysis: MRL 3: Should be assessed at TRL 3: [Empty]; Material solution analysis: MRL 4 (Milestone A): Should be assessed at TRL 4: Technology development article component list developed with associated lead-time estimates; Technology development: MRL 5: Should be assessed at TRL 5: Technology development part list maturing. Make/buy evaluations begin and include production considerations reflecting pilot line, LRIP, and FRP needs. Lead times and other risks identified; Technology development: MRL 6 (Milestone B): Should be assessed at TRL 6: Most material decisions complete (make/buy), material risks identified, and mitigation plans developed. Bill of Materials (BOM) initiated; Engineering and manufacturing development: MRL 7: Should be assessed at TRL 7: Make/buy decisions and BOM complete for pilot line build. Material planning systems in place for pilot line build; Engineering and manufacturing development: MRL 8 (Milestone C): Should be assessed at TRL 7: Make/buy decisions and BOM complete to support LRIP. Material planning systems in place for LRIP build; Low rate initial product: MRL 9 (Milestone FRP): Should be assessed at TRL 8: Make/buy decisions and BOM complete to support FRP. Material planning systems in place for FRP; Full rate product: MRL 10: Should be assessed at TRL 9: Material planning systems validated on FRP build. Source: GAO. [End of figure] [End of section] Appendix IV: Comments from the Department of Defense: Office Of The Director Of Defense Research And Engineering: 3040 Defense Pentagon: Washington, DC 20301-3040: April 19, 2010: Mr. Michael Sullivan: Director, Acquisition and Sourcing Management: U.S. Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Dear Mr. Sullivan: This is the Department of Defense response to the GAO draft report, GAO-10439, 'Best Practices: DoD Can Achieve Better Outcomes by Standardizing the Way Manufacturing Risks Are Managed' dated March 12, 2010 (GAO Code 120793). Detailed comments on the report recommendations are enclosed. The Department appreciates the opportunity to respond to your draft report and looks forward to working with you as we continue to improve manufacturing readiness of our acquisition programs. Sincerely, Signed by: Stephen P. Welby: Director: Systems Engineering: Enclosure: As stated: [End of letter] DoD Response to GA0-10-439 Recommendations: GAO Draft Report Dated March 12, 2010: GAO-10-439 (GAO Code 120793): "Best Practice: DOD Can Achieve Better Outcomes By Standardizing The Way Manufacturing Risks Are Managed" Department Of Defense Comments To The GAO Recommendations: Recommendation 1: The GAO recommends that the Secretary of Defense require the assessment of manufacturing readiness across DoD programs using consistent MRL criteria as a basis for measuring, assessing, reporting, and communicating manufacturing readiness and risk on science and technology transition projects and acquisition programs. DOD Response: Partially concur: The Department of Defense recognizes that mature manufacturing processes and readiness are critical to achieving predictable and successful program outcomes. It also recognizes the value in assessing manufacturing risks during science and technology research on technologies planned to be incorporated into acquisition programs. Department of Defense Instruction (DoDI) 5000.02, Operation of the Defense Acquisition System, dated 8 December 2008 reflects an increased focus on manufacturing throughout the acquisition lifecycle for programs of all acquisition categories. Specifically, it establishes a framework to continually assess and mitigate manufacturing risks during the Analysis of Alternatives, 2366b certifications to Congress, Preliminary and Critical Design Reviews; and acquisition milestones. The Department's new manufacturing readiness criteria will form the basis for assessing pertinent science and technology efforts, and acquisition programs throughout the acquisition lifecycle on programs of all acquisition categories. These criteria will be a tool to identify relevant manufacturing risks which require mitigation. These manufacturing readiness criteria are expected to be tailored for programs and will be included in the Department's criteria for systems engineering technical reviews; the Department's templates for Preliminary Design Review/Critical Design Review reports; and acquisition phase exit criteria. These manufacturing readiness criteria will also be assessed as part of the Program Support Reviews which the Department conducts on Major Defense Acquisition Programs. These reviews evaluate manufacturing as part of an overall integrated program assessment. These manufacturing readiness criteria and products will be made available to government and industry. Their use by the Services on lower ACAT programs will also be encouraged. The Navy's Gate Review process currently assesses manufacturing risks but is being updated with the new manufacturing readiness criteria. Recommendation 2: The GAO recommends that the Secretary of Defense direct the Office of the Director, Defense Research and Engineering to examine strengthening the MRL criteria related to the process capability and control of critical components and/or interfaces prior to the Milestone C low rate initial production decision. DOD Response: Concur. Department of Defense Instruction 5000.02 directs that programs at Milestone C have no significant manufacturing risks; that manufacturing processes have been effectively demonstrated in a pilot line environment; and manufacturing processes are under control (if Milestone C is full-rate production). While the Department notes that all manufacturing processes do not warrant the same level of process capability and control, appropriate levels of control are certainly warranted on a case by case basis. The Department will examine strengthening the manufacturing readiness criteria related to process capability and control of critical components and/or interfaces prior to the Milestone C low rate initial production decision. However, program offices and contractors should continue to have the latitude to jointly agree on the targets and specific process control demonstrations required on the pilot production line during the Engineering and Manufacturing Development to ensure success. Recommendation 3: The GAO recommends that the Secretary of Defense direct the Office of the Director, Defense Research and Engineering to assess the need for analytical models and tools to support MRL assessments. DOD Response: Concur. The Department will collaborate with government services, contractors, and academia to capture knowledge and provide improved tools for government and contractor usage in conducting assessments of manufacturing readiness as part of systems engineering technical reviews and milestone reviews. Recommendation 4: The GAO recommends that the Secretary of Defense assess the adequacy of the manufacturing workforce knowledge and skills base across the military services and defense agencies and develop a plan to address current and future workforce gaps. DOD Response: Concur. We agree that the production, quality and manufacturing (PQM) career field has suffered erosion, as have other DoD career fields. The USD (AT&L) Director of Human Capital has launched a review of the PQM career field design to identify the skills, knowledge and training required at each level of career progression in order to develop training courses and evaluate progression of anticipated DoD planned new hires. The Department has started to implement hiring and retention strategies to mitigate the potential loss in experienced, senior-level PQM talent and increase the size of the manufacturing workforce. As part of the Secretary's growth strategy and other initiatives, the PQM career field is projected to grow approximately 1,300 (13%) by FY2015. Each of the military services and other DOD components has been actively planning and deploying initiatives that support the DOD acquisition workforce growth strategy. Components have submitted planning inputs to OSD and to the Defense Acquisition Workforce Senior Steering Board, and growth is underway. [End of section] Appendix V: GAO Contact and Staff Acknowledgments: GAO Contact: Michael Sullivan, (202) 512-4841 or sullivanm@gao.gov: Staff Acknowledgments: Key contributors to this report were Karen Zuckerstein, Assistant Director; John M. Ortiz, Jr.; Beverly Breen; Leigh Ann Nally; Dr. W. Kendal Roberts; Andrea Bivens; Kristine Hassinger; Kenneth Patton; Bob Swierczek; and Dr. Timothy Persons, Chief Scientist. [End of section] Related GAO Products: Defense Acquisitions: Assessments of Selected Weapon Programs. [hyperlink, http://www.gao.gov/products/GAO-10-388SP]. Washington, D.C.: March 30, 2010. Defense Acquisitions: Assessments of Selected Weapon Programs. [hyperlink, http://www.gao.gov/products/GAO-09-326SP]. Washington, D.C.: March 30, 2009. Best Practices: Increased Focus on Requirements and Oversight Needed to Improve DOD's Acquisition Environment and Weapon System Quality. [hyperlink, http://www.gao.gov/products/GAO-08-294]. Washington, D.C.: February 1, 2008. Best Practices: Stronger Practices Needed to Improve DOD Technology Transition Processes. [hyperlink, http://www.gao.gov/products/GAO-06-883]. Washington, D.C., September 14, 2006. Defense Acquisitions: Major Weapon Systems Continue to Experience Cost and Schedule Problems under DOD's Revised Policy. [hyperlink, http://www.gao.gov/products/GAO-06-368]. Washington, D.C.: April 13, 2006. Best Practices: Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes. [hyperlink, http://www.gao.gov/products/GAO-02-701]. Washington, D.C.: July 15, 2002. Defense Acquisitions: DOD Faces Challenges in Implementing Best Practices. [hyperlink, http://www.gao.gov/products/GAO-02-469T]. Washington, D.C.: February 27, 2002. Best Practices: DOD Can Help Suppliers Contribute More to Weapon System Programs. [hyperlink, http://www.gao.gov/products/GAO/NSIAD-98-87]. Washington, D.C.: March 17, 1998. Best Practices: Commercial Quality Assurance Practices Offer Improvements for DOD. [hyperlink, http://www.gao.gov/products/GAO/NSIAD-96-162]. Washington, D.C.: August 26, 1996. Why Some Weapon Systems Encounter Production Problems While Others Do Not: Six Case Studies. [hyperlink, http://www.gao.gov/products/GAO/NSIAD-85-34]. Washington, D.C.: May 24, 1985. [End of section] Footnotes: [1] GAO, Defense Acquisitions: Assessments of Selected Weapon Programs, [hyperlink, http://www.gao.gov/products/GAO-09-326SP] (Washington, D.C.: Mar. 30, 2009). [2] Department of Defense Instruction 5000.02, Operation of the Defense Acquisition System (May 12, 2003). [3] Department of Defense Instruction 5000.02, Operation of the Defense Acquisition System (Dec. 8, 2008). (Hereafter cited as DODI 5000.02 (Dec. 8, 2008)). [4] On June 8, 1999, the Joint Defense Manufacturing Technology Panel was chartered by the Office of the Director, Defense Research & Engineering, the military services, and the Defense Logistics Support Command to conduct joint program planning. The Joint Defense Manufacturing Technology Panel develops joint strategies for the Manufacturing Technology programs conducted by the Army, Navy, Air Force, and Defense Logistics Agency (the components). Other duties include, but are not limited to (1) reviews and assessments of defense- related manufacturing issues; (2) annual planning activities with the Office of the Director of Defense Research & Engineering; and (3) information exchange with government agencies, private industry, academia, and professional associations--conducting and supporting an annual Defense Manufacturing Conference. [5] GAO, Best Practices: Commercial Quality Assurance Practices Offer Improvements for DOD, [hyperlink, http://www.gao.gov/products/GAO/NSIAD-96-162] (Washington, D.C.: Aug. 26, 1996). [6] GAO, Best Practices: DOD Can Help Suppliers Contribute More to Weapon System Programs, [hyperlink, http://www.gao.gov/products/GAO/NSIAD-98-87] (Washington, D.C.: Mar. 17, 1998). [7] GAO, Best Practices: Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes, [hyperlink, http://www.gao.gov/products/GAO-02-701] (Washington, D.C.: July 15, 2002). [8] GAO, Best Practices: Increased Focus on Requirements and Oversight Needed to Improve DOD's Acquisition Environment and Weapon System Quality, [hyperlink, http://www.gao.gov/products/GAO-08-294] (Washington, D.C.: Feb. 1, 2008). [9] GAO, Defense Acquisitions: Assessments of Selected Weapon Programs, [hyperlink, http://www.gao.gov/products/GAO-09-326SP] (Washington, D.C.: Mar. 30, 2009). [10] DODI 5000.02 (Dec. 8, 2008). [11] GAO, Defense Acquisitions: Major Weapon Systems Continue to Experience Cost and Schedule Problems under DOD's Revised Policy, [hyperlink, http://www.gao.gov/products/GAO-06-368] (Washington, D.C.: Apr. 13, 2006). [12] The three decision points within the DOD acquisition framework include: milestone A (entry point for the technology development acquisition phase); milestone B (entry point for the engineering and manufacturing development period--which is comprised of two phases called integrated system design, and system capability and manufacturing process demonstration); and milestone C (entry point for the production and deployment acquisition phase). DODI 5000.02 (Dec. 8, 2008), enc. 2, figure 1. [13] GAO, Defense Acquisitions: Major Weapon Systems Continue to Experience Cost and Schedule Problems under DOD's Revised Policy, [hyperlink, http://www.gao.gov/products/GAO-06-368] (Washington, D.C.: Apr. 13, 2006). [14] Major defense acquisition programs are those identified by DOD that require total research, development, test, and evaluation (RDT&E) expenditures, including all planned increments, of more than $365 million or procurement expenditures, including all planned increments, of more than $2.19 billion in fiscal year 2000 constant dollars. [15] In April 9, 2007, the Secretary of the Air Force reported the average procurement unit cost increased more than 50 percent above the initial baseline estimate and that the majority of the cost growth was attributed to a reliability improvement program to deal with the missile's reliability problems, addition of the extended range variant, and reduction in missile quantities. However, the program reported that, even without the quantities associated with the extended range variant, the program would have experienced a significant breach to the original baseline. [16] Automated production processes have been implemented primarily for the sensor, and for other limited activities, but have not been implemented across the entire program. [17] Technical reviews fall under the Systems and Software Engineering organization, within the Office of the Secretary of Defense, where responsibilities for quality and oversight of defense programs reside. [18] The preliminary design review ensures that the system under review has a reasonable expectation of satisfying the requirements within the currently allocated budget and schedule. The review includes evaluation areas and checklist questions. [19] The critical design review ensures that the system under review has a reasonable expectation of satisfying the requirements of the Capability Development Document within the currently allocated budget and schedule. This review assesses the final design as captured in product specifications for each configuration item in the system and ensures that each product specification has been captured in detailed design documentation. [20] The production readiness review examines a program to determine if the design is ready for production and if the prime contractor and major subcontractors have accomplished adequate production planning without incurring unacceptable risks that will breach thresholds of schedule, performance, cost, or other established criteria. The review evaluates the full, production-configured system to determine if it correctly and completely implements all system requirements. [21] An ISO certification against a standard, such as ISO 9000, means that an independent external body has audited an organization's quality-management system and verified that it conforms to the requirements specified in the standard. [22] Process controls are the use of statistical methods to monitor, track, and reduce the variability of manufacturing processes. [23] National Defense Authorization Act for Fiscal Year 2006, Pub. L. No. 109-163, § 801, codified at 10 U.S.C. § 2366b. [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s Web site, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: