Best Practices:

Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes

GAO-02-701: Published: Jul 15, 2002. Publicly Released: Jul 15, 2002.

Additional Materials:

Contact:

Michael J. Sullivan
(202) 512-4841
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

This report examines how best practices offer improvements to the way the Department of Defense (DOD) develops new weapons systems, primarily the design and manufacturing aspects of the acquisition process. Knowledge about a product's design and producibility facilitates informed decisions about whether to significantly increase investments and reduces the risk of costly design changes later in the program. Leading commercial companies employ practices to capture design and manufacturing knowledge in time to make key decisions during product development. First, the companies kept the degree of the design challenge manageable before starting a new product development program by using an evolutionary approach. Second, the companies captured design and manufacturing knowledge before the two critical decision points in product development: when the design was demonstrated to be stable--the second knowledge point--and when the product was demonstrated to be producible at an affordable cost--the third knowledge point. DOD has made changes to its acquisition policy in an attempt to improve its framework for developing weapons systems, but the policy does not require the capture of design or manufacturing knowledge or sufficient criteria to enter the system demonstration and production phases. In addition, it does not require a decision review to enter the demonstration phase of product development.

Status Legend:

More Info
  • Review Pending-GAO has not yet assessed implementation status.
  • Open-Actions to satisfy the intent of the recommendation have not been taken or are being planned, or actions that partially satisfy the intent of the recommendation have been taken.
  • Closed-implemented-Actions that satisfy the intent of the recommendation have been taken.
  • Closed-not implemented-While the intent of the recommendation has not been satisfied, time or circumstances have rendered the recommendation invalid.
    • Review Pending
    • Open
    • Closed - implemented
    • Closed - not implemented

    Recommendations for Executive Action

    Recommendation: DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require the capture of specific knowledge to be used as exit criteria for decision making at two key points--when transitioning from system integration to system demonstration and from system demonstration into production. The knowledge to be captured when moving from system integration to system demonstration should include completed subsystem and system design reviews.

    Agency Affected: Department of Defense

    Status: Closed - Implemented

    Comments: In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is a design readiness review to assess design maturity mid-way during System Development and Demonstration similar to best practices. Successful completion of the review ends system integration and continues System Development and Demonstration into system demonstration. While not required, the capture of design knowledge in the form of completed subsystem and system design reviews was cited as evidence that could be used for decisionmaking.

    Recommendation: DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require the capture of specific knowledge to be used as exit criteria for decision making at two key points--when transitioning from system integration to system demonstration and from system demonstration into production. The knowledge to be captured when moving from system integration to system demonstration should include ninety percent of drawings completed.

    Agency Affected: Department of Defense

    Status: Closed - Implemented

    Comments: In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is a design readiness review to assess design maturity mid-way during System Development and Demonstration similar to best practices. Successful completion of the review ends system integration and continues System Development and Demonstration into system demonstration. While not required, the capture of design knowledge in the form of completed engineering drawings was cited as evidence that could be used for decisionmaking.

    Recommendation: DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require the capture of specific knowledge to be used as exit criteria for decision making at two key points--when transitioning from system integration to system demonstration and from system demonstration into production. The knowledge to be captured when moving from system integration to system demonstration should include demonstration that design meets requirements--prototype or variant testing.

    Agency Affected: Department of Defense

    Status: Closed - Implemented

    Comments: In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is a design readiness review to assess design maturity mid-way during System Development and Demonstration similar to best practices. Criteria for transitioning from system integration to system demonstration includes demonstrating the design with prototypes.

    Recommendation: DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require the capture of specific knowledge to be used as exit criteria for decision making at two key points--when transitioning from system integration to system demonstration and from system demonstration into production. The knowledge to be captured when moving from system integration to system demonstration should include stakeholders' (cross functional design team that includes design engineers, manufacturing, key supplier) assurance that drawings are complete.

    Agency Affected: Department of Defense

    Status: Closed - Implemented

    Comments: In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is a design readiness review to assess design maturity mid-way during System Development and Demonstration similar to best practices. Successful completion of the review ends system integration and continues System Development and Demonstration into system demonstration. The policy does not include a requirement that stakeholders have assurance that drawings are complete. However, the capture of design knowledge in the form of completed drawings was cited as evidence that could be used for decisionmaking.

    Recommendation: DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require the capture of specific knowledge to be used as exit criteria for decision making at two key points--when transitioning from system integration to system demonstration and from system demonstration into production. The knowledge to be captured when moving from system integration to system demonstration should include completed failure modes and effects analysis.

    Agency Affected: Department of Defense

    Status: Closed - Implemented

    Comments: In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is a design readiness review to assess design maturity mid-way during System Development and Demonstration similar to best practices. Successful completion of the review ends system integration and continues System Development and Demonstration into system demonstration. While not required, the capture of design knowledge in the form of completed failure modes and effects analysis was cited as evidence that could be used for decisionmaking.

    Recommendation: DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require the capture of specific knowledge to be used as exit criteria for decision making at two key points--when transitioning from system integration to system demonstration and from system demonstration into production. The knowledge to be captured when moving from system integration to system demonstration should include identification of key system characteristics.

    Agency Affected: Department of Defense

    Status: Closed - Implemented

    Comments: In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is a design readiness review to assess design maturity mid-way during System Development and Demonstration similar to best practices. Successful completion of the review ends system integration and continues System Development and Demonstration into system demonstration. While not required, the capture of design knowledge in the form of the identification of key system characteristics was cited as evidence that could be used for decisionmaking.

    Recommendation: DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require the capture of specific knowledge to be used as exit criteria for decision making at two key points--when transitioning from system integration to system demonstration and from system demonstration into production. The knowledge to be captured when moving from system integration to system demonstration should include identification of critical manufacturing processes.

    Agency Affected: Department of Defense

    Status: Closed - Implemented

    Comments: In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is a design readiness review to assess design maturity mid-way during System Development and Demonstration similar to best practices. Successful completion of the review ends system integration and continues System Development and Demonstration into system demonstration. While not required, the capture of design knowledge in the form of the identification of critical manufacturing processes was cited as evidence that could be used for decisionmaking.

    Recommendation: DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require the capture of specific knowledge to be used as exit criteria for decision making at two key points--when transitioning from system integration to system demonstration and from system demonstration into production. The knowledge to be captured when moving from system demonstration into production should include set reliability targets and growth plan.

    Agency Affected: Department of Defense

    Status: Closed - Implemented

    Comments: In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is the intent of a knowledge based acquisition process where managers should provide knowledge about key aspects of a system at key points in the acquisition process, which includes reducing manufacturing risk and demonstrating producibility prior to full rate production. According to the policy, available knowledge to support approval into this phase includes demonstration that product reliability is acceptable.

    Recommendation: DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require the capture of specific knowledge to be used as exit criteria for decision making at two key points--when transitioning from system integration to system demonstration and from system demonstration into production. The knowledge to be captured when moving from system demonstration into production should include demonstrated manufacturing processes.

    Agency Affected: Department of Defense

    Status: Closed - Implemented

    Comments: In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is the intent of a knowledge based acquisition process where managers should provide knowledge about key aspects of a system at key points in the acquisition process, which includes reducing manufacturing risk and demonstrating producibility prior to full rate production. According to the policy, available knowledge to support approval into this phase includes the demonstrated control of manufacturing processes.

    Recommendation: DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require the capture of specific knowledge to be used as exit criteria for decision making at two key points--when transitioning from system integration to system demonstration and from system demonstration into production. The knowledge to be captured when moving from system demonstration into production should include built production representative prototypes.

    Agency Affected: Department of Defense

    Status: Closed - Implemented

    Comments: In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is the intent of a knowledge based acquisition process where managers should provide knowledge about key aspects of a system at key points in the acquisition process, which includes reducing manufacturing risk and demonstrating producibility prior to full rate production. The policy includes entrance criteria into production of acceptable performance in test and evaluation with prototypes systems.

    Recommendation: DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require the capture of specific knowledge to be used as exit criteria for decision making at two key points--when transitioning from system integration to system demonstration and from system demonstration into production. The knowledge to be captured when moving from system demonstration into production should include tested prototypes to achieve reliability goal.

    Agency Affected: Department of Defense

    Status: Closed - Implemented

    Comments: In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is the intent of a knowledge based acquisition process where managers should provide knowledge about key aspects of a system at key points in the acquisition process, which includes reducing manufacturing risk and demonstrating producibility prior to full rate production. According to the policy, available knowledge to support approval into this phase includes demonstration that product reliability is acceptable. The policy also includes entrance criteria into production of acceptable performance in test and evaluation with prototype systems.

    Recommendation: DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require the capture of specific knowledge to be used as exit criteria for decision making at two key points--when transitioning from system integration to system demonstration and from system demonstration into production. The knowledge to be captured when moving from system demonstration into production should include tested prototypes to demonstrate product in operational environment.

    Agency Affected: Department of Defense

    Status: Closed - Implemented

    Comments: In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is the intent of a knowledge based acquisition process where managers should provide knowledge about key aspects of a system at key points in the acquisition process this includes reducing manufacturing risk and demonstrating producibility prior to full rate production. According to the policy, available knowledge to support approval into this phase includes acceptable performance in development, test and evaluation, and an operational assessment.

    Recommendation: DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require the capture of specific knowledge to be used as exit criteria for decision making at two key points--when transitioning from system integration to system demonstration and from system demonstration into production. The knowledge to be captured when moving from system demonstration into production should include collected statistical process control data.

    Agency Affected: Department of Defense

    Status: Closed - Implemented

    Comments: In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is the intent of a knowledge based acquisition process where managers should provide knowledge about key aspects of a system at key points in the acquisition process. This includes reducing manufacturing risk and demonstrating producibility prior to full rate production. According to the policy, available knowledge to support approval into this phase includes demonstrated control of the manufacturing process and the collection of statistical process control data.

    Recommendation: DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require the capture of specific knowledge to be used as exit criteria for decision making at two key points--when transitioning from system integration to system demonstration and from system demonstration into production. The knowledge to be captured when moving from system demonstration into production should include demonstration that critical processes are capable and in control.

    Agency Affected: Department of Defense

    Status: Closed - Implemented

    Comments: In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is the intent of a knowledge based acquisition process where managers should provide knowledge about key aspects of a system at key points in the acquisition process. This includes reducing manufacturing risk and demonstrating producibility prior to full rate production. According to the policy, available knowledge to support approval into this phase includes demonstrated control of the manufacturing process and the collection of statistical process control data.

    Recommendation: DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require that the interim process review, currently identified in DOD's policy as that point in the process between system integration and system demonstration, be a mandatory decision review. At this point, the design should be demonstrated to be stable so that during the next phase of development attention can be focused on demonstrating manufacturing processes and product reliability. The program manager should have proof--based on the exit criteria for moving out of system integration in the above recommendation--that the product design is stable. The exit criteria should be demonstrated and verified by the program manager before the program can make the substantial investments needed to begin manufacturing production representative prototypes in the next phase of development--system demonstration. To ensure visibility of demonstrated exit criteria to decision makers, the criteria and the program's status in achieving them should be included in each program's Defense Acquisition Executive Summary and Selected Acquisition Reports. If the program does not meet the exit criteria, investments should be delayed until such time as the criteria are satisfied. To proceed without completing the required demonstrations should require approval by the decision authority.

    Agency Affected: Department of Defense

    Status: Closed - Not Implemented

    Comments: This is now over 4 years old and has not been implemented. We have made similar recommendations in two subsequent reports, GAO-04-53 and GAO-06-368, and will use those reports to follow up on this recommendation.

    Recommendation: DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should expand exit criteria for the Milestone C decision to include the knowledge to be captured during the system demonstration phase as one identified in recommendation one. This will require that the program office demonstrate that the critical manufacturing processes are under statistical control and that product reliability has been demonstrated before entering production of the new weapon system. These are the best practices and indicate that the product design is mature and the program is ready to begin production of units for operational use that will meet the cost, schedule, and quality goals of the program.

    Agency Affected: Department of Defense

    Status: Closed - Not Implemented

    Comments: In commenting on GAO's report, DOD agreed that demonstration of critical manufacturing processes and reliability must occur prior to rate production. However, they stated that it is unlikely that the criteria could be satisfied by their milestone C decision (limited production). In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is the intent of a knowledge based acquisition process where managers should provide knowledge about key aspects of a system at key points in the acquisition process, which includes prior to full rate production. The available knowledge to support approval includes demonstrated control of manufacturing processes and acceptable reliability, and the collection of statistical control data. Best practices suggest that the knowledge should be captured by production commitment, which is milestone C in DOD's process. While policy states limited production should be limited this is frequently not the case in DOD.

    Recommendation: To ensure that contracts support a knowledge-based process, the Department of Defense should structure its contracts for major weapon system acquisitions so that (a) the capture and use of knowledge described in recommendation one for beginning system demonstration is a basis for DOD's decision to invest in the manufacturing capability to build initial prototypes and (b) the capture and use of manufacturing and reliability knowledge discussed in recommendation one for moving from system demonstration to production is a basis for DOD's decision to invest in production.

    Agency Affected: Department of Defense

    Status: Closed - Not Implemented

    Comments: This is now over 4 years old and has not been implemented. We have made similar recommendations in two subsequent reports, GAO-04-53 and GAO-06-368, and will use those reports to follow up on this recommendation.

    Apr 18, 2014

    Apr 16, 2014

    Apr 11, 2014

    Apr 10, 2014

    Apr 9, 2014

    Apr 8, 2014

    Apr 3, 2014

    Looking for more? Browse all our products here