Best Practices: A More Constructive Test Approach Is Key to Better Weapon System Outcomes
NSIAD-00-199
Published: Jul 31, 2000. Publicly Released: Jul 31, 2000.
Skip to Highlights
Highlights
Pursuant to a congressional request, GAO provided information on the role of testing and evaluation in product development, focusing on: (1) how the conduct of testing and evaluation affects commercial and Department of Defense (DOD) program outcomes; (2) how best commercial testing and evaluation practices compare with DOD's; and (3) what factors account for the differences in these practices.
Recommendations
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Defense | To lessen the dependence on testing late in development and foster a more constructive relationship between program managers and testers, the Secretary of Defense should instruct the managers and testers of weapon system programs to work together to define levels of product maturity that need to be validated, structure test plans around reaching increasing levels of product maturity, and orchestrate the right mix of tools to validate these levels. Acquisition strategies should then be built and funded to carry out this approach. Such a focus on attaining knowledge, represented by product maturity levels, can guard against the pressures to forego valuable tests to stay on schedule or to hold tests that do not add value to the product. This approach, which creates common ground between testers and product managers in leading commercial firms without compromising independence, still demands that the product or weapon system being matured meet the needs of the customer. |
Closed – Implemented
DOD concurred with this recommendation, and addressed it in the latest revision to DOD Directive 5000.1 and DOD Instruction 5000.2.
|
| Department of Defense | The Secretary of Defense should not let the validation of lower levels of product maturity--individual components or systems in a controlled setting--be deferred to the higher level of system testing in a realistic setting. Although the mix of testing and evaluation tools may change and the acquisition strategy may be altered during the course of a weapon system development, the focus on attaining product maturity levels should not change. This discipline should also help guard against the practice of setting cost and schedule constraints for programs without considering the time and money it takes to sensibly validate maturity. |
Closed – Implemented
DOD concurred with this recommendation, and addressed it in the latest revision to DOD Directive 5000.1 and DOD Instruction 5000.2.
|
| Department of Defense | The Secretary of Defense should require weapon systems to demonstrate a specified level of product maturity before a major programmatic approvals. In doing so, the Secretary may also need to establish interim indicators of product maturity to inform budget requests, which are made well in advance of programmatic decisions. Testing and evaluation could then be cast in a more constructive role of helping a weapon system reach these levels and would ease some of the burden currently placed on program managers to rely on judgment, rather than demonstrated product maturity, in promising success at times when major funding commitments have to be made. |
Closed – Not Implemented
DOD did not concur with this recommendation, and has not taken any action since the report was issued in fiscal year 2000.
|
Full Report
Office of Public Affairs
Topics
Best practicesComparative analysisCost effectiveness analysisDefense cost controlDefense procurementOperational testingPrivate sector practicesProcurement planningWeapons research and developmentWeapons systemsComparative benchmarking products