Pursuant to a congressional request, GAO provided information on the role of testing and evaluation in product development, focusing on: (1) how the conduct of testing and evaluation affects commercial and Department of Defense (DOD) program outcomes; (2) how best commercial testing and evaluation practices compare with DOD's; and (3) what factors account for the differences in these practices.
Recommendations for Executive Action
|Department of Defense||1. To lessen the dependence on testing late in development and foster a more constructive relationship between program managers and testers, the Secretary of Defense should instruct the managers and testers of weapon system programs to work together to define levels of product maturity that need to be validated, structure test plans around reaching increasing levels of product maturity, and orchestrate the right mix of tools to validate these levels. Acquisition strategies should then be built and funded to carry out this approach. Such a focus on attaining knowledge, represented by product maturity levels, can guard against the pressures to forego valuable tests to stay on schedule or to hold tests that do not add value to the product. This approach, which creates common ground between testers and product managers in leading commercial firms without compromising independence, still demands that the product or weapon system being matured meet the needs of the customer.|
|Department of Defense||2. The Secretary of Defense should not let the validation of lower levels of product maturity--individual components or systems in a controlled setting--be deferred to the higher level of system testing in a realistic setting. Although the mix of testing and evaluation tools may change and the acquisition strategy may be altered during the course of a weapon system development, the focus on attaining product maturity levels should not change. This discipline should also help guard against the practice of setting cost and schedule constraints for programs without considering the time and money it takes to sensibly validate maturity.|
|Department of Defense||3. The Secretary of Defense should require weapon systems to demonstrate a specified level of product maturity before a major programmatic approvals. In doing so, the Secretary may also need to establish interim indicators of product maturity to inform budget requests, which are made well in advance of programmatic decisions. Testing and evaluation could then be cast in a more constructive role of helping a weapon system reach these levels and would ease some of the burden currently placed on program managers to rely on judgment, rather than demonstrated product maturity, in promising success at times when major funding commitments have to be made.|