Test and Evaluation: DOD Has Been Slow In Improving Testing of Software-Intensive Systems
NSIAD-93-198
Published: Sep 29, 1993. Publicly Released: Sep 29, 1993.
Skip to Highlights
Highlights
GAO reviewed the Department of Defense's (DOD) efforts to improve its software acquisition processes and test and evaluation of software-intensive systems, focusing on: (1) how software-related problems affect defense acquisition programs during operational test and evaluation (OT&E); and (2) barriers that limit OT&E effectiveness.
Recommendations
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Defense | To realize lasting improvements in test and evaluation of software-intensive systems and to enhance the life-cycle affordability of such programs, the Secretary of Defense should issue and implement a software test and evaluation policy that defines testing requirements for software maturity, regression testing, and the use of temporary software fixes during testing. |
Closed – Implemented
The Director, Operational Test and Evaluation, issued a policy memorandum on May 31, 1994, detailing changes in DOD's software OT&E policies to address GAO's recommendations. According to DOD officials, the requirements of these guidance memoranda have been incorporated in Test and Evaluation Master Plans (TEMPS). Formal implementation of the new guidance is included in the revised DOD Regulation 5000.2-R. Although the DOD regulation does not define testing requirements for software maturity or regression testing requirements, the effects of other requirements (i.e., certification for test readiness) should compensate. The Director, Development Test and Evaluation, also issued an automated metrics guidance tool, "Metrics Guided Maturity," dated June 1996, for use by the program offices.
|
| Department of Defense | To realize lasting improvements in test and evaluation of software-intensive systems and to enhance the life-cycle affordability of such programs, the Secretary of Defense should strengthen controls to ensure that operational testing does not begin until results of development test and evaluation demonstrate an appropriate level of software maturity. |
Closed – Implemented
The Director, Operational Test and Evaluation, issued a policy memorandum on May 31, 1994, detailing changes in DOD's software OT&E policies to address GAO's recommendations. According to DOD officials, the requirements of these guidance memoranda have been incorporated in TEMPS. Formal implementation of the new guidance is included in the revised DOD Regulation 5000.2-R. Although the guidance requires the DT agency to certify the systems as ready for OT but does not require the OT authority to define OT readiness, the implementation of this guidance is fully responsive to the recommendations. The Director, Development Test and Evaluation, also issued an automated metrics guidance tool, "Metrics Guided Maturity," dated June 1996, for use by the program offices.
|
| Department of Defense | To realize lasting improvements in test and evaluation of software-intensive systems and to enhance the life-cycle affordability of such programs, the Secretary of Defense should require program management officials to define exit criteria for certifying a systems' readiness for operational testing at the beginning of full-scale development (i.e., milestone II). |
Closed – Implemented
The Director, Operational Test and Evaluation, issued a policy memorandum on May 31, 1994, detailing changes in DOD's software OT&E policies to address GAO's recommendations. According to DOD officials, the requirements of these guidance memoranda have been incorporated in TEMPS. Formal implementation of the new guidance is included in the revised DOD Regulation 5000.2-R. Although the guidance requires the DT agency to provide maturity and performance exit criteria necessary to certify OT readiness but does not specifically specify Milestone II, the implementation of this guidance is fully responsive to the recommendations. The Director, Development Test and Evaluation, also issued an automated metrics guidance tool, "Metrics Guided Maturity," dated June 1996, for use by the program offices.
|
| Department of Defense | To realize lasting improvements in test and evaluation of software-intensive systems and to enhance the life-cycle affordability of such programs, the Secretary of Defense should require the services to develop a common core set of management metrics for software (i.e., cost, schedule, and quality) for major defense programs early in the development cycle to be approved at milestone II. |
Closed – Implemented
The Director, Operational Test and Evaluation, issued a policy memorandum on May 31, 1994, detailing changes in DOD's software OT&E policies to address GAO's recommendations. According to DOD officials, the requirements of these guidance memoranda have been incorporated in TEMPS. Formal implementation of the new guidance is included in the revised DOD Regulation 5000.2-R. The guidance requires the DT agency to use a core set of metrics and requires an impact statement for those metrics that are unmet before certification for OT readiness, but does not require a core set of metrics across the services. This guidance is fully responsive to the recommendations. The Director, Development Test and Evaluation, also issued an automated metrics guidance tool, "Metrics Guided Maturity," dated June 1996, for use by the program offices.
|
Full Report
Office of Public Affairs
Topics
Benchmark testingSoftware verification and validationDefense procurementOperational testingProcurement policyProduct evaluationRequirements definitionSystems designDevelopmental testingMilitary forces