Test and Evaluation:

DOD Has Been Slow In Improving Testing of Software-Intensive Systems

NSIAD-93-198: Published: Sep 29, 1993. Publicly Released: Sep 29, 1993.

Additional Materials:

Contact:

Louis J. Rodrigues
(202) 512-4841
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

GAO reviewed the Department of Defense's (DOD) efforts to improve its software acquisition processes and test and evaluation of software-intensive systems, focusing on: (1) how software-related problems affect defense acquisition programs during operational test and evaluation (OT&E); and (2) barriers that limit OT&E effectiveness.

GAO found that: (1) DOD software-intensive systems often fail to meet their performance expectations, experience cost overruns and schedule delays, and are not ready for OT&E because they are not subject to rigorous developmental testing and evaluation and software development is often not fully mature when OT&E begins; (2) DOD has not adequately addressed barriers that inhibit software acquisition and OT&E effectiveness; (3) DOD has not adequately addressed the importance of software development, lacks a consistent policy that defines and coordinates software maturity and operational testing readiness, has not adequately defined and managed software requirements, and lacks reliable cost, schedule, and performance data; (4) DOD has not implemented previous recommendations to ensure that quality software is adequately developed and tested and software-intensive systems are effective; (5) although the military services have tried to improve their software development processes, they lack a DOD-wide coordinated strategy; and (6) the creation of a single executive-level office for software development and testing would help to resolve long-standing software problems.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Comments: The Director, Operational Test and Evaluation, issued a policy memorandum on May 31, 1994, detailing changes in DOD's software OT&E policies to address GAO's recommendations. According to DOD officials, the requirements of these guidance memoranda have been incorporated in Test and Evaluation Master Plans (TEMPS). Formal implementation of the new guidance is included in the revised DOD Regulation 5000.2-R. Although the DOD regulation does not define testing requirements for software maturity or regression testing requirements, the effects of other requirements (i.e., certification for test readiness) should compensate. The Director, Development Test and Evaluation, also issued an automated metrics guidance tool, "Metrics Guided Maturity," dated June 1996, for use by the program offices.

    Recommendation: To realize lasting improvements in test and evaluation of software-intensive systems and to enhance the life-cycle affordability of such programs, the Secretary of Defense should issue and implement a software test and evaluation policy that defines testing requirements for software maturity, regression testing, and the use of temporary software fixes during testing.

    Agency Affected: Department of Defense

  2. Status: Closed - Implemented

    Comments: The Director, Operational Test and Evaluation, issued a policy memorandum on May 31, 1994, detailing changes in DOD's software OT&E policies to address GAO's recommendations. According to DOD officials, the requirements of these guidance memoranda have been incorporated in TEMPS. Formal implementation of the new guidance is included in the revised DOD Regulation 5000.2-R. Although the guidance requires the DT agency to certify the systems as ready for OT but does not require the OT authority to define OT readiness, the implementation of this guidance is fully responsive to the recommendations. The Director, Development Test and Evaluation, also issued an automated metrics guidance tool, "Metrics Guided Maturity," dated June 1996, for use by the program offices.

    Recommendation: To realize lasting improvements in test and evaluation of software-intensive systems and to enhance the life-cycle affordability of such programs, the Secretary of Defense should strengthen controls to ensure that operational testing does not begin until results of development test and evaluation demonstrate an appropriate level of software maturity.

    Agency Affected: Department of Defense

  3. Status: Closed - Implemented

    Comments: The Director, Operational Test and Evaluation, issued a policy memorandum on May 31, 1994, detailing changes in DOD's software OT&E policies to address GAO's recommendations. According to DOD officials, the requirements of these guidance memoranda have been incorporated in TEMPS. Formal implementation of the new guidance is included in the revised DOD Regulation 5000.2-R. Although the guidance requires the DT agency to provide maturity and performance exit criteria necessary to certify OT readiness but does not specifically specify Milestone II, the implementation of this guidance is fully responsive to the recommendations. The Director, Development Test and Evaluation, also issued an automated metrics guidance tool, "Metrics Guided Maturity," dated June 1996, for use by the program offices.

    Recommendation: To realize lasting improvements in test and evaluation of software-intensive systems and to enhance the life-cycle affordability of such programs, the Secretary of Defense should require program management officials to define exit criteria for certifying a systems' readiness for operational testing at the beginning of full-scale development (i.e., milestone II).

    Agency Affected: Department of Defense

  4. Status: Closed - Implemented

    Comments: The Director, Operational Test and Evaluation, issued a policy memorandum on May 31, 1994, detailing changes in DOD's software OT&E policies to address GAO's recommendations. According to DOD officials, the requirements of these guidance memoranda have been incorporated in TEMPS. Formal implementation of the new guidance is included in the revised DOD Regulation 5000.2-R. The guidance requires the DT agency to use a core set of metrics and requires an impact statement for those metrics that are unmet before certification for OT readiness, but does not require a core set of metrics across the services. This guidance is fully responsive to the recommendations. The Director, Development Test and Evaluation, also issued an automated metrics guidance tool, "Metrics Guided Maturity," dated June 1996, for use by the program offices.

    Recommendation: To realize lasting improvements in test and evaluation of software-intensive systems and to enhance the life-cycle affordability of such programs, the Secretary of Defense should require the services to develop a common core set of management metrics for software (i.e., cost, schedule, and quality) for major defense programs early in the development cycle to be approved at milestone II.

    Agency Affected: Department of Defense

 

Explore the full database of GAO's Open Recommendations »

Sep 26, 2016

Sep 15, 2016

Sep 14, 2016

Sep 8, 2016

Jun 29, 2016

Jun 22, 2016

Jun 10, 2016

Jun 9, 2016

Jun 2, 2016

May 25, 2016

Looking for more? Browse all our products here