Management Controls Are Critical to Effective Year 2000 Testing
AIMD-99-172: Published: Jun 30, 1999. Publicly Released: Jun 30, 1999.
- Full Report:
Pursuant to a congressional request, GAO reviewed the Department of Defense's (DOD) efforts to integrate and coordinate its various year 2000 end-to-end test activities, focusing on whether: (1) DOD's plans recognize relationships and dependencies among these test and evaluation activities; and (2) DOD has established the management controls to ensure that its various year 2000 end-to-end test and evaluation activities are effectively integrated.
GAO noted that: (1) DOD's end-to-end test and evaluation plans that were available at the time of GAO's review recognize relationships and dependencies among various end-to-end test and evaluation activities; (2) GAO also found that the Office of the Secretary of Defense (OSD) and the Joint Chiefs of Staff (JCS), in order to integrate its various year 2000 end-to-end test activities, are establishing test and evaluation management controls that are consistent with the end-to-end test management controls specified in GAO's year 2000 test guide; (3) both OSD and JCS subsequently issued guidance to the military services, Defense agencies and activities, and the Commanders in Chiefs specifying how these respective year 2000 test and evaluation activities were to be planned, executed, and reported; (4) further, JCS and OSD have established data bases to collect specified data on the respective end-to-end test and evaluation activities; (5) OSD has also established a year 2000 test and evaluation function to independently evaluate end-to-end test and evaluation results; (6) to do this, the designated test director is in the process of defining an assurance-based approach and metrics for measuring the confidence that can be attached to specific test event results; (7) however, this approach and associated metrics have yet to be established, and little time remains for doing so; (8) with less than 7 months remaining before the year 2000 deadline, DOD cannot afford major slippages in its test and evaluation schedule nor does it have the luxury of redoing tests that prove ineffective or incomplete; (9) according to DOD, 245 of DOD's 2,038 mission-critical systems--some of which are needed to execute tests and evaluation activities--are not yet year 2000 compliant, and thus many require invocation of system contingency plans as part of the test and evaluation event; (10) with so little time remaining for DOD's many organizational components to conduct hundreds of related end-to-end test events, it will be important that end-to-end test and evaluation events are well-managed; and (11) DOD must ensure that its established controls are effectively implemented for each test event.
Recommendation for Executive Action
Status: Closed - Implemented
Comments: OSD has established a quality assurance function under the Y2K Test Director for assuring the quality of and its confidence in its end-to-end test activities. The function leverages off of and integrates several existing activities, such as DOD/IG independent audits and military service test agencies' reviews of test results, use of computer industry testing models and scanning tools, and military service and Defense agency use of independent verification and validation agents. OSD has also emphasized the importance of ensuring that end-to-end test and evaluation management controls are implemented and enforced during such forums as DOD Executive-Service Y2K Principals' meetings, Y2K synchronization meetings, Y2K Steering Committee meetings, and by disseminating related policy guidance, and has reiterated that deviations from controls are to be evaluated to determine the reason and impact on testing quality and disclosed through existing Y2K reporting mechanisms.
Recommendation: The Secretary of Defense should: (1) direct the Assistant Secretary for Command, Control, Communications, and Intelligence to immediately implement a quality assurance program for end-to-end test and evaluation activities under the newly designated year 2000 Test Director to provide independent evaluations of test event results; and (2) reiterate to the OSD, JCS, and military service end-to-end testing principals the importance of ensuring that established end-to-end test and evaluation management controls are implemented and enforced on their respective end-to-end events, and that deviations from these controls be discussed through year 2000 reporting mechanisms.
Agency Affected: Department of Defense