Test and Evaluation:

Impact of DOD's Office of the Director of Operational Test and Evaluation

NSIAD-98-22: Published: Oct 24, 1997. Publicly Released: Oct 24, 1997.

Contact:

Nancy R. Kingsbury
(202) 512-3652
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

Pursuant to a congressional request, GAO reviewed the Department of Defense's (DOD) Office of the Director of Operational Test and Evaluation's (DOT&E) operations and organizational structure for overseeing operational testing, focusing on: (1) DOT&E's efforts and their impact on the quality of operational testing and evaluation in DOD; and (2) the strengths and weaknesses of the current organizational framework in DOD for operational testing.

GAO noted that: (1) GAO's review of 13 case studies indicated that DOT&E oversight of operational testing and evaluation increased the probability that testing would be more realistic and more thorough; (2) specifically, DOT&E was influential in advocating increasing the reliability of the observed performance and reducing the risk of unknowns through more thorough testing, conducting more realistic testing, enhancing data collection and analysis, reporting independent findings, and recommending follow-on operational test and evaluation when suitability or effectiveness was not fully demonstrated prior to initiating full-rate production; (3) the independence of DOT&E--and its resulting authority to report directly to Congress--is the foundation of its effectiveness; (4) that independence, along with its legislative mandate, provides sufficient freedom and authority to exercise effective oversight of the operational testing and evaluation of new systems before a decision is made to begin full-rate production; (5) DOT&E can reduce the risk that systems are not adequately tested prior to the full-rate production decision but DOT&E cannot ensure that: (a) only systems whose operational effectiveness and suitability have been demonstrated through operational testing will proceed to the full-rate production decision; or (b) new fielded systems will accomplish their missions as intended or that the fielded systems are safe, survivable, and effective; (6) DOT&E management must balance its oversight responsibilities for operational testing with the broader acquisition priorities of program managers and service test agencies; (7) though supportive of DOT&E's mission and independence, program and service representatives frequently considered the time, expense, and resources expended to accommodate DOT&E concerns to be ill-advised; (8) several current trends may challenge DOT&E's ability to manage its workload and its ability to impact operational test and evaluation: (a) service challenges to DOT&E's authority to require and oversee follow-on operational testing and evaluation; (b) a decline in resources available for oversight; (c) an expansion of DOT&E involvement in activities other than oversight of major acquisition programs; (d) participation of DOT&E in the acquisition process as a member of working-level integrated product teams; and (e) greater integration of developmental and operational testing; and (9) these trends make it imperative that DOT&E prioritize its workload to achieve a balance between the oversight of major defense acquisition programs and other initiatives important to the quality of operational test and evaluation.

Status Legend:

More Info
  • Review Pending-GAO has not yet assessed implementation status.
  • Open-Actions to satisfy the intent of the recommendation have not been taken or are being planned, or actions that partially satisfy the intent of the recommendation have been taken.
  • Closed-implemented-Actions that satisfy the intent of the recommendation have been taken.
  • Closed-not implemented-While the intent of the recommendation has not been satisfied, time or circumstances have rendered the recommendation invalid.
    • Review Pending
    • Open
    • Closed - implemented
    • Closed - not implemented

    Recommendations for Executive Action

    Recommendation: The Secretary of Defense should revise DOD's operational test and evaluation policies by requiring the Under Secretary of Defense for Acquisition and Technology, in those cases where affirmative full-rate production decisions are made for major systems that have yet to demonstrate their operational effectiveness or suitability, to: (1) take corrective actions to eliminate deficiencies in effectiveness or suitability; and (2) conduct follow-on test and evaluation of corrective actions until the systems are determined to be operationally effective and suitable by the DOT&E.

    Agency Affected: Department of Defense

    Status: Closed - Implemented

    Comments: DOD Instruction 5000.2R -- Operation of the Defense Acquisition System -- was updated in January 2001 and included a new section incorporating GAO's recommendation. Specifically, DOD 5000.2-R C3.6.2.3 states: DOT&E-oversight programs beyond LRIP, shall require continued DOT&E test plan approval, monitoring, and Follow-On operational Test and Evaluation (FOT&E) reporting to complete IOT&E activity; to refine IOT&E estimates; to verify correction of deficiencies; to evaluate significant changes to system design or employment; and to evaluate whether or not the system continues to meet operational needs and retain operational effectiveness in a substantially new environment, as appropriate.

    Recommendation: The Secretary of Defense should revise DOD's operational test and evaluation policies by requiring the DOT&E, to: (1) review and approve follow-on test and evaluation master plans and specific operational test plans for major systems before operational testing related to suitability and effectiveness issues left unresolved at the full-rate production decision; and (2) upon the completion of follow-on operational test and evaluation, report to the Congress, the Secretary of Defense, and the Under Secretary of Defense for Acquisition and Technology whether the testing was adequate and whether the results confirmed the system is operationally suitable and effective.

    Agency Affected: Department of Defense

    Status: Closed - Implemented

    Comments: DOD Instruction 5000.2R -- Operation of the Defense Acquisition System -- was updated in January 2001 and included a new section incorporating GAO's recommendation. Specifically, DOD 5000.2-R C3.6.2.3 states: DOT&E-oversight programs beyond LRIP, shall require continued DOT&E test plan approval, monitoring, and Follow-On operational Test and Evaluation (FOT&E) reporting to complete IOT&E activity; to refine IOT&E estimates; to verify correction of deficiencies; to evaluate significant changes to system design or employment; and to evaluate whether or not the system continues to meet operational needs and retain operational effectiveness in a substantially new environment, as appropriate.

    Jul 9, 2014

    May 14, 2014

    Apr 30, 2014

    Mar 26, 2014

    Jan 13, 2014

    Dec 9, 2013

    Dec 6, 2013

    Nov 20, 2013

    Oct 29, 2013

    Sep 25, 2013

    Looking for more? Browse all our products here