DOD Simulations:

Improved Assessment Procedures Would Increase the Credibility of Results

PEMD-88-3: Published: Dec 29, 1987. Publicly Released: Feb 5, 1988.

Additional Materials:

Contact:

Kwai-Cheung Chan
(202) 512-3652
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

GAO reviewed selected Department of Defense (DOD) weapons systems simulations to determine: (1) what factors DOD should consider in assessing simulation credibility; (2) the results of assessing weapons systems simulations with respect to these factors; and (3) DOD efforts to reinforce simulation credibility.

GAO developed an assessment framework of 14 factors covering theory, model design, input data, correspondence between the model and the real world, management, documentation, and reporting, applied it to three Army air defense simulations, and found that: (1) the simulations' representations of combat reality were limited; and (2) DOD efforts to validate simulation results were weak. GAO also found that: (1) the Army support structures for simulation activities were very limited; (2) although DOD believes that the credibility of simulation results is important, it has not established a systematic and uniform system for determining credibility; and (3) although DOD issued directives and regulations on related topics, there was little guidance with direct relevance to simulations.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Comments: Many DOD directorates have or are in the process of taking action. There are others, however, that have this action under study.

    Recommendation: To reinforce its efforts and to ensure that such practices are followed, the Secretary of Defense should develop and implement guidance on producing, validating, documenting, managing, maintaining, using, and reporting weapon-system effectiveness simulations. The guidance should include a provision for routine reviews of a simulation's credibility and, in this way, the identification of problems that should be resolved.

    Agency Affected: Department of Defense

  2. Status: Closed - Not Implemented

    Comments: DOD is revising all acquisition guidance.

    Recommendation: The Secretary of Defense should explore the possibility of requiring that a statement regarding validation accompany the report of a simulation's results.

    Agency Affected: Department of Defense

  3. Status: Closed - Not Implemented

    Comments: Carmonette and ADAGE are no longer used by the Army. COMO III is modified for specific studies. Actions are being taken to implement an Army-wide program requiring simulation accreditation.

    Recommendation: To make the ADAGE, Carmonette, and COMO III models more useful in future applications, the agency responsible for managing each simulation should explore the feasibility of remedying the limitations identified, especially in the area of validation.

    Agency Affected: Department of Defense

 

Explore the full database of GAO's Open Recommendations »

Dec 15, 2020

Oct 7, 2020

Sep 23, 2020

Jan 23, 2020

Dec 19, 2019

  • budget icon, source: GAO

    State and Local Governments' Fiscal Outlook:

    2019 Update
    GAO-20-269SP: Published: Dec 19, 2019. Publicly Released: Dec 19, 2019.

Nov 14, 2019

Oct 16, 2019

Jun 26, 2019

Apr 10, 2019

Mar 27, 2019

Looking for more? Browse all our products here