DOD Simulations:

Improved Assessment Procedures Would Increase the Credibility of Results

PEMD-88-3: Published: Dec 29, 1987. Publicly Released: Feb 5, 1988.

Additional Materials:

Contact:

Kwai-Cheung Chan
(202) 512-3652
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

GAO reviewed selected Department of Defense (DOD) weapons systems simulations to determine: (1) what factors DOD should consider in assessing simulation credibility; (2) the results of assessing weapons systems simulations with respect to these factors; and (3) DOD efforts to reinforce simulation credibility.

GAO developed an assessment framework of 14 factors covering theory, model design, input data, correspondence between the model and the real world, management, documentation, and reporting, applied it to three Army air defense simulations, and found that: (1) the simulations' representations of combat reality were limited; and (2) DOD efforts to validate simulation results were weak. GAO also found that: (1) the Army support structures for simulation activities were very limited; (2) although DOD believes that the credibility of simulation results is important, it has not established a systematic and uniform system for determining credibility; and (3) although DOD issued directives and regulations on related topics, there was little guidance with direct relevance to simulations.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Comments: Many DOD directorates have or are in the process of taking action. There are others, however, that have this action under study.

    Recommendation: To reinforce its efforts and to ensure that such practices are followed, the Secretary of Defense should develop and implement guidance on producing, validating, documenting, managing, maintaining, using, and reporting weapon-system effectiveness simulations. The guidance should include a provision for routine reviews of a simulation's credibility and, in this way, the identification of problems that should be resolved.

    Agency Affected: Department of Defense

  2. Status: Closed - Not Implemented

    Comments: DOD is revising all acquisition guidance.

    Recommendation: The Secretary of Defense should explore the possibility of requiring that a statement regarding validation accompany the report of a simulation's results.

    Agency Affected: Department of Defense

  3. Status: Closed - Not Implemented

    Comments: Carmonette and ADAGE are no longer used by the Army. COMO III is modified for specific studies. Actions are being taken to implement an Army-wide program requiring simulation accreditation.

    Recommendation: To make the ADAGE, Carmonette, and COMO III models more useful in future applications, the agency responsible for managing each simulation should explore the feasibility of remedying the limitations identified, especially in the area of validation.

    Agency Affected: Department of Defense

 

Explore the full database of GAO's Open Recommendations »

Sep 22, 2016

Sep 21, 2016

Sep 19, 2016

Sep 12, 2016

Sep 8, 2016

Sep 7, 2016

Sep 6, 2016

Aug 25, 2016

Looking for more? Browse all our products here