Army and Marine Corps Training:
Better Performance and Cost Data Needed to More Fully Assess Simulation-Based Efforts
GAO-13-698: Published: Aug 22, 2013. Publicly Released: Aug 22, 2013.
What GAO Found
Over the past several decades, the Army and Marine Corps have increased their use of simulation-based training--simulators and computer-based simulations. Historically, the aviation communities in both services have used simulators to train servicemembers in tasks such as takeoffs, and emergency procedures that could not be taught safely live. In contrast, the services' ground communities used limited simulations prior to 2000. However, advances in technology, and emerging conditions in Iraq and Afghanistan have led to increased use of simulation-based training in the ground forces. For example, in response to increases in vehicle rollovers, both services began using simulators to train servicemembers to safely evacuate vehicles. The services are also collaborating in the development of some simulation-based training devices. For instance, according to Marine Corps officials, the service reused 87 percent of the Army's Homestation Instrumentation Training System's components in its own training system, achieving about $11 million in cost avoidance and saving an estimated 7 years in fielding time. The services are also taking steps to better integrate live and simulation-based training, developing technical capabilities to connect previously incompatible simulation-based training devices. The Army's capability is now being fielded, and the Marine Corps' is in the initial development phase.
The Army and Marine Corps consider various factors in determining whether to use live or simulation-based training, but lack key performance and cost information that would enhance their ability to determine the optimal mix of training and prioritize related investments. As the services identify which requirements can be met with either live or simulation-based training or both, they consider factors such as safety and training mission. Also, they have cited numerous benefits of simulation-based training, such as improving servicemember performance in live training events, and reducing operating costs. Both services rely on subject matter experts, who develop their training programs, and after action reports from deployments and training exercises for information on how servicemembers may have benefited from simulation-based training. However, neither service has established outcome metrics to assist them in more precisely measuring the impact of using simulation-based devices to improve performance or proficiency. Leading management practices recognize that performance metrics can help agencies determine the contributions that training makes to improve results. Army and Marine Corps officials also generally consider simulation-based training to be less costly than live training and analyze some data, such as life cycle costs, when considering options to acquire a particular simulation-based training device. However, once simulation-based training devices are fielded, the services neither reevaluate cost information as they determine the mix of training nor have a methodology for determining the costs associated with simulation-based training. Federal internal control standards state that decision makers need visibility over a program's financial data to determine whether the program is meeting the agencies' goals and effectively using resources. Without better performance and cost data, the services lack the information they need to make more fully informed decisions in the future regarding the optimal mix of training and how best to target investments for simulation-based training capabilities.
Why GAO Did This Study
The Army and Marine Corps use live and simulation-based training to meet training goals and objectives. Service officials have noted benefits from the use of simulation-based training--both in terms of training effectiveness and in cost savings or cost avoidance. A House report accompanying the bill for the National Defense Authorization Act for 2012 mandated GAO to review the status of the military services' training programs. This report follows GAO's reports on the Navy and Air Force, and assesses (1) changes in the Army's and Marine Corps' use of simulation-based training, including efforts to integrate live and simulation-based training capabilities; and (2) the factors the Army and Marine Corps consider in determining whether to use live or simulation-based training, including the extent to which they consider performance and cost information. GAO focused on a broad cross-section of occupations (e.g., aviation, armor, artillery), and analyzed service training strategies and other documents; and conducted six site visits and interviewed service officials involved with training and training development for the selected occupations.
What GAO Recommends
GAO recommends that the services develop metrics, and a methodology to compare live and simulation-based training costs. DOD partially concurred, but noted that it captures all relevant costs needed for decision making. GAO continues to believe the services may not be considering some important simulation-based training costs and a specific methodology is needed to more fully identify the universe of costs needed for comparison purposes.
For more information, contact Sharon Pickup at (202) 512-9619 or firstname.lastname@example.org.
Recommendations for Executive Action
Comments: As of 18 Aug 2014, the Army and Marine Corps actions for this recommendation are currently ongoing and the recommendation status currently remains open. On 14 June 2014, the DOD Inspector General reported in the Defense Audit Management Information System that "the Office of the Deputy Assistant Secretary of Defense(Readiness) developed a decision algorithm to determine which military tasks could be taught virtually and which military tasks should only be taught in classroom or field environments (i.e., live). The algorithm was provided to the Services for peer-review and possible implementation. The Army is reviewing its progressive training models through a process called Training Summit IV (TS IV). These models establish how virtual and constructive based training is integrated with live training to optimize training readiness. The TS IV will include training model review by proponent schools, as well as a cross-section of unit commanders and leaders. This effort will be completed in Fiscal Year (FY) 2014 and presented for validation and G-3/5/7 approval at the Army Training General Officer Steering Committee in November 2014. Also, the Marine Corps initiated a request for an internal servicewide study of existing and potential approaches to this topic (4th Quarter FY 2013). The initial focus is in determining how metrics can be better used to assess the impact of simulation based on meeting Marine Corps Training Standards. Furthermore, a targeted study began in the 1st Quarter FY 2014 and is focused initially on enhancing the methodology for assessing individual based simulators against Training and Readiness (T&R) Standards. In FY 2015, the study results will shape policy on how future T&R manuals will identify the appropriateness of simulators and simulations for training."
Recommendation: To improve decision makers' abilities to make fully informed decisions concerning whether training requirements can be met with live and simulation-based training and determine optimal mixes of live and simulation-based training, the Secretary of Defense should direct the Secretary of the Army and the Commandant of the Marine Corps to develop outcome-oriented performance metrics that can be used to assess the impact of simulation-based training on improving the performance or proficiency of servicemembers and units.
Agency Affected: Department of Defense
Comments: As of 18 Aug 2014, the Army and Marine Corps actions for this recommendation are currently ongoing and the recommendation status currently remains open. On 14 June 2014, the DOD Inspector General reported in the Defense Audit Management Information System that "the Office of the Deputy Assistant Secretary of Defense (Readiness) has coordinated with the Army and Marine Corps to identify standard approaches to capture costs and cost benefit analysis that could be used DoD-wide. The Army has undertaken a "cost of training" analysis that is an on-going action to determine cost of readiness and/or training. One area of concentration is to look at the "Other Burdened Resources Required for Training Readiness." This area is further broken down into two areas: Investment/Modernization and Installation Services. The Investment/Modernization area will look at Non-System Training Aids, Devices, Simulators and Simulations while Installation Services will look at Post Deployment Software Support. In addition, the Army is gathering data to validate an existing model developed by the Simulations to Mission Command Interoperability Director (Program Executive Office for Simulation, Training, and Instrumentation) for the Value of Simulation Study consisting of five phases: Phase one focused on development of a working methodology to assess both quantitative and qualitative value of simulations used to support collective training (completed). Phase two is currently gathering data for model validation. Phase three will be an expansion to other simulation capabilities. Phase four is data gathering and validation. Phase five is expanded testing/methodology use case study/validation for return on investment use. The Marine Corps established a study, described in response to Recommendation 1, which will evaluate and propose the initial cost factors not currently captured during Programming yet would be relevant in determining the appropriate mix of live and simulated training. The initial results are expected in FY 2015."
Recommendation: To improve decision makers' abilities to make fully informed decisions concerning whether training requirements can be met with live and simulation-based training and determine optimal mixes of live and simulation-based training, the Secretary of Defense should direct the Secretary of the Army and the Commandant of the Marine Corps to develop a methodology--to include identifying the costs that should be included and how these costs should be captured--for comparing the costs associated with the use of live and simulation-based training.
Agency Affected: Department of Defense