DOD Needs to Develop Performance Goals and Metrics for Advanced Distributed Learning in Professional Military Education
GAO-04-873: Published: Jul 30, 2004. Publicly Released: Jul 30, 2004.
As part of its transformation to prepare the armed forces to meet current and future challenges, the Department of Defense (DOD) is expanding its use of advanced distributed learning (ADL) techniques in senior- and intermediate-level officer professional military education (PME). ADL instruction does not require an instructor's presence, and it facilitates the use of varied learning management systems. To date, the application of ADL has been targeted to nonresident students. To determine whether DOD uses a systematic process for evaluating the results of ADL application, GAO was asked to examine DOD's metrics for assessing program effectiveness, to compare DOD's criteria for converting courses to ADL with those of private-sector institutions, and to identify the challenges to ADL implementation.
DOD does not have specific performance goals and metrics with which to assess ADL effectiveness in PME. Furthermore, although GAO and private-sector organization have established frameworks for assessing the effectiveness of educational programs by focusing on metrics for learning outcomes--that is, the knowledge, skills, and abilities that students attain through learning activities--DOD's oversight focuses instead on educational inputs such as facilities, student to faculty ratios, and student body composition. Since ADL is still a new and evolving tool, systematic evaluative processes have not yet been required. Without clear goals and an effective process for evaluating the results of ADL application, DOD cannot ensure that its program is achieving an appropriate return on investment and other goals. The criteria for converting PME courses and curricula to ADL vary by school and by military service, are based on subjective choices as to which content is suited for online delivery, and are focused solely on nonresident programs. The private sector similarly lacks systematic criteria in its use of ADL. However, DOD's implementation of ADL programs for PME compares favorably with private-sector institutions. Cultural, technological, and resource challenges affect ADL implementation. For example, some military policies reflect a lower estimation of the value of nonresident PME, and many respondents to a survey of ADL students and alumni indicated that its quality and achievement of outcomes did not compare favorably, in their view, with those of resident education programs. The technological challenges of balancing computer access with network security, along with resource challenges of funding and increased burdens on limited administrative staff, are additional concerns.
Recommendations for Executive Action
Status: Closed - Not Implemented
Comments: While DOD partially concurred with this recommendation, they did not indicate any corrective action has been taken and officially closed the recommendation as of 10/1/2004. DOD stated in the most recent DAMIS report that they support the use of specific performance effectiveness goals for professional military education but do not believe it appropriate to develop such goals for a specific delivery method such as ADL. They further stated that educational outcomes are based on common standards defined in the Professional Military Education Policy, and that ADL is one delivery system that will help them accomplish these outcomes. A discussion with an OSD official responsible for ADL stated that DOD's position has not changed and there is no plan to develop specific performance effectiveness goals for ADL. Based on this response and lack of DOD action, GAO does not consider it a reasonable expectation that DOD will implement this recommendation.
Recommendation: The Secretary of Defense should direct the Under Secretary of Defense for Personnel and Readiness, in concert with the Joint Staff, service headquarters, and the PME schools, to promote the development of specific performance effectiveness goals for ADL.
Agency Affected: Department of Defense
Status: Closed - Implemented
Comments: DOD concurred with this recommendation and the most recent DAMIS report and discussions with an OSD official responsible for ADL, indicates that a number of actions have been completed. First, a comprehensive analysis of evaluations of learning outcomes in ADL was completed, establishing a baseline for comparisons of ongoing and future ADL courses. Additionally, a database of 25 variables that influence the effectiveness of ADL will be updated and maintained as new information on the ADL is reported. Also, a Ph.D. level evaluation analyst to head an Evaluation Practices cell at the ADL Co-Laboratory has been hired to in part develop best practices for evaluating ADL programs. Finally, as a part of an ongoing effort to evaluate the value of ADL programs, the department issued on 03/24/2005 a Broad Agency Announcement to obtain competitive bids seeking technologies, methodologies and techniques for continuously collecting metrics online about a learner's performance. These bids have been reviewed and contracts awarded.
Recommendation: The Secretary of Defense should direct the Under Secretary of Defense for Personnel and Readiness, in concert with the Joint Staff, service headquarters, and the PME schools, to promote the use of ADL technologies to capture data to provide knowledge about learning outcomes.
Agency Affected: Department of Defense