This is the accessible text file for GAO report number GAO-14-134 entitled 'Defense Efficiencies: Action Needed to Improve Evaluation of Initiatives' which was released on January 17, 2014. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: GAO: Report to Congressional Committees: January 2014: Defense Efficiencies: Action Needed to Improve Evaluation of Initiatives: GAO-14-134: GAO Highlights: Highlights of GAO-14-134, a report to congressional committees. Why GAO Did This Study: In May 2010, the Secretary of Defense announced a department-wide initiative with the goal of achieving efficiencies and reducing excess overhead costs while reinvesting those savings in sustaining DOD’s force structure and modernizing its weapons portfolio. The Secretary tasked the military departments and SOCOM to find estimated savings of about $100 billion over the period of fiscal years 2012 to 2016. For fiscal years 2013 and 2014, DOD identified additional efficiency initiatives. The National Defense Authorization Act for Fiscal Year 2012 mandated that GAO assess the extent to which DOD has tracked and realized savings proposed pursuant to the initiative to identify $100 billion in efficiencies. As the second report in response to this mandate, this report addresses 1) DOD's progress in adjusting its approach to tracking and reporting on the implementation of its efficiency initiatives since GAO’s December 2012 report, and 2) the extent to which DOD is evaluating the impact of its initiatives. GAO reviewed guidance, and analyzed and discussed information developed after December 2012 with DOD officials. What GAO Found: The Department of Defense (DOD) has refined its approach for tracking and reporting on the status of efficiency initiatives by establishing specific requirements to standardize and expand the type of information that the military departments (Army, Navy, and Air Force) and U.S. Special Operations Command (SOCOM) report to senior decision makers. Initially, DOD provided general direction through emails, briefings, and training, which gave the military departments and SOCOM flexibility to selectively report on the initiatives that they believed were important, resulting in inconsistencies. For example, prior to February 2013, all but the Navy had chosen to report on all their initiatives. In February 2013, the DOD Comptroller issued written guidance that specified the type of information to be reported, including 1) whether original net savings projections are being met, and 2) any associated program or milestone risks. In instances where original net savings projects were not met or risks were identified, the guidance required further detail such as how implementation would be achieved. As a result, in their March 2013 reports, the military departments and SOCOM only reported details on those initiatives that were not achieving original net savings estimates or where risk had been identified. GAO observed, during this review, that information on all initiatives was now unavailable to DOD decision makers, thus hindering their ability to assess implementation progress across the full range of initiatives. Comptroller officials agreed that such information would enhance DOD’s oversight, and in October 2013, the DOD Comptroller issued updated guidance, directing that this information also be reported on initiatives on track to achieve savings or not experiencing risk. The military departments and SOCOM subsequently began submitting reports with this broader set of information. The military departments and SOCOM have taken steps to evaluate the impact of some of their efficiency initiatives, such as establishing performance measures to assess their impact on achieving desired outcomes. However, this has largely occurred on an ad hoc basis and varies by initiative because DOD has not required such evaluations. As a result, DOD lacks a systematic basis for evaluating whether its various initiatives have improved the efficiency or effectiveness of its programs or activities. In setting forth initiatives, the Secretary of Defense intended for DOD to improve the effectiveness and efficiency of its programs and activities, and that related initiatives should be specific, actionable, and measurable. While DOD has provided direction on how the military departments and SOCOM are to report on implementation status, this direction does not require them to develop approaches for evaluating the impact of their initiatives. In practice, the military departments and SOCOM varied in the extent to which they evaluated initiatives, including whether they had established measures or other indicators to assess outcomes. For example, GAO found instances where the military departments and SOCOM had established measures and assessed progress for some but not all initiatives. Developing a more systematic approach to evaluating the impact of its initiatives could provide DOD with more complete information to assess whether the initiatives are accomplishing desired outcomes, beyond achieving savings, and whether adjustments are needed in the scope of implementing the initiatives. What GAO Recommends: GAO recommends that DOD establish a requirement for the military departments and SOCOM to develop approaches for evaluating the impact of their efficiency initiatives, such as developing performance measures or other indicators. DOD concurred with GAO’s recommendation, and provided additional comments that it will cease tracking initiatives that strictly call for program terminations. GAO believes this to be a reasonable approach. View [hyperlink, http://www.gao.gov/products/GAO-14-134]. For more information, contact Zina D. Merritt at 202-512-5257 or merrittz@gao.gov. [End of section] Contents: Letter: Background: DOD Has Taken Steps to Further Refine Its Approach to Track and Report on Its Efficiency Initiatives: Military Departments and SOCOM Vary in the Extent and Nature of Their Efforts to Evaluate the Impact of Their Efficiency Initiatives: Conclusions: Recommendation for Executive Action: Agency Comments and Our Evaluation: Appendix I: Scope and Methodology: Appendix II: Comments from the Department of Defense: Appendix III: GAO Contact and Staff Acknowledgments: Table: Table 1: Projected Savings Identified by the Military Departments and SOCOM under the Secretary of Defense's Efficiency Initiative (Fiscal Years 2012 through 2016): Abbreviations: DOD: Department of Defense: DCMO: Deputy Chief Management Officer: MDUR: More Disciplined Use of Resources: OMB: Office of Management and Budget: SOCOM: U.S. Special Operations Command: [End of section] GAO: United States Government Accountability Office: 441 G St. N.W. Washington, DC 20548: January 17, 2014: Congressional Committees: In May 2010,[Footnote 1] the Secretary of Defense announced that the Department of Defense (DOD) was to undertake a department-wide initiative to assess how the department is staffed, organized, and operated with the goal of reducing excess overhead costs, and reinvesting those savings in sustaining DOD's current force structure[Footnote 2] and modernizing its weapons portfolio. The Secretary's initiative targeted both short-and long-term improvements and set specific goals and targets for achieving savings and efficiencies, which are expected to be achieved from fiscal years 2012 through 2016. As part of this effort, the Secretary of Defense tasked the military departments (Army, Navy, and Air Force) and other components including U.S. Special Operations Command (SOCOM) to find savings of about $100 billion in overhead costs over the specified time period. About one-third of these savings would be used to fund higher-than-expected operating costs and the remaining two-thirds-- over $70 billion--would be reinvested in high-priority military capabilities in fiscal years 2012 through 2016. Of this amount, about $11 billion was projected to be achieved in fiscal year 2012 and available for reinvestment, and about $14 billion was projected to be achieved and available for reinvestment in fiscal year 2013. Information accompanying DOD's fiscal year 2012 budget request outlined specific efficiency initiatives identified by the military departments and SOCOM. Prior to the beginning of fiscal year 2012, DOD's Comptroller and Deputy Chief Management Officer (DCMO) began developing an approach for the military departments and SOCOM to track and report on their efforts to implement efficiency initiatives and realize savings. The National Defense Authorization Act for Fiscal Year 2012 mandated GAO to assess the extent to which DOD has tracked and realized the savings proposed pursuant to the initiative to identify at least $100 billion in efficiencies during fiscal years 2012 through 2016, and to submit a report annually for each of those fiscal years.[Footnote 3] In December 2012, we issued our first report in response to this requirement.[Footnote 4] We reported that DOD had developed an approach to track and report on the implementation of its efficiency initiatives, but the approach had some limitations affecting its ability to monitor progress toward achieving programmatic and financial goals. To ensure more complete and consistent reporting, we recommended that DOD develop guidance that would standardize definitions and methodologies for DOD components to use in reporting their efficiency initiatives and savings, including defining specific types of costs associated with implementing its initiatives. Since then, DOD has continued to refine its approach to monitoring implementation of its efficiency initiatives, including issuing additional guidance clarifying requirements for tracking and reporting. In addition, DOD included additional efficiency initiatives in information accompanying its fiscal years 2013 and 2014 budget requests, and has announced its intent to implement an additional round of efficiency initiatives in fiscal year 2015. This report addresses (1) DOD's progress in adjusting its approach to tracking and reporting on the implementation of its efficiency initiatives since we issued our December 2012 report; and (2) the extent to which DOD is evaluating the impact of its initiatives on DOD's programs and activities. To perform our work, we reviewed guidance and documentation issued at the department-wide level as well as within the military departments and SOCOM to identify adjustments to tracking and reporting requirements since our December 2012 report. We also interviewed officials from the offices of the Comptroller and DCMO, the military departments, and SOCOM who are involved in monitoring the implementation of its efficiency initiatives to discuss their approach to tracking and reporting on the initiatives. Specifically, we obtained available information from each of the military departments and SOCOM, including briefings prepared for DOD officials, on the current status of efficiency initiatives, how original estimates of savings compared with savings realized to date, and any program or timeline risks associated with implementing the initiatives. Additionally, we reviewed existing guidance to identify any requirements for the evaluation of efficiency initiatives. We then analyzed information provided by each of the military departments and SOCOM as well as interviewed officials serving as focal points to determine the expected outcome or impact for individual initiatives, and what steps they have taken to evaluate the impact of those efficiency initiatives. For example, based on discussions with officials, we identified examples of initiatives where performance measures or other indicators had or had not been established, but did not evaluate the application or appropriateness of the measures or other indicators. We conducted this performance audit from May 2013 through January 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Background: As part of its fiscal year 2012 budget request, DOD outlined estimated savings of about $178 billion to be realized over a 5-year time period beginning in fiscal year 2012. According to DOD, these savings included about $154 billion from the Secretary's initiative and about $24 billion from other sources. Specifically, * The military departments and SOCOM identified a total of $100 billion in savings as a result of their efforts to support the Secretary's initiative. A majority of the projected savings identified by the military departments and SOCOM (approximately $70 billion, or 70 percent) was planned to be reinvested in high-priority military needs--such as enhancing weapon systems--while the remainder was planned to be used to address operating costs resulting from areas such as health care and training. * In addition to the $100 billion from the military departments and SOCOM, DOD proposed a $78 billion reduction in its overall budget plan over a 5-year time period, covering fiscal years 2012 through 2016, which reflected a 2.6 percent reduction from DOD's fiscal year 2011 budget submission over the same time period. Of this amount, $54 billion reflected projected savings identified from a health care policy assessment, government-wide civilian pay freeze, and other specific areas identified by the Secretary where immediate action could be taken department-wide. The remaining $24 billion reflected revised economic assumptions, projected savings from restructuring the Joint Strike Fighter weapon program, and projected savings from reducing the size of the Army and Marine Corps. Information accompanying DOD's fiscal year 2012 budget request cataloged the $100 billion in savings from the military departments and SOCOM under the following four categories: * reorganizations, such as restructuring headquarters management and eliminating unneeded task forces; * better business practices, such as reducing energy consumption; * program reductions and terminations, such as terminating weapon system programs; and: * reductions in lower priority programs, such as shifting funding requests from military construction projects to base operations. Table 1 shows the specific amounts of projected savings reported for each category and military department. Table 1: Projected Savings Identified by the Military Departments and SOCOM under the Secretary of Defense's Efficiency Initiative (Fiscal Years 2012 through 2016): Category of reduction: Reorganizations; Fiscal years 2012 through 2016: Army: $5.4 billion; Navy: $15.4 billion; Air Force: $4.2 billion; SOCOM: 0; Total for fiscal years 2012 through 2016: $25.0 billion. Category of reduction: Better business practices; Fiscal years 2012 through 2016: Army: $10.3 billion; Navy: $14.1 billion; Air Force: $20.6 billion; SOCOM: $0.4 billion; Total for fiscal years 2012 through 2016: $45.4 billion. Category of reduction: Program reductions/terminations; Fiscal years 2012 through 2016: Army: $11.0 billion; Navy: $5.5 billion; Air Force: $3.7 billion; SOCOM: $1.3 billion; Total for fiscal years 2012 through 2016: $21.5 billion. Category of reduction: Reductions in lower priority programs; Fiscal years 2012 through 2016: Army: $2.8 billion; Navy: 0; Air Force: $4.8 billion; SOCOM: $0.6 billion; Total for fiscal years 2012 through 2016: $8.2 billion. Category of reduction: Total; Fiscal years 2012 through 2016: Army: $29.5 billion; Navy: $35 billion; Air Force: $33.3 billion; SOCOM: $2.3 billion; Total for fiscal years 2012 through 2016: $100.1 billion. Source: DOD. [End of table] Under the Consolidated Appropriations Act for 2012, the Secretary of Defense was required to submit a report to the congressional defense committees on the $100 billion in savings that derived from the efficiency initiatives identified by the military departments. [Footnote 5] Among other things, the Act also required the Secretary to report on the following: * the budget account from which each savings identified will be derived; * the number of military personnel and full-time civilian employees of the federal government affected by such savings; * the estimated reductions in the number and funding of contractor personnel caused by such savings; * a specific description of activities or services that will be affected by such savings, including the locations of such activities or services; and: * certain information regarding each reinvestment planned to be funded with efficiency initiative savings.[Footnote 6] In June 2012, DOD submitted its report to the congressional defense committees and provided some information on the categories above. Among other things, at that time, DOD reported that it was on track to meet estimated savings targets for all of its efficiency initiatives. However, the report did not include a comprehensive analysis of reinvestments because, according to DOD, many areas in which reinvestments had occurred due to the efficiency initiatives included in the fiscal year 2012 President's Budget request had been offset by major force structure changes and other reductions in its fiscal year 2013 budget request. In briefings to the Comptroller delivered in February 2013 and March 2013, the military departments and SOCOM reported that they remain, with a few exceptions, on track to meet original savings estimates associated with their individual efficiency initiatives. At the time of our review, DOD had not compiled DOD-level summary information on its progress in achieving its original savings estimate of $100 billion. DOD officials cited various reasons why compiling and reporting on this information may not be feasible. For example, they noted that the need to apply spending reductions in response to sequestration[Footnote 7] affected funding levels for many programs, including areas targeted for efficiency initiatives. As a result, DOD had to adjust plans for executing programs as well as for implementing initiatives, such as adjusting the scope of initiatives or the timing of actual or planned actions for implementation. Because of the variability in its programs and funding amounts, DOD officials stated that, at a certain point, it becomes difficult to isolate whether savings were achieved solely due to implementing initiatives rather than a combination of factors. We have previously reported on opportunities for DOD to improve tracking and reporting on cost savings and efficiencies. For example, in March 2012, we reported that DOD took steps to examine its headquarters resources for potential efficiencies, but that it faced an underlying challenge of not having complete and reliable headquarters information available to make related assessments and decisions.[Footnote 8] To improve DOD's ability to identify how many headquarters personnel it has, we recommended that the Secretary of Defense direct the Director of Administration and Management, in consultation with the Under Secretary of Defense for Personnel and Readiness, to revise its DOD Instruction 5100.73, Major DOD Headquarters Activities, to, among other things, include all major DOD headquarters activity organizations. DOD partially concurred with our recommendation and commented that the shortcomings in the instruction have limited impact on the management of the department. In July 2012, we reported that, as part of one of its efficiency initiatives, the Air Force estimated it could save about $1.7 billion in its training program by reducing live flying hours and taking other steps, such as increasing the use of virtual training, but lacked a methodology for determining the costs of virtual training and therefore, did not consider these costs in its estimate.[Footnote 9] To improve decision makers' visibility over the costs related to virtual training, we recommended that the Secretary of Defense direct the Secretary of the Air Force to develop a methodology for collecting and tracking cost data for virtual training and use this cost data to help inform future decisions regarding the mix of live and virtual training. DOD concurred with our recommendation and identified actions being taken to enhance its ability to capture costs related to virtual training. Additionally, in December 2012, we reported that DOD had developed an approach for the military departments and SOCOM to follow in tracking and reporting on the status of efficiency initiatives; however, DOD's approach had some limitations that resulted in incomplete reporting which could limit the visibility of senior leaders in monitoring progress toward achieving programmatic and financial goals. Specifically, the offices of the Comptroller and DCMO had provided general direction through emails, briefings, and training, and, according to officials, had given the military departments and SOCOM flexibility to report on the efficiency initiatives that they felt were most important. In practice, the Army, Air Force, and SOCOM had reported on all of their efficiency initiatives, while the Navy reported on a subset of its initiatives based on what it deemed to be at medium or high risk of experiencing implementation issues or adversely affecting the Navy's ability to carry out its mission. With respect to realized savings, we reported that the military departments and SOCOM told us they were on track to realize estimated savings, but found some instances where certain costs were not considered. For example, for its initiative to reduce fleet shore command personnel from U.S. Pacific Fleet and U.S. Fleet Forces Command, the Navy did not account for potential increases in relocation costs for moving personnel to other areas within the Navy. We found that the military departments and SOCOM were not reporting consistent information or complete cost information because they had not received written guidance with standardized definitions and methodologies. Rather, the direction provided by the offices of the Comptroller and DCMO did not specify whether all of the costs associated with implementing an efficiency initiative, including costs not initially identified, should be included. To ensure more complete and consistent reporting, we recommended that DOD develop guidance with standardized definitions and methodologies for the military departments and SOCOM to use in reporting. Further, we recommended that guidance should define reporting requirements for such things as the specific types of costs associated with implementing the initiatives, including implementation costs that were not initially identified in calculations of net savings. DOD agreed with the spirit and intent of our recommendation and indicated it planned to issue additional formal guidance in the February 2013 timeframe. The status of DOD's implementation of this recommendation is discussed in more detail later in this report. Since initiating its initial round of initiatives for fiscal year 2012, DOD has continued to identify and implement efficiency initiatives. Specifically, in information accompanying its fiscal years 2013 and 2014 budget requests, DOD identified additional efficiency initiatives, referred to as More Disciplined Use of Resources (MDUR) initiatives. These initiatives are expected to generate $60 billion in savings for the period of fiscal years 2013 through 2017 and an additional $34 billion for the period of fiscal years 2014 through 2018. While savings generated by the Secretary's fiscal year 2012 efficiency initiatives were to be reinvested, savings from the MDUR initiatives were intended to help the department meet reductions to its budget, and therefore will not be reinvested. More recently, on July 31, 2013, as part of DOD's Strategic Choices and Management Review, the Secretary announced his plan to implement an additional round of efficiency initiatives in fiscal year 2015. According to the Secretary, a tenet of the review was the need to maximize savings from reducing DOD's overhead, administrative costs, and other institutional expenses. These initiatives would include management reforms, coupled with consolidations, personnel cuts, and spending reductions that would reduce DOD's overhead and operating costs by some $10 billion over the next 5 years and almost $40 billion over the next decade. DOD Has Taken Steps to Further Refine Its Approach to Track and Report on Its Efficiency Initiatives: DOD has taken steps to further refine its approach to its tracking and reporting on the implementation of its efficiency initiatives. Specifically, DOD issued written guidance that standardizes and expands the type of information on efficiency initiatives that the military departments and SOCOM are expected to report, which may improve visibility on the progress and risks in implementation for DOD decision makers. Moreover, in commenting on a draft of this report, DOD stated that it will cease to track initiatives once they have been implemented and will select for detailed tracking only those initiatives where this information will help it manage more effectively. Following our December 2012 report, the Comptroller issued written guidance in February 2013 establishing a standardized format for reporting on the fiscal year 2012 efficiency initiatives as well as the fiscal year 2013 MDUR initiatives.[Footnote 10] According to DOD officials, this guidance is also applicable to initiatives identified in fiscal year 2014 and any future initiatives. In contrast to the way they reported before, the military departments and SOCOM were now expected to report consistently and provide the status of their efficiency initiatives, including summary information related to (1) whether original net savings projections across the Future Years Defense Program[Footnote 11] are being met, (2) risks to program(s), mission(s), or resources associated with the efficiency initiative (characterized as "low", "medium", or "high" risk), and (3) any risks to "milestones" or the implementation status of the efficiency initiative (e.g., characterized as "on track," "off track but can meet major milestones," or "off track and cannot meet major milestones"). Only in instances where the military departments and SOCOM identified programs that were not achieving original net savings estimates or where program or milestone risk had been identified, the guidance requires further detail, including how implementation would be achieved. Further, all of the information was to be reported in a manner that mirrored the descriptions contained in DOD's fiscal year 2012 budget request justification book for the efficiency initiatives, whereby some of the efficiency initiatives were collapsed into broader groups of initiatives referred to by descriptive titles.[Footnote 12] In February and March 2013, using the new February 2013 guidance, the military departments and SOCOM completed the first round of semi- annual reporting on the fiscal year 2012 efficiency initiatives and fiscal year 2013 MDUR initiatives. In reviewing the military departments' and SOCOM's February 2013 and March 2013 reports, we observed that, consistent with the aforementioned February 2013 guidance, the military departments and SOCOM reported details associated with only those efficiency initiatives that were not achieving original net savings estimates or where program or milestone risk had been identified. As a result, detailed information on the full range of efficiency initiatives and related programs was not included in their reports. For example, the Air Force has as many as 10 individual initiatives that comprise its acquisition management initiative. Absent a requirement in the February 2013 guidance to report on each of those underlying initiatives, DOD decision makers would only receive information on the overall acquisition management initiative. Moreover, as a result of the reporting direction, DOD decision makers would receive detailed information on the overall acquisition management initiative only if the initiative is not meeting original savings estimates, or where program or milestone risk had been identified. Prior to the February 2013 guidance, some departments and SOCOM had previously chosen to report on all their initiatives. In reviewing the reports developed by the military departments and SOCOM in February 2013 and March 2013, we observed that information on all initiatives was now unavailable to DOD decision makers, thus hindering their ability to assess implementation progress across the full range of efficiencies. We discussed with DOD Comptroller officials whether reporting on only those efficiency initiatives not achieving their original estimates or facing risk had provided the Comptroller with sufficient details to oversee all of the initiatives. Comptroller officials agreed that reporting on each of the individual efficiency initiatives would improve DOD decision makers' visibility and therefore provide information needed for their oversight. They also noted that it would facilitate DOD's ability to address any future congressional reporting requirements. As a result, the Comptroller's office subsequently issued guidance in October 2013[Footnote 13] that, according to these officials, superseded the February 2013 guidance and expanded the amount of information to be reported. Specifically, this guidance directed the military departments and SOCOM to submit further detail for all efficiency initiatives, rather than merely those not achieving the original estimates or at risk. Beginning in October 2013, the military departments and SOCOM began submitting reports that included this broader set of information. While obtaining this broader set of information, DOD stated in its written comments on a draft of this report, provided on January 6, 2014, that it will narrow the scope of efficiency initiatives that will be tracked due to the period of constrained resources it is experiencing. DOD stated that it will cease tracking initiatives once they have been implemented, and will select for detailed tracking only those initiatives where this information will help it manage more effectively. In clarifying its written comments, DOD officials stated that all of its efficiency initiatives, except implemented or those which strictly call for terminations of programs, such as weapons systems, will be selected for detailed tracking. While the October 2013 guidance, which requires the military departments and SOCOM to report more detailed information on the full range of ongoing efficiency initiatives, does not specify that initiatives that strictly call forprogram terminations should no longerbe tracked, we believe it is reasonable for DOD to cease the tracking of initiatives that strictly call for program terminations. We note that, in issuing additional guidance on its tracking and reporting on efficiency initiatives, DOD did not include any direction as to the specific types of costs that the military departments and SOCOM should consider in determining realized savings associated with implementation, such as costs that were not initially identified in calculations of net savings, as we had recommended in our December 2012 report. According to a Comptroller official, DOD has various guidance on developing cost estimates that the military departments and SOCOM can use in determining savings associated with the implementation of their efficiency initiatives. We reviewed the documents and discussed with the efficiency initiative focal points how, or if, this guidance was applied in developing their cost estimates. Some of the program managers with whom we spoke confirmed that while they were aware of existing guidance on developing cost estimates, they had not been instructed to use this guidance to determine specific types of costs that should be considered in calculating net savings. As a result, we continue to believe that our prior recommendation in our December 2012 report has merit and should be implemented. Military Departments and SOCOM Vary in the Extent and Nature of Their Efforts to Evaluate the Impact of Their Efficiency Initiatives: The military departments and SOCOM have taken steps to evaluate some of their efficiency initiatives, such as establishing performance measures and collecting performance data. However, these efforts have largely occurred on an ad hoc basis and vary by efficiency initiative because DOD has not established a requirement for performing such evaluations. As a result, DOD lacks a systematic basis for evaluating the impact of its efficiency initiatives on improving program efficiency or effectiveness. In setting forth the initial efficiency initiatives, the Secretary of Defense intended for DOD to improve the effectiveness and efficiency of its programs and activities. The Secretary also directed that any efficiency initiative must be specific, actionable, and measurable. Our prior review of federal agencies' efficiency efforts concluded that an improvement in efficiency need not only involve a reduction in costs, but also can be achieved by maintaining federal government services or outcomes with fewer resources (such as time or money), or improving or increasing the quality or quantity of services while maintaining (or reducing) resources.[Footnote 14] In addition, we and other agencies, such as the Office of Management and Budget (OMB), have documented the need to develop performance measures for evaluating progress toward achieving desired outcomes.[Footnote 15] For example, as we have previously concluded, performance measures should be measurable, outcome-oriented, and actively tracked and reported. As our prior work has shown, leading organizations that employ result-oriented management use performance information as a basis for decision making and have found this approach improves program results.[Footnote 16] As previously discussed, the Comptroller's October 2013 guidance provides direction on how the military departments and SOCOM are to approach reporting on the status of their efficiency initiatives, but does not require them to develop approaches for evaluating the impact of initiatives on achieving desired outcomes. In practice, we found that the military departments and SOCOM varied in the extent to which they evaluated individual efficiency initiatives, including whether they had established measures or indicators to gauge the impact on program efficiency or effectiveness beyond savings. The following paragraphs provide examples of the services' and SOCOM's efforts to evaluate certain efficiency initiatives. * Air Force's Facility Sustainment Initiative: This initiative is intended to reduce infrastructure maintenance costs by a total of $1.4 billion during the period of fiscal years 2012 through 2016 by performing preventative maintenance before critical failures occur. The Air Force uses a model to predict and prioritize infrastructure most at risk for critical failures and then focuses preventive maintenance efforts on such infrastructure.[Footnote 17] Furthermore, the Air Force has established measures to track the amount of hours spent performing preventive and corrective maintenance over the course of the initiative to determine whether this effort achieved the intended outcome, which was to reduce the amount of more costly corrective maintenance performed. We have previously concluded that deferring facility sustainment can lead to shortened facility service lives and increased future costs for recapitalization.[Footnote 18] * SOCOM's Information Technology Services Efficiency Initiative: SOCOM established a new approach for its information technology services that is intended to reduce costs by a total of $394 million during the period of fiscal years 2012 through 2016. According to SOCOM officials, the new approach involved a contract framework for information technology services that reduces costs by awarding funds directly to the organizations that provide the services on a competitive basis, rather than through an intermediary that selects the organizations that provide the information technology services. The approach also adopts other best practices for procurement, such as providing performance-based incentives. As part of this initiative, SOCOM established multiple individual measures to assess contractor performance, such as answering a help desk call within a set amount of time or tracking trends on resolving information technology issues such as user access, but does not have measures in place to evaluate how the overall impact of the initiative affects the delivery of information technology services relative to the previous approach. SOCOM officials explained that because the implementation of its Information Technology Services Efficiency Contract occurred prior to it being identified as an efficiency initiative, methods of evaluating its effectiveness or efficiency, other than cost, were not established. * Navy's Total Ownership Cost Initiative: This initiative seeks to achieve efficiencies through life cycle management of the Navy's ships and encompasses multiple underlying initiatives, such as the Navy's Revised Virginia Class Drawings Efficiency Initiative. This initiative is intended to reduce costs of $30.3 million during the period of fiscal years 2012 through 2016 by moving away from reliance on paper documents toward an electronic system that allows multiple users to make revisions and access up-to-date documents. The Navy has not yet identified measures to evaluate how increasing the use of the electronic system to process technical documents will maintain or improve work processes. Navy officials indicated the same is true for other fiscal year 2012 initiatives that make up the Navy's Total Ownership Cost initiative and commented that the focus was on tracking savings and not on developing efficiency measures to assess whether its initiatives, once implemented, improve the effectiveness or efficiency of these programs. * Army and Air Force Data Center Consolidation Efficiency Initiatives: These initiatives are part of the larger Federal Data Center Consolidation Initiative, directed by OMB, that seeks to consolidate information technology infrastructure and activities to save energy costs, among other goals, and are expected to reduce the Army and Air Force's costs by $490 million and $180 million, respectively, during fiscal years 2012 through 2016. Both the Army and Air Force have taken steps to implement these initiatives. According to Air Force officials, they had begun to establish measures that could be used to evaluate the impact of these initiatives, but faced some challenges due to changing guidance. For example, officials discussed that OMB guidance issued after the initiatives were underway expanded the definition of a data center and effectively increased the scope of the military departments' consolidation effort after these initiatives were submitted in the fiscal year 2012 budget request. This resulted in the reconfiguration of planning and implementation schedules. Air Force officials also stated that they had begun to develop performance measures to assess impact when a new DOD effort to establish a secure, joint information environment[Footnote 19] was put in place. Therefore, the Air Force had to adjust their implementation plans and postpone the development of their measures to ensure actions taken on this initiative conformed to the new DOD requirement. While the focus of DOD's effort was to quickly identify funds that could be reinvested into other higher priority programs, military department and SOCOM officials explained that because the initial effort to identify efficiency initiatives occurred late at the end of the cycle they used to build their fiscal year 2012 budget submission, their effort focused on tracking savings targets and not on developing measures to evaluate impact. In the subsequent budget cycles that included the MDUR initiatives, the focus remained on identifying areas for reductions in spending. DOD officials agreed that additional measures could be useful to evaluate impacts--beyond savings--of their efficiency initiatives. Our prior work concluded that such measures can assist managers in determining whether desired outcomes were being achieved or if adjustments were needed, such as in the scope of the initiative or to the nature or timing of implementation actions. Without a systematic way to evaluate the impact of its efficiency initiatives, DOD is limited in its ability to assess whether the efficiency initiatives have improved the effectiveness or efficiency of its programs and activities. Conclusions: Over the past few years, in light of mounting fiscal pressures, DOD has continued to identify and implement efficiency initiatives with certain goals in mind, including achieving cost savings and seeking opportunities to enhance the efficiency or effectiveness of its programs and activities. DOD's recent efforts to refine its approach for tracking and reporting on its current efficiency initiatives has the potential for providing greater oversight to decision makers on progress of the military departments and SOCOM on the status of their implementation efforts. However, its efforts to date do not sufficiently ensure that leaders have the information they need to fully assess the impact these initiatives are having on DOD's programs and activities. Having a systematic way to evaluate the impact of its efficiency initiatives beyond cost savings could provide DOD the ability to determine whether or not its initiatives are improving the efficiency and effectiveness of its programs and activities while also achieving savings. Such information could also inform DOD as to whether actions are needed to make adjustments to the scope of any given initiative and related programmatic actions necessary for implementation. Recommendation for Executive Action: To enhance DOD's ability to determine whether its efficiency initiatives are having the desired effect of improving efficiency and effectiveness, we recommend that the Secretary of Defense require the military departments and SOCOM to develop approaches for evaluating the impact of their efficiency initiatives, such as establishing performance measures or other indicators, collecting related performance information, and using this information to measure progress in achieving intended outcomes associated with their initiatives until implemented. Agency Comments and Our Evaluation: In written comments on a draft of this report, DOD concurred with our recommendation and provided additional comment. Specifically, DOD concurred with having the military departments and SOCOM develop performance measures and other indicators for evaluating the impact of its efficiency initiatives, and commented that it has decided to cease tracking initiatives once they have been implemented. DOD also provided technical comments, which were incorporated as appropriate. The full text of DOD's comments is reprinted in appendix II. In its overall comments, DOD stated it intends to continue to refine its procedures, guidance, and oversight in order to achieve its goal of identifying and implementing efficiencies. DOD also stated that during this period of constrained resources, it must avoid creating a costly and redundant oversight process. To that end, DOD stated that it will cease tracking initiatives once they have been implemented, and will select for detailed tracking only those initiatives where this information will help it manage more effectively. We have modified the report to reflect DOD's decision. In clarifying its written comments, DOD officials stated that DOD will select for detailed tracking all of its efficiency initiatives except those implemented or which strictly call for terminations of programs, such as weapons systems. While the October 2013 guidance does not specify that initiatives that strictly call for program terminations should no longer be tracked, in a resource-constrained environment, we believe it is reasonable for DOD to do so. We also expect that DOD will clarify this revised approach in any future guidance to the military departments and SOCOM. In addition, we have modified the recommendation to clarify its intent that DOD should develop an approach for evaluating the impact of its efficiency initiatives until those initiatives have been implemented. We are sending copies of this report to the Secretary of Defense and appropriate congressional committees. In addition, the report is available at no charge on the GAO website at hyperlink, http://www.gao.gov]. If you or your staff have any questions about this report, please contact me at (202) 512-5257 or merrittz@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III. Signed by: Zina D. Merritt: Director: Defense Capabilities and Management: List of Committees: The Honorable Carl Levin: Chairman: The Honorable James Inhofe: Ranking Member: Committee on Armed Services: United States Senate: The Honorable Richard J. Durbin: Chairman: The Honorable Thad Cochran: Ranking Member: Subcommittee on Defense: Committee on Appropriations: United States Senate: The Honorable Howard P. McKeon: Chairman: The Honorable Adam Smith: Ranking Member: Committee on Armed Services: House of Representatives: The Honorable Rodney Frelinghuysen: Chairman: The Honorable Pete Visclosky: Ranking Member: Subcommittee on Defense: Committee on Appropriations: House of Representatives: [End of section] Appendix I: Scope and Methodology: To determine the progress DOD has made in adjusting its approach to tracking and reporting on the implementation of its efficiency initiatives since we last reported in December 2012 and to assess the extent to which DOD is evaluating the impact of its efficiency initiatives on DOD programs and activities, we reviewed guidance and documentation issued at the department-wide level as well as within the military departments and SOCOM. We also interviewed officials from the offices of the Comptroller and DCMO, the military departments, and SOCOM who are involved in monitoring the implementation of its efficiency initiatives to discuss their approach to tracking and reporting on the initiatives. Specifically, we obtained available information from each of the military departments and SOCOM, including briefings prepared for senior DOD officials, on the current status of initiatives, how original estimates of savings compared with savings realized to date, and any program or timeline risks associated with implementing the efficiency initiatives. Additionally, we reviewed existing guidance to identify any requirements for evaluation of the efficiency initiatives. We then analyzed information provided by each of the military departments and SOCOM as well as interviewed officials from each of the military services and SOCOM serving as focal points for the efficiency initiatives to determine the expected outcome or impact for individual initiatives, and what steps they have taken to evaluate the impact of efficiency initiatives. We conducted this performance audit from May 2013 through January 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of section] Appendix II: Comments from the Department of Defense: Office of The Under Secretary Of Defense: Comptroller: (Program/Budget) 1100 Defense Pentagon: Washington, DC 20301-1100: Ms. Zina D. Merritt: Director, Defense Capabilities and Management: U.S. Government Accountability Office: 441 G Street, N.W. Washington, D.C. 20548: Dear Ms. Merritt: This is the Department of Defense (DOD) response to the GAO draft report GAO-14-134, "Defense Efficiencies: Actions Needed to Improve Evaluation of Initiatives," December 4, 2013. The Department continues to place great emphasis on identifying and implementing efficiencies. DoD appreciates that GAO recognizes its efforts to improve oversight of our efficiency initiatives. The Department concurs, with comment, regarding GAO's recommendation to have the military departments and SOCOM develop performance measures and other indicators. We intend to continue to refine DoD-wide procedures, guidance, and oversight in order to achieve these goals. However, during this period of constrained resources, we must avoid creating a costly and redundant oversight process. To that end, we will cease tracking initiatives once they have been implemented, and we will select for detailed tracking only those initiatives where this information will help us manage more effectively. Thank you for the opportunity to comment on the draft report. If you have further questions, please direct them to Ms. Marcia Case, Associate Director, Military Operations, Program and Budget, Office of the Comptroller, 703-697-2808, marcia.a.case.civ@mail.mil. Sincerely, Signed by: Monique L. Dilworth: Director of Operations: [End of section] Appendix III: GAO Contact and Staff Acknowledgments: GAO Contact: Zina D. Merritt, (202)512-5257 or merrittz@gao.gov: Staff Acknowledgments: In addition to the contact named above, Tina Won Sherman, Assistant Director; Grace Coleman; Susan Langley; Ricardo A. Marquez; Sharon L. Pickup; Mike Silver; Michael Shaughnessy; Susan Tindall; and Sarah Veale made key contributions to this report. [End of section] Footnotes: [1] Remarks as delivered by former Secretary of Defense Robert M. Gates, Abilene, Kansas, May 8, 2010. [2] Force structure is the number, size, and composition of the units that make up U.S. defense forces (e.g., divisions, brigades, ships, air wings, and squadrons). [3] See Pub. L. No.112-81, § 1054 (2011). [4] GAO, Defense Management: Opportunities Exist to Improve Information Used in Monitoring Status of Efficiency Initiatives, [hyperlink, http://www.gao.gov/products/GAO-13-105R] (Washington, D.C.: Dec. 4, 2012). [5] See Pub. L. No. 112-74, § 8123 (2011). Although the provisions do not discuss SOCOM, DOD's report included similar information for the command. [6] See § 8123(1), (2). [7] Sequestration is the reduction or cancellation of budgetary resources by presidential order. The Budget Control Act of 2011, Pub. L. No. 112-25 (2011), as amended, requires the President to order a sequestration under certain circumstances. The Act also imposes discretionary spending limits. [8] GAO, Defense Headquarters: Further Efforts to Examine Resource Needs and Improve Data Could Provide Additional Opportunities for Cost Savings, [hyperlink, http://www.gao.gov/products/GAO-12-345] (Washington, D.C.: Mar. 21, 2012). [9] GAO, Air Force Training: Actions Needed to Better Manage and Determine Costs of Virtual Training Efforts, [hyperlink, http://www.gao.gov/products/GAO-12-727] (Washington, D.C.: July 19, 2012). [10] See Memorandum from the Under Secretary of Defense (Comptroller), Fiscal Year (FY) 2013 More Disciplined Use of Resources (MDUR) and Congressional Interest Items Reporting Requirements (Feb. 8, 2013). This memorandum was addressed to the heads of various DOD components; for the purposes of this report, we focus on the military departments and SOCOM. [11] The Future Years Defense Program is DOD's financial plan over a 6- year period. [12] The February 2013 OUSD Comptroller memo called for a listing of each initiative by title that was reflected in DOD's 2012 budget request justification book. The justification book summarized initiative savings under four general categories, seven funding categories, and, for each military department and SOCOM, more descriptive titles, such as "reduce recruiting and retention incentives" or "reduce energy consumption." The military departments and SOCOM used the descriptive titles when reporting on their efficiency initiatives. [13] Memorandum from the Office of the Under Secretary of Defense Comptroller, More Disciplined Use of Resources (MDUR) and Congressional Interest Items Reporting Requirements (Oct. 16, 2013). [14] GAO, Streamlining Government: Key Practices from Select Efficiency Initiatives Should Be Shared Governmentwide, [hyperlink, http://www.gao.gov/products/GAO-11-908] (Washington, D.C.: Sept. 30, 2011) and Streamlining Government: Opportunities Exist to Strengthen OMB's Approach to Improving Efficiency, [hyperlink, http://www.gao.gov/products/GAO-10-394] (Washington, D.C.: May 7, 2010). [15] See generally OMB Circular No. A-11, Preparation, Submission, and Execution of the Budget, Circular No. A-11, Part 6 (July 26, 2013). GAO, Data Center Consolidation: Strengthened Oversight Needed to Achieve Billions of Dollars in Savings, [hyperlink, http://www.gao.gov/products/GAO-13-627T] (Washington, D.C.: May 14, 2013). Also, with the enactment of the GPRA Modernization Act of 2010 which updated the Government Performance and Results Act of 1993, Congress established a new framework aimed at taking a more crosscutting and integrated approach to focusing on results and improving government performance. See generally Pub. L. No. 111-352 (2011). [16] GAO, Managing for Results: Enhancing Agency Use of Performance Information for Management Decision Making, [hyperlink, http://www.gao.gov/products/GAO-05-927] (Washington, D.C.: Sept. 9, 2005); Results-Oriented Cultures: Creating a Clear Linkage between Individual Performance and Organizational Success, [hyperlink, http://www.gao.gov/products/GAO-03-488] (Washington, D.C.: Mar. 14, 2003); and Executive Guide: Effectively Implementing the Government Performance and Results Act, [hyperlink, http://www.gao.gov/products/GAO/GGD-96-118] (Washington, D.C.: June 1996). [17] GAO previously reported that accuracy and supportability issues with two of DOD's facility sustainment model key inputs have affected the reliability of the model's estimates. See GAO, Defense Infrastructure: Continued Management Attention Is Needed to Support Installation Facilities and Operations, [hyperlink, http://www.gao.gov/products/GAO-08-502] (Washington, D.C.: Apr. 24, 2008). Our current body of work did not analyze the model or the reliability of the model's estimates. [18] See [hyperlink, http://www.gao.gov/products/GAO-08-502]. [19] The DOD Joint Information Environment is an initiative to establish within the department a secure, joint information environment comprised of shared information technology infrastructure and services. [End of section] GAO’s Mission: The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to [hyperlink, http://www.gao.gov] and select “E-mail Updates.” Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO: Connect with GAO on facebook, flickr, twitter, and YouTube. Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts. Visit GAO on the web at [hyperlink, http://www.gao.gov]. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; E-mail: fraudnet@gao.gov; Automated answering system: (800) 424-5454 or (202) 512-7470. Congressional Relations: Katherine Siggerud, Managing Director, siggerudk@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, DC 20548. Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, DC 20548. [End of document]