This is the accessible text file for GAO report number GAO-05-715R entitled 'Defense Transportation: DOD Has Adequately Addressed Congressional Concerns Regarding the Cost of Implementing the New Personal Property Program Initiatives' which was released on June 10, 2005. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: Washington, DC 20548: June 9, 2005: The Honorable John Warner: Chairman: The Honorable Carl Levin: Ranking Minority Member: Committee on Armed Services: United States Senate: The Honorable Duncan L. Hunter: Chairman: The Honorable Ike Skelton: Ranking Minority Member: Committee on Armed Services: House of Representatives: Subject: Defense Transportation: DOD Has Adequately Addressed Congressional Concerns Regarding the Cost of Implementing the New Personal Property Program Initiatives: Military personnel and their families can expect to relocate many times during a servicemember's career. As the moving industry's single largest customer, the Department of Defense (DOD) spends more than $1.7 billion annually for its personal property program, which provides household goods transportation and storage services for military personnel and their families when they relocate. The program manages more than 600,000 personal property shipments each year. For more than 10 years, DOD has been pursuing various initiatives for improving the quality of its personal property program. In June 2002, the U.S. Transportation Command completed an extensive study that compared the features of the current personal property program with three pilot programs that tested alternative approaches for improving the current program. In November 2002, DOD issued a report to Congress that included three recommended program improvement initiatives resulting from this study and estimated that an additional 13 percent increase over current program costs would be required to implement two of these initiatives.[Footnote 1] In April 2003, we reported on the pilot program evaluation and stated that the recommendations contained in DOD's November 2002 report offered solutions to long-standing problems in the personal property program and should be implemented within budget constraints.[Footnote 2] However, we raised concerns about whether the two recommendations related to the claims and contracting processes could be implemented within the projected 13 percent cost estimate. Our concerns were partially based on DOD not adequately substantiating the expected economies of scale that supported its assumption of a 5 percent reduction in average prices under the pilot program when a full-scale program is implemented. As a result, we recommended that DOD quantify the risk associated with the cost estimate before DOD's recommended initiatives are implemented. In its May 2004 report on the National Defense Authorization Act for Fiscal Year 2005,[Footnote 3] the House Armed Services Committee directed the Secretary of Defense to reevaluate DOD's proposed cost estimate, quantify the risk or likelihood of achieving its goals within the 13 percent projected cost increase, and develop a range of possible cost increases associated with the risk. The committee also directed GAO to review and report on whether DOD adequately performed these tasks. To conduct our work, we reviewed prior DOD and GAO reports on the personal property program and interviewed DOD officials and officials from DOD's contractor who were involved in the current cost reevaluation study. We assessed the methodologies used by DOD to reevaluate the cost estimate and quantify the risk associated with implementing the proposed initiatives within the cost estimate, including establishing ranges of possible cost increases. We also assessed the reasonableness of the methodologies used by DOD to perform these tasks. We performed our review from November 2004 through May 2005 in accordance with generally accepted government auditing standards. The scope and methodology we used in our review are described in further detail in enclosure I. Results in Brief: The methodology used by DOD to address congressional concerns--- reevaluate its cost estimate, quantify the risk associated with achieving the 13 percent cost increase, and establish a range of possible cost increases--was reasonable for conducting such assessments. In its reevaluation of the cost estimate, DOD validated the current mix of household goods shipments to ensure that the mix used in the original assessment was appropriate. DOD also provided additional support for its assessment of the anticipated savings that would result from expanding the pilot to all of DOD. Furthermore, the results of DOD's risk-based simulation analysis, which included establishing ranges of possible cost increases, indicated that about 80 percent of the time, DOD could implement the recommended initiatives to improve the current program with an increase of 15 percent or less above current program costs. DOD concurred with the content of this letter. Background: DOD's personal property program is managed centrally by the Military Surface Deployment and Distribution Command, formerly known as the Military Traffic Management Command.[Footnote 4] DOD has experienced long-standing problems with its current personal property program, including excessive loss of or damage to property, high claims costs incurred by the government, and poor quality of service from moving companies. Moreover, the program's data management system does not provide reliable information on the status of individual shipments or on the types of shipments and their costs. In its November 12, 2002, report to Congress, DOD made three recommendations aimed at improving its current personal property program.[Footnote 5] The three recommendations were to (1) reengineer the liability and claims process by adopting commercial practices of minimum valuation, simplifying the filing of claims, and providing direct settlement with the carrier; (2) change the acquisition process to implement performance-based service contracts; and (3) implement information technology improvements, which could interface functions across such areas as personnel, transportation, financial, and claims. DOD reported that the estimated cost of implementing its information technology improvements recommendation would be from $4 million to $6 million, and estimated that the cost of implementing the claims process and performance-based service contract recommendations would require an additional 13 percent over the current personal property program's costs. In our April 2003 report, we stated that the three recommendations contained in DOD's report were supported by the Transportation Command's evaluation of the pilot programs' findings and offered solutions to the long-standing problems that had plagued the current program for many years.[Footnote 6] Our review showed that the soundness of the methodologies DOD used to develop cost estimates for implementing the three recommendations varied. We found that the estimates DOD reported to Congress might understate the total initial cost for implementing the information technology improvements recommendation and contained a questionable adjustment for costs associated with the claims and contracting process recommendations. In its original cost assessment for the recommendations related to the claims and contracting processes, DOD made three adjustments to the average costs for the pilot programs: (1) reducing the average weight of shipments, (2) reducing costs to adjust for a mix of small and large businesses, and (3) reducing the pilot programs' costs to reflect anticipated savings based on "economies of scale." In our April 2003 report, we agreed that the first two adjustments were reasonable; however, we questioned the extent to which the third adjustment could be achieved. Part of our concern was based on DOD not adequately substantiating the expected economies of scale that supported its assumption of a 5 percent reduction in the average prices under the pilot programs when a full-scale program is implemented. Since DOD had not quantified the risk associated with its projected implementation cost estimate of a 13 percent increase over the current program's cost, the military services and Congress lacked information needed to develop and review future budget requests for the program. Therefore, we questioned the extent to which the recommendations could be implemented within DOD's projected cost estimate. As a result of our findings, we recommended that DOD take the following actions to improve its personal property program: (1) initiate actions that will implement the recommendations contained in DOD's report to Congress within budget constraints, (2) quantify the risk associated with achieving implementation of the recommended initiatives within the projected 13 percent cost estimate, (3) monitor costs during the implementation phase to ensure that the proposed changes are being achieved within an acceptable and a predefined range, and (4) assess the personal property program after the recommendations have been implemented to determine whether anticipated improvements are being achieved at a reasonable cost. DOD partially concurred with our second recommendation---to provide the military services and Congress with additional information to quantify the risk associated with achieving implementation of the recommended initiatives within the projected 13 percent cost increase. DOD believed it could incorporate the two recommendations into a new program within its proposed 13 percent cost estimate due to the conservative approach it took in developing the estimate. However, in May 2004, the House Armed Services Committee, in its report on the National Defense Authorization Act for Fiscal Year 2005, directed DOD to reevaluate its proposed cost estimate, quantify the risk or likelihood of achieving its goals within the 13 percent cost projection, and develop a range of possible cost increases associated with that risk. DOD issued a report addressing these congressional concerns on March 29, 2005.[Footnote 7] DOD Adequately Addressed Congressional Concerns: In addressing congressional concerns, DOD used a reasonable methodological approach to adequately reevaluate its cost estimate and quantify the risk associated with implementing the proposed personal property program initiatives within the 13 percent cost increase, including establishing a range of potential cost increases. DOD's analysis included three key components: (1) an assessment of any changes in the distribution of shipments among the services and the mix of continental and overseas household goods shipments, (2) new support for its assessment of anticipated savings through "full-program efficiencies" rather than "economies of scale," and (3) a simulation based on differing estimates of "full-program efficiencies" to determine the likelihood that DOD could implement its initiatives within its cost estimate. We believe that each of these components was necessary to reassess the reasonableness of DOD's previous cost estimate. DOD first validated the current mix of household goods shipments to ensure that the mix used in the original assessment was appropriate. DOD's reevaluation assessed changes, if any, in the distribution of shipments among the services and the mix of continental and overseas U.S. household goods shipments. Because the original cost estimate was based on household goods shipments picked up and delivered during the last half of fiscal year 2001, DOD needed to determine if the mix of household goods shipments had changed. Using data from its management information system, the Transportation Operational Personal Property Standard System, DOD found some minor differences in shipments among the services from the baseline year (fiscal year 2001) to fiscal year 2004, but concluded that the differences were insignificant and would have no effect on the calculations developed based upon the fiscal year 2001 data. This was a reasonable methodological approach to validate the mix of shipments. DOD then used a reasonable methodology to provide additional support for its assessment of the anticipated savings that would result from expanding the pilot to all of DOD. In its reassessment of the cost estimate, DOD reported that the anticipated savings could be better described as "full-program efficiencies" instead of "economies of scale." For example, DOD's reevaluation explained that prospective participants in the pilot programs faced limitations inherent in the pilot programs in a competitive market because they could not anticipate the longevity of the pilot program nor accurately predict the volume of potential shipments. Potential participants in the pilot programs also faced additional uncertainties that included not knowing their daily or monthly volume for this new program, having to project rates into option years without economic price adjustment, and facing possible penalties for withdrawing from the program. Consequently, the rates submitted by prospective participants reflected these uncertainties and other risk factors. It could reasonably be expected that as experience under the new program is gained and a steady-state program is reached, these risks would be reduced and would be reflected in lower rates. We believe that with this detailed description of the full-program efficiencies, DOD has provided reasonable additional support for the anticipated savings of the program, thereby reaffirming the 5 percent efficiency gains it previously reported. Finally, DOD used a risk-based simulation approach to provide a range of possible cost increases within statistical confidence intervals around the cost estimate, which is a reasonable methodology. This simulation quantified the risk associated with implementing the proposed initiatives within the 13 percent cost estimate. The results of this risk analysis indicated that about 80 percent of the time, DOD could implement the recommended initiatives to improve the current program within an increase of 15 percent or less above current program costs. All simulations used by DOD involved 10,000 iterations of six different cases of variation in the full-program efficiencies estimate to assess how different levels of full-program efficiencies affected the cost estimates.[Footnote 8] Table 1 summarizes DOD's simulation results for each scenario. As the table shows, only case two would result in 80 percent of all estimates being 12.5 percent or less--an amount lower than the original cost estimate. The remaining five simulations of the random effect of the full-program efficiencies contain the original 13.4 percent estimate within 90 percent confidence intervals. Table 1: Simulation Results Showing the Effect of Varying Program Efficiency Distributions on Achieving the Projected Cost Estimate: Case: 1; Program efficiency estimate: Low value: 5%; Program efficiency estimate: High value: 5%; Program efficiency estimate: Most frequent value: 5%; Program efficiency estimate: 80th percentile: 14.4%; 90% confidence interval for cost estimate: Low value: 11%; 90% confidence interval for cost estimate: High value: 15%. Case: 2; Program efficiency estimate: Low value: 0%; Program efficiency estimate: High value: 20%; Program efficiency estimate: Most frequent value: 5%; Program efficiency estimate: 80th percentile: 12.5%; 90% confidence interval for cost estimate: Low value: 4%; 90% confidence interval for cost estimate: High value: 15%. Case: 3; Program efficiency estimate: Low value: 0%; Program efficiency estimate: High value: 15%; Program efficiency estimate: Most frequent value: 5%; Program efficiency estimate: 80th percentile: 13.8%; 90% confidence interval for cost estimate: Low value: 7%; 90% confidence interval for cost estimate: High value: 16%. Case: 4; Program efficiency estimate: Low value: 0%; Program efficiency estimate: High value: 20%; Program efficiency estimate: Most frequent value: 2.5%; Program efficiency estimate: 80th percentile: 13.7%; 90% confidence interval for cost estimate: Low value: 4%; 90% confidence interval for cost estimate: High value: 16%. Case: 5; Program efficiency estimate: Low value: 0%; Program efficiency estimate: High value: 15%; Program efficiency estimate: Most frequent value: 2.5%; Program efficiency estimate: 80th percentile: 14.9%; 90% confidence interval for cost estimate: Low value: 8%; 90% confidence interval for cost estimate: High value: 17%. Case: 6; Program efficiency estimate: Low value: 0%; Program efficiency estimate: High value: 10%; Program efficiency estimate: Most frequent value: 5%; Program efficiency estimate: 80th percentile: 15.1%; 90% confidence interval for cost estimate: Low value: 10%; 90% confidence interval for cost estimate: High value: 17%. Source: GAO analysis of DOD data. [End of table] We reviewed the results of the simulation to assess its rigor and how well the simulation reflects the household goods shipment costs. Based on our review, a briefing of the methodology described in the report, and our knowledge of the analytical approaches for conducting such an assessment, we believe that DOD used a reasonable methodology that is consistent with professional standards to quantify the risk associated with implementing the program improvement initiatives within its cost estimate, because DOD ran a simulation that is a reasonable approach for assessing risk, using a commercially available program. See enclosure II for a detailed summary of the results of DOD's risk assessment. Agency Comments: DOD concurred with the content of this letter. DOD's letter is included in enclosure III. We are sending copies of this report to the appropriate congressional committees; the Secretary of Defense; the Commander, U.S. Transportation Command; and the Director, Office of Management and Budget. We will also make copies available to others upon request. In addition, the report will be made available at no charge on the GAO Web site at http://www.gao.gov. Please contact me at (202) 512-8365 or solisw@gao.gov if you or your staff have any questions concerning this report. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report were Jacqueline S. McColl, Arthur L. James, Jr., Charles W. Perdue, Karen N. Harms, Renee S. Brown, and Ann Borseth. Signed by: William M. Solis: Director, Defense Capabilities and Management: Enclosures: [End of section] Enclosure I: Scope and Methodology: To assess the methods used by the Department of Defense (DOD) to reevaluate the cost estimate and quantify the risk associated with implementing the program improvement initiatives within the stated cost projection, including establishing a range of possible cost increases, we reviewed the cost projection methodology used by DOD in its report on the reevaluation of the projected cost estimate. We also met with DOD officials and its contractor officials to discuss the methodology used to perform the cost reevaluation and risk assessment. To assess the reasonableness of the methodology used by DOD to perform these tasks, we compared DOD's reevaluation methodology to analytical approaches applicable for this type of evaluation, assessed adjustments made in the evaluation strategy to address issues that could affect the validity of the results, and reviewed the evaluation techniques used to analyze data. We reviewed the results of the simulation DOD used to conduct its risk assessment to assess its rigor and how well the simulation reflects the household goods shipment costs. We also reviewed the contents of the DOD contractor's briefings on the planned approach and evaluation techniques used to reevaluate the projected 13 percent cost estimate. We did not assess whether the anticipated benefits to be derived from implementing the three recommendations would warrant the additional costs that DOD estimates will be required to fund these improvements. Furthermore, we did not independently test the reliability of data DOD extracted from its data system to develop costs. During this review of DOD's evaluation efforts, we met with officials and obtained documents from the Office of the Assistant Deputy Under Secretary of Defense (Transportation Policy), Washington, D.C.; the U.S. Transportation Command, Scott Air Force Base, Illinois; the Surface Deployment and Distribution Command, Alexandria, Virginia; and LMI (Surface Deployment and Distribution Command contractor), McLean, Virginia. In addition to these agency meetings and documents, we drew upon information contained in a previous GAO report resulting from our prior review of this program. Our work for this review was performed from November 2004 through May 2005 in accordance with generally accepted government auditing standards. [End of section] Enclosure II: Summary of DOD's Risk Assessment: DOD's methodology to quantify the risk associated with the cost estimate included using a commercially available program to run a Monte Carlo simulation,[Footnote 9] which is a reasonable methodological approach for assessing risk. The simulation contained six cases of variation in the full-program efficiencies estimate to assess how these full-program efficiencies affected the cost estimates. Case one was the baseline case involving the original 13.4 percent increase resulting from an assumption of a fixed 5 percent reduction for full-program efficiencies. This case represents the original estimates provided by DOD in its November 2002 report to Congress. The remaining five cases randomly varied the assumption for the full-program efficiencies over five different ranges, from a very compact range between zero and 10 percent to a very diverse range from zero to 20 percent. Two cases in particular illustrated the effects of the simulation. Case six, the very compact case, allowed the frequency of the full-program efficiencies estimate to increase from zero smoothly to 5 percent and decline smoothly to 10 percent. The result of this simulation was very similar to the base case (case one, in which the original estimate of full-program efficiencies was held equal to 5 percent). The results indicate that 90 percent of the time the cost estimate would fall from 10 to 17 percent, with 80 percent of all estimates less than 15.1 percent. The most "generous" case, case two, was based upon the assumption of the frequency for full-program efficiencies increasing smoothly to 5 percent and declining smoothly to 20 percent. This case is most generous because it assumes that large cost savings (up to 20 percent) are achievable even though the most likely cost savings would be 5 percent. The results indicate that 90 percent of the time the cost estimate would fall from 4 to 15 percent, with 80 percent of all estimates less than 12.5 percent. This was the one case where the original estimate was outside the 80th percentile and thus a lower cost estimate was determined to be most likely. However, the case was most generous in its assumption of the frequency with which "large" full- program efficiencies would occur and could reasonably be expected to indicate a smaller increase in estimated costs over the baseline program. The other three cases, between the two extremes, varied the frequency of full-program efficiencies over different ranges and the simulation results contained the original 13.4 percent estimate within the 90 percent confidence intervals. Based on our review of DOD's analysis, we believe that DOD used a reasonable approach that was consistent with professional standards to quantify the risk associated with implementing the program improvement initiatives within its cost estimate. [End of section] Enclosure III: Comments from the Department of Defense: DEPUTY UNDER SECRETARY OF DEFENSE FOR LOGISTICS AND MATERIEL READINESS: 3500 DEFENSE PENTAGON: WASHINGTON, DC 20301-3500: June 2. 2005: Mr. William M. Solis: Director, Defense Capabilities and Management: U.S. Government Accountability Office: Washington, D.C. 20548: Dear Mr. Solis: This is the Department of Defense (DoD) response to the Government Accountability Office Draft Report, GAO-05-715R, "Defense Transportation: DoD Has Adequately Addressed Congressional Concerns Regarding the Cost of Implementing the New Personal Property Program Initiatives," dated May 20, 2005 (GAO Code 350619). The Department has reviewed the draft GAO report and concurs without comment. We appreciate the opportunity to review the draft report. Sincerely, Signed by: Bradley Berkson: Acting: [End of section] Related GAO Products: Defense Transportation: Monitoring Costs and Benefits Needed While Implementing a New Program for Moving Household Goods. GAO-03-367. Washington, D.C.: April 18, 2003. Defense Transportation: Final Evaluation Plan Is Needed to Assess Alternatives to the Current Personal Property Program. GAO/NSIAD-00- 217R. Washington, D.C.: September 27, 2000. Defense Transportation: The Army's Hunter Pilot Project Is Inconclusive but Provides Lessons Learned. GAO/NSIAD-99-129. Washington, D.C.: June 23, 1999. Defense Transportation: Plan Needed for Evaluating the Navy Personal Property Pilot. GAO/NSIAD-99-138. Washington, D.C.: June 23, 1999. Defense Transportation: Efforts to Improve DOD's Personal Property Program. GAO/T-NSIAD-99-106. Washington, D.C.: March 18, 1999. Defense Transportation: The Army's Hunter Pilot Project to Outsource Relocation Services. GAO/NSIAD-98-149. Washington, D.C.: June 10, 1998. Defense Transportation: Reengineering the DOD Personal Property Program. GAO/NSIAD-97-49. Washington, D.C.: November 27, 1996. FOOTNOTES [1] Department of Defense, U.S. Transportation Command Personal Property Pilot Programs Evaluation Report (Washington, D.C.: November 2002). [2] GAO, Defense Transportation: Monitoring Costs and Benefits Needed While Implementing a New Program for Moving Household Goods, GAO-03-367 (Washington, D.C.: Apr. 18, 2003). [3] H.R. Rep. No. 108-491, at 298 (2004). [4] The Military Surface Deployment and Distribution Command, a component of the U.S. Transportation Command, is the DOD executive agent responsible for managing the relocation process for servicemembers and their families. [5] Department of Defense, U.S. Transportation Command Personal Property Pilot Programs Evaluation Report. [6] GAO-03-367. [7] Department of Defense, Reevaluation of Cost Estimate for DOD Families First Program (Washington, D.C.: March 2005). [8] In its report, DOD noted that the advantage of the probability- based simulation technique used to conduct its risk assessment is that it can account for multiple input costs that have variability in risk in relationship to how they affect the overall cost estimate. [9] A Monte Carlo simulation is a method of evaluating hypotheses by developing a computer model of a process, defining the parameters of the process to reflect the "real world" situation, calculating multiple results of varying parameters through some range of values, and evaluating the distribution of results obtained from these samples. The Monte Carlo method assigns a value to the model's parameters from a sequence of random numbers, runs the model multiple times through randomly differing parameter values, captures the outcome of each iteration of the model, and assesses the distribution of outcomes using standard statistical methods.