This is the accessible text file for GAO report number GAO-03-367 entitled 'Defense Transportation: Monitoring Costs and Benefits Needed While Implementing a New Program for Moving Household Goods' which was released on April 18, 2003. This text file was formatted by the U.S. General Accounting Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products’ accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. Report to Congressional Committees: United States General Accounting Office: GAO: April 2003: DEFENSE TRANSPORTATION: Monitoring Costs and Benefits Needed While Implementing a New Program for Moving Household Goods: GAO-03-367: GAO Highlights: Highlights of GAO-03-367, a report to Congressional Committees. Why GAO Did This Study: The Department of Defense (DOD) spends more than $1.7 billion each year to move and store over 600,000 household goods shipments when relocating military personnel. It conducted and evaluated several pilot program studies aimed at fixing its problem-plagued program and, in 2002, issued a report to Congress with three recommendations. The 1997 Defense Appropriations Act Conference Report directed GAO to validate the results achieved by the pilot programs. In response, GAO examined the extent to which DOD’s recommendations to Congress (1) offer solutions to long-standing problems in the current program and (2) are supported by the evaluation’s findings and should be implemented. GAO also assessed the soundness of methodologies used by DOD to develop cost estimates to implement the recommendations. What GAO Found: The recommendations in DOD’s report to Congress have the potential to resolve several long-standing problems found in the current personal property program, which manages the transportation and storage of household goods. The recommendations, if implemented, would * reengineer the claims process to reduce the length of time it currently takes to resolve claims for lost, destroyed, or damaged household goods and increase the reimbursement rates that military personnel currently receive for their losses; * use performance-based service contracts to improve the generally low quality of service that DOD currently gets from the moving industry; and * put in place new information technology with interface capabilities to enable program managers and users to monitor in-transit shipments and track the number and cost of shipments processed each year. The recommendations in DOD’s report to Congress are supported by the Transportation Command’s evaluation of the pilot programs’ findings and should be implemented within budget constraints. DOD’s approach in conducting the evaluation was methodologically sound: It developed an evaluation plan to guide its work and adjusted the plan when necessary to address differences in the pilot programs’ approaches. While the shipments included in the evaluation do not represent all shipment types managed annually by DOD, GAO believes that the evaluation results provide sufficient information to allow DOD to initiate actions to improve its current personal property program. GAO found that the soundness of methodologies used to develop DOD’s cost estimates varied. Therefore, DOD’s ability to implement changes to the existing program within the cost estimates DOD reported to Congress is uncertain. GAO found that the estimated to implement the information technology recommendation was $7 million rather than the $4 million to $6 million estimate DOD reported to Congress. In developing cost estimates for the remaining recommendations, DOD did not provide the same level of evidentiary support for one of the three adjustments it used to align the pilot programs’ costs to current program costs. As a result, GAO questions the extent to which these recommendations can be implemented within DOD’s estimated 13 percent increase over current program costs. While DOD believes it used a conservative approach in developing this 13 percent estimate, it has not quantified the risk associated with the projection, which could provide the military services and Congress information needed to develop and review future budget requests for this program. Without providing the range of possible cost increases and the risk regarding the likelihood of achieving this 13 percent projection within that range, DOD may find a repeat of what occurred during the pilots, where the military services terminated participation in one of the pilot programs due to costs exceeding projections. GAO also found that without carefully monitoring costs during the implementation phase and assessing costs and benefits from a period succeeding full implementation of the recommendations, DOD would not have the information needed to determine if anticipated improvements in the program are being achieved at a reasonable cost. What GAO Recommends: GAO recommends that DOD implement the recommendations within budget constraints, quantify the risk associated with achieving its cost estimates, monitor costs during the implementation phase, and assess the new program to determine if anticipated improvements were achieved at a reasonable cost. DOD agreed with three recommendations, but did not agree with the need to quantify the risk associated with achieving its cost estimates. www.gao.gov/cgi-bin/getrpt?GAO-03-367. To view the full report, including the scope and methodology, click on the link above. For more information, contact William M. Solis at (202) 512-8365 or solisw@gao.gov. [End of section] Contents: Letter: Results in Brief: Background: DOD’s Recommendations Offer Solutions to Long-Standing Problems: Transportation Command’s Evaluation of Pilot Programs Supports DOD’s Three Recommendations: Ability to Implement New Program within Cost Estimates Reported to Congress Is Uncertain: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Scope and Methodology: Appendix II: Overview of Current Personal Property Program and Pilot Programs: Current Program: Military Traffic Management Command’s Reengineered Personal Property Program: The Department of Defense’s Full Service Moving Project: Navy’s Service Member Arranged Move Pilot Program: Appendix III: Comments from the Department of Defense: Related GAO Products: Tables: Table 1: Claims-Related Features of the Current Personal Property Program and Pilot Programs: Table 2: Quality of Service-Related Features of the Current Personal Property Program and Pilot Programs: Table 3: Data Reliability-Related Features of the Current Personal Property Program and Pilot Programs: This is a work of the U.S. Government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. It may contain copyrighted graphics, images or other materials. Permission from the copyright holder may be necessary should you wish to reproduce copyrighted materials separately from GAO’s product. United States General Accounting Office: Washington, DC 20548: April 18, 2003: Congressional Committees: Military personnel and their families can expect to relocate many times during a service member’s career. As the moving industry’s single largest customer, the Department of Defense (DOD) spends more than $1.7 billion annually for its personal property program, which provides household goods transportation and storage services for military personnel and their families when they relocate. The program manages more than 600,000 personal property shipments each year. DOD has experienced long-standing problems with its current personal property program, including excessive loss or damage to property, high claims costs incurred by the government, and poor quality of service from moving companies. Moreover, the program’s data management system does not provide reliable information on the status of individual shipments or on the types of shipments and their costs. In an effort to test alternative approaches and improve the quality of its personal property program, DOD has carried out several initiatives over the past 8 years. In 2000, the U.S. Transportation Command[Footnote 1] began to collect data from one of the three pilot programs to evaluate alternative approaches for improving the current program.[Footnote 2] The Transportation Command compared the features of the current program with those of the three pilot programs, and in June 2002 it submitted a report to the Deputy Under Secretary of Defense (Logistics) outlining its evaluation strategy, findings, and recommendations. In its November 12, 2002, report to Congress, DOD included the three recommendations resulting from the evaluation and the estimated additional costs required to implement program improvements based on a subsequent assessment of pilot and current programs’ operations. Our involvement in this issue stems from the conference report on the 1997 Defense Appropriations Act, which directed us to review the data collected from the pilot programs and validate the results before DOD expands any of the pilot programs, and a subsequent request from the Subcommittee on Readiness of the House Committee on Armed Services. This report supplements our previous testimony[Footnote 3] on DOD’s efforts to enhance its current program, as well as our status report[Footnote 4] and status briefings to Congress on DOD’s efforts to implement a methodologically sound evaluation of the pilot programs. As agreed with your offices, for this report we assessed the extent to which the recommendations in DOD’s November 2002 report (1) offer solutions to long-standing problems in the personal property program and (2) are supported by the evaluation’s findings and should be implemented. In addition, we assessed the soundness of the methodology used by DOD to develop the cost estimates for implementing the recommendations. To conduct our assessment of DOD’s three recommendations to Congress, we reviewed DOD and GAO prior reports on the personal property program, interviewed DOD officials and private-sector contractors involved in the current and pilot personal property programs and the Transportation Command’s evaluation, examined the methodology and findings of the Transportation Command’s evaluation of the three pilot programs, reviewed the methodology and data generated by each of the three pilot programs, and assessed the methodologies used to develop the cost estimates for implementing DOD’s recommendations. We did not make an assessment of whether the anticipated benefits to be derived from implementing the three recommendations would warrant the additional costs DOD projects will be required to fund these improvements. The scope and methodology we used in our review are described in further detail in appendix I. Results in Brief: The three recommendations in DOD’s report to Congress offer solutions to several long-standing problems in the current personal property program. Specifically, the recommendations address previously identified problems with the liability/claims process (including the lengthy claims process, reimbursement rates that may not fully compensate service members for loss and damage incurred during their moves, and high claims costs to the government), the low quality of shipping services, and the inability to track shipments and their costs. If implemented, DOD’s first recommendation--to reengineer the claims process--has the potential to reduce the length of time it currently takes to resolve claims for lost, destroyed, or damaged household property and increase the reimbursement rates that military personnel currently receive for their losses. The recommendation also has the potential to reduce the high cost of claims incurred by DOD. The second recommendation--to use performance-based service contracts- -has the potential to improve the generally low quality of service that DOD currently receives from moving companies. The third recommendation- -to put in place new information technology with interface capabilities--has the potential to enable the personal property program managers and other users of the program to reliably monitor in-transit shipments and track the number and the cost of shipments processed each year so that accurate reporting can be provided to DOD and Congress. The recommendations in DOD’s report to Congress are supported by the Transportation Command’s evaluation of the pilot programs’ findings and should be implemented within budget constraints. We found that the Transportation Command’s approach in conducting the evaluation was methodologically sound: It developed an evaluation plan to guide its work and adjusted the plan when necessary to address differences in the pilot programs’ approaches. While the shipments included in the evaluation do not represent all shipment types managed annually by DOD, we believe that the evaluation results provide sufficient information to allow DOD to initiate actions to improve its current personal property program. Our review showed that the soundness of the methodologies used by DOD to develop cost estimates for implementing the three recommendations varied. Therefore, DOD’s ability to implement changes to the existing program within the cost estimates reported to Congress is uncertain. Our review found that the total initial cost for implementing the information technology improvements recommendation would more likely be $7 million rather than DOD’s $4 million to $6 million estimate. We agreed that the premise of two of three adjustments DOD used to develop the 13 percent cost increase to implement the claims process and performance-based service contract recommendations was sound. However, we are less assured about the extent to which the projected cost savings related to a third adjustment may occur because the adjustment was not supported by historical experience or by the same quality of data provided for the other adjustments. While DOD believes it can incorporate the three recommendations into a new program within its proposed 13 percent increase due to the conservative approach it took in developing this estimate, we believe that by quantifying the risk associated with this projection, DOD could provide the military services and Congress information needed to develop and review future budget requests for this program. The need for this type of information is further supported based on the long-standing problems associated with the current program and the large increase in costs contributing to the military services’ decision to terminate participation in one of the pilot programs. Further, without carefully monitoring costs during the implementation process and assessing costs and benefits from a period succeeding full implementation of the recommendations, DOD will not have the information needed to determine if anticipated improvements in the program are being achieved at a reasonable cost. Currently, DOD is beginning planning efforts to implement the recommendations. These efforts do not include monitoring and evaluating costs and benefits during the implementation phase and post implementation of the recommendations in a new program. We are recommending that DOD initiate actions that will implement the recommendations contained in its report to Congress within budget constraints, quantify the risk associated with achieving its cost estimates, monitor costs during the implementation phase to ensure that the proposed changes are being achieved within an acceptable and a predefined range, and assess the personal property program after the recommendations have been implemented to determine whether anticipated improvements are being achieved at a reasonable cost. In commenting on a draft of this report, DOD concurred with three of our four recommendations, including initiating actions to implement the recommendations contained in its report to Congress, monitoring costs during the implementation phase, and assessing the program after DOD’s recommendations have been implemented. DOD partially concurred with the remaining recommendation, i.e., to provide the military services and Congress with additional information to quantify the risk associated with achieving the projected 13 percent cost estimate. In its response, DOD stated that the 5 percent reduction that it made to the pilot programs’ average costs to adjust for economies of scale/program efficiencies was reasonable and very conservative. Further, DOD continues to believe that the program can be implemented within the 13 percent increase and noted that one of the military services validated this estimate. Therefore, DOD did not see the value added in providing the military services or Congress a formal risk assessment. We find that these statements still do not provide a basis for the 5 percent reduction and do not indicate the level of risk associated with implementing the recommendation within this estimate. We continue to believe that this information needs to be developed to help the military services prepare their budgets. The military services terminated participation in one of the pilot programs because actual costs were significantly greater than the projections provided to them for budgetary purposes. Providing a measure of risk for the 13 percent projection could help prevent a repeat of what occurred during the pilot programs. Without this risk information, the military services will have to wait until after the moving industry submits bids to find out if DOD’s projection was reliable. Background: DOD’s personal property program is managed centrally by the Military Traffic Management Command headquarters and administered locally by about 200 military service and DOD transportation offices around the world. The program relies on more than 1,200 domestic commercial carriers and 150 freight forwarders for international shipments to provide household goods transportation and storage services for military personnel and their families when they relocate. The military services pay shipment and storage-related costs from their military personnel accounts and loss and damage claims and personal property shipment office expenses through their operations and maintenance accounts. The program has remained virtually unchanged for nearly 40 years. It involves a complex process of qualifying carriers, soliciting rates, distributing moves, evaluating transportation providers’ performance, paying invoices, and settling claims. Among the program’s many challenges is ensuring that the moving industry provides adequate year-round capacity, especially during the summer peak-moving season when most service members, as well as the general public, schedule their moves. In prior reports, both DOD and GAO have identified problems related to the loss and damage claims process and the low quality of service from movers. In designing and implementing its evaluation plan, the Transportation Command also noted that weaknesses in the current program’s data management system precluded DOD from being able to track shipments in transit and from being able to extract reliable data on the number and types of shipments managed annually and their associated costs. In response to the long-standing problems, DOD has undertaken a number of pilot program studies to find ways to improve the process of shipping service members’ household goods. In August 1996, the Deputy Under Secretary of Defense (Logistics) tasked the Transportation Command with evaluating alternative approaches to the current program and recommending changes in the program based on the results of its evaluation. The Transportation Command identified three ongoing or planned pilot programs to include in its evaluation and began to collect data for its analysis from one of them in 2000. These three pilot programs shared some common features, such as testing performance-based service contracts and providing full replacement (rather than depreciated) value for loss or damage. Each one also had some distinctive features, such as allowing service members to participate in selecting their movers and contracting out installation personal property shipment office functions to private-sector move manager companies. The three pilot programs are summarized below. Further information on each program, as well as DOD’s current personal property program, is provided in appendix II. * The Military Traffic Management Command’s Reengineered Personal Property Program operated from the military services’ and the Coast Guard’s installations located in North Carolina, South Carolina, and Florida. It used military installation personal property shipment office personnel, as the current program does, and developed a new data management system that tracked both the movement of individual shipments and information on the number and cost of shipments. * The Department of Defense’s Full Service Moving Project operated from the military services’ and the Coast Guard’s installations located in the National Capital Region (the Washington, D.C., metropolitan area), Georgia, and North Dakota. It contracted the management of shipments to private-sector companies and offered optional relocation services, such as referrals for rental assistance and purchase and sale of real estate services, to personnel participating in the pilot program. * The Navy’s Service Member Arranged Move Pilot Program operated from Navy installations located in the states of California, Connecticut, Virginia, and Washington. It designated current staff within the installation personal property shipment offices as “move coordinators” to provide assistance, allowed participants to pre-select transportation providers, and paid for moves through government purchase cards. In June 2002, the Transportation Command submitted a report containing its evaluation results and proposed three recommendations to the Deputy Under Secretary of Defense (Logistics). After reviewing the results and receiving comments from the military services, DOD submitted its report, dated November 12, 2002, to Congress. DOD’s report contained the same three recommendations contained in the Transportation Command’s report. DOD also provided cost estimates for implementing the recommendations. The three recommendations were to: * reengineer the liability/claims process by adopting commercial practices of minimum valuation, simplifying the filing of claims, and providing direct settlement with the carrier; * change the acquisition process to implement performance-based service contracts; and: * implement information technology improvements, which could interface functions across such areas as personnel, transportation, financial, and claims. In its report to Congress, DOD estimated that reengineering the liability/claims process and changing the acquisition process to implement performance-based service contracts would increase the current program’s estimated $1.7 billion cost by 13 percent. Implementing the information technology improvements to enhance its data management capabilities and to provide training to users was estimated at an additional $4 million to $6 million. DOD also estimated that efforts to implement the changes to the current program would be completed by the first quarter of fiscal year 2005. DOD has developed a plan of action and milestones for designing the new personal property program. This initial effort identifies several teams, which are exploring the following issues: the acquisition/ solicitation process, quality assurance, the liability/claims process, information systems technology, and electronic billing and payment. Four of these issues address the recommendations included in DOD’s November 2002 report to Congress. The plan identifies a list of essential activities needed to carry out the responsibilities required to build the future personal property program. It also includes time lines and identifies a process to monitor problems and delays. However, it does not include monitoring costs and benefits during the implementation phase and the extent the proposed changes are being achieved within an acceptable and a predefined range. Further, it does not include evaluating the extent the benefits from the pilot programs will be achieved after the new program is implemented to determine whether the anticipated improvements were achieved at a reasonable cost. DOD’s Recommendations Offer Solutions to Long-Standing Problems: The three recommendations in DOD’s report to Congress offer solutions to several of the current program’s long-standing problems, such as the liability/claims process and the low quality of service. These problems have been identified in DOD and GAO’s prior reports, as well as in surveys conducted as part of the pilot program evaluation. The inability to monitor shipments and shipping information has been long recognized and was highlighted as an additional problem during DOD’s evaluation. If implemented, the first recommendation (i.e., reengineering the liability/claims process by adopting commercial practices of minimum valuation, simplifying the filing of claims, and providing direct settlement with the carrier) has the potential to help reduce the length of time it currently takes to resolve claims for lost, destroyed, or damaged household goods because the carrier recovery time would be eliminated for most moves, increase the reimbursement rates military personnel receive for their losses, and reduce DOD’s claims-related costs. The second recommendation (i.e., changing the acquisition process to implement performance-based service contracts) has the potential to help improve the generally low quality of service that DOD currently receives from the moving industry. The third recommendation (i.e., implementing information technology improvements, which could interface functions across areas such as personnel, transportation, financial, and claims) has the potential to improve the program’s ability to reliably monitor and collect data on the status and costs of shipments so that accurate reporting can be provided to DOD and Congress. Reengineered Liability/Claims Process Has the Potential to Shorten Time, Increase Reimbursement Rates, and Reduce Claims-Related Costs to DOD: As part of its evaluation, the Transportation Command cited that one of the long-standing problems with military household goods shipments is the liability/claims process, including the (1) length of time it takes to resolve claims, (2) low reimbursement rates, and (3) high cost of claims that DOD must pay. Length of Time to Resolve Claims: In a study conducted in 1999, the Military Traffic Management Command reported that 146 days are expended between the time a claim is filed by a service member to recovery of costs from the carrier by the government. During this period, military personnel file their claims for lost, destroyed, or damaged household goods with their respective military service’s claims offices and receive settlements (this occurs, on average, within 23 days), and then these offices file the claims against the carriers to recover the costs (this step is completed within the 146 day period).[Footnote 5] In the Transportation Command’s pre-evaluation survey completed in 2000, responses from military personnel who had recently moved indicated that one of the lowest performance ratings involved the time required to settle a claim. Based on the Transportation Command’s evaluation of the claims process under the pilot programs, one of the results from implementing the pilot programs was the 146 day average required under the current program to settle claims and recover costs was reduced to an average of 30 days since the service member filed directly with the carrier and the military services did not have to recover costs. Under each pilot program, military personnel settled claims directly with the carriers. Service members who were not satisfied with offers made by the carriers could file their unresolved claims directly with DOD. Military services worked these claims with the carriers and if a claim was justified, the service member received just settlement under the pilot programs (i.e., if the items were lost or destroyed, the member received full replacement value, while damaged items were repaired). Low Reimbursement Rates for Loss and Damage: In its pre-evaluation survey, the Transportation Command found that military personnel cited low reimbursement amounts that typically do not cover the loss or damage of household goods as a major concern during their moves. Under the current program, a carrier’s liability is limited to $1.25 per pound multiplied by the shipment weight. Personnel receive only the depreciated value of lost, destroyed, or damaged items, up to a maximum of $40,000 per move.[Footnote 6] When arranging their moves under the current program, military personnel can buy increased insurance coverage from their carrier, up to a full replacement value limit of $3.50 per pound times the shipment weight, at a cost of 85 cents per $100 of the stated value of the shipment. However, only military personnel making moves within the continental United States can buy this additional coverage; it is not available to those moving to or from overseas posts. Another option that military personnel have to increase their protection for loss and damage is to buy additional coverage from private-sector sources. Unlike the current program, the pilot programs provided full replacement value for lost and destroyed goods, with maximum amounts ranging from $63,000 to $75,000. Damaged items were repaired. Two of the three pilot programs reported that their cost per pound times the shipment weight rates were $3.50 for $63,000 maximum coverage and $6.00 for $75,000 maximum coverage, respectively. The remaining pilot program did not give a cost per pound, stating only that its maximum coverage rate was $72,000. High Costs to Government for Claims: DOD has reported that, historically, approximately 35 percent of all moves result in loss or damage claims. A 1997 Military Traffic Management Command survey of 3,000 moves revealed that while 65 percent of shipments had loss or damage, only 35 percent resulted in claims being filed. DOD pays approximately $100 million a year in claims but recovers only 60 to 65 percent of the amounts paid to military personnel from the moving industry. These figures understate the actual loss and damage, since all military personnel do not file claims, apparently because the process takes a long time and reimbursement rates do not always cover the losses. DOD incurs these losses due to the structure of its current program. The military claims offices assist service members by arranging to pay their claims and then submitting the claims to the respective movers for reimbursement. As indicated above, DOD receives only partial reimbursement from the moving industry. If the recommendation is implemented, DOD expects to reduce a substantial portion of the estimated $100 million it currently pays in claims each year to service members and eliminate much of the 35 to 40 percent in losses it incurs from settling claims with the moving industry because service members will be settling claims directly with their carriers. DOD also expects additional savings because fewer demands would be placed on military claims officials to manage the claims process. DOD believes that these savings will help offset the higher costs of providing full replacement value to service members for any loss and damage incurred during the shipment and storage of their personal property. Change to Performance-Based Service Contracts Could Improve Quality of Moving Services: Our work has shown that another long-standing problem with the current personal property program is the poor quality of moving services provided to military personnel. The high number of loss and damage claims that military personnel file underscores this problem. According to the two pre-evaluation surveys cited in the Transportation Command’s evaluation, around 55 to 65 percent of respondents reported suffering some loss or damage of household goods during a recent move. Moreover, in the pre-evaluation survey conducted by the Transportation Command, the top four factors identified by service members as being of greatest importance to them in the moving process were the quality of packing, the care in handling personal property, the condition of their property upon receipt at the end of the move, and the receipt of fair payment for any losses or damages they suffered. In individual comments obtained during the pre-evaluation survey, the Transportation Command reported that some service members also cited the lack of professionalism and quality of customer service on behalf of moving crews as a concern. The problem stems primarily from the current program placing greater emphasis on costs (i.e., the lowest bids) than on the quality of service that carriers provide when moving shipments of military household goods. While the current program established its Total Quality Assurance Program to measure quality, data collected to develop scores for each carrier includes three measures (timeliness of pickup, timeliness of delivery, and reported loss and damage), which are not collected for all household goods shipments. The best indication of quality, customer satisfaction, is not measured in the current program. The problem of quality is further exacerbated by the program’s use of a 20 year-old tariff schedule that carriers use in developing their bids. This tariff contains lower rates than the current commercial tariff used during the pilot programs. Unlike the current program, the pilot programs screened carriers that wanted to participate in their programs by emphasizing the quality of carriers’ prior performance rather than the amount of their bids. For example, the Full Service Moving Project contracted a financial services company to conduct a financial and performance assessment of potential movers. The pilot program emphasized best value and placed more emphasis on performance (70 percent) than cost (30 percent) in determining which providers were awarded shipments. The pilot programs showed that these types of contracts could allow the government to pre-screen carriers for financial viability and, more importantly, to institute and maintain a quality assurance process to reduce losses and improve service. In addition to prescreening carriers for quality control purposes, the pilot programs also surveyed military personnel who participated in the programs and used the results to distribute future shipments to carriers that received the best performance scores. To address concerns about the obsolete tariff schedule, the pilot programs adopted current commercial tariffs for carriers to use in establishing their bids. The pilot programs also showed that the solicitation process could be streamlined by eliminating detailed statements of work and that the pilot programs could place responsibility for successful performance on carriers, allowing the government to focus on outcomes, rather than processes. Finally, the pilot programs demonstrated that using these types of performance-based service contracts did not have an adverse effect on small business participation, a major concern of the moving industry. On the basis of the total dollar value of shipments, each pilot program exceeded the Small Business Administration’s goal of 23 percent participation for the industry. Specifically, 48 percent of the Military Traffic Management Command’s Reengineered Personal Property Program’s revenues, 74 percent of the Full Service Moving Project’s revenues, and 100 percent of the Navy’s Service Member Arranged Move’s revenues went to small businesses. Implementing Information Technology Improvements to Address Data Reliability Problems: Another ongoing problem with the current personal property program is its inability to provide reliable data on the status of in-transit shipments or on the number and associated costs of shipments managed by DOD each year. Because of the lack of reliable data on shipments and costs, program managers have no way of knowing the actual costs of moving military personnel’s household goods. In addition, they have no access to real-time tracking data that they could use to manage transportation and storage costs and to help cut down on the need for temporary storage by reducing the number of failed deliveries. Two of the pilot programs included features to address the problems associated with the current program’s stand-alone data management system. The pilot programs each developed a Web-based data management system to enhance the visibility of individual shipments and provide more reliable data on shipments and costs. For example, the Military Traffic Management Command’s Reengineered Personal Property Program’s data management system provided in-transit visibility. This made it possible to track the status of individual shipments and gave real-time access to those sections of the shipment records that various parties involved in the relocation process needed for data entry or status review. The pilot program’s data management system provided a complete picture of the service member’s move from start (the move application process) to finish (the claims submission and resolution process). In addition, the data management system demonstrated the potential to provide information to personnel in various functional areas involved in the service members’ relocation process (such as personnel, transportation, financial, and claims). Finally, the data management system demonstrated the potential to provide data for planning and budgeting purposes on the types of shipments made annually across DOD and their costs. The Full Service Moving Project’s data management system was developed but not fully implemented because the military services terminated their participation in the pilot program due to its high costs. While the Navy’s pilot program developed a database near the end of the Transportation Command’s evaluation, the database was not fully implemented nor assessed as part of the evaluation. Transportation Command’s Evaluation of Pilot Programs Supports DOD’s Three Recommendations: Our analysis indicated that DOD’s three recommendations are supported by the results of the Transportation Command’s evaluation of the three pilot programs. The Transportation Command adopted a sound methodology to conduct its evaluation, and it adjusted this methodology when circumstances warranted. The results of the Transportation Command’s evaluation are based on data collected from a limited number of geographical areas. While the shipments included in the evaluation do not represent all shipment types managed annually by DOD, we believe that the evaluation results provide sufficient information to allow DOD to initiate actions to improve its current personal property program within budget constraints. Transportation Command Implemented a Methodologically Sound Evaluation: We found that the Transportation Command used a methodologically sound approach to evaluate the results of the three pilot programs and make its recommendations. Before it started the evaluation process, the Transportation Command considered some lessons learned that had emerged from our review of the Hunter Pilot Program in 1999,[Footnote 7] and it followed through with several of them. For example, it obtained assistance from a contractor to design an evaluation plan that met professional standards. The Transportation Command identified four aspects, or factors, of the property program that served as the focus of its evaluation (i.e., quality of life, total costs, small business participation, and process improvements). The evaluation plan also prescribed that only one quality of life survey be administered to each participating service member in order to avoid survey “fatigue” that can result from subjecting a person to multiple surveys, and thus avoid the resulting potential for questionable results. In designing the evaluation plan, the Transportation Command incorporated a number of important evaluation features. These features included assessing the four factors consistently across all three pilot programs, ensuring that the evaluation received data from the pilot programs during the same time period to avoid the need to make adjustments due to potential changes in carrier operations and costs, conducting a survey of service members using the current program to establish a baseline from which to measure the pilot program results, and developing a method to provide estimates of what DOD would have paid for comparable shipments under the current program for those shipments completed under the pilot programs. The Transportation Command made appropriate adjustments to the evaluation plan when it learned that the three pilot programs would not be underway at the same time and that they would not provide all of the information originally outlined in the plan. For example, the Full Service Moving Project began later and terminated earlier than was expected, the Navy’s Service Member Arranged Move Pilot Program failed to conduct a quality of life survey and collect cost data as outlined in the evaluation plan, and none of the three pilot programs provided costs associated with individual process improvements. The Transportation Command included qualitative analytical techniques so that it could include as much information on each pilot program as possible in its evaluation while also dealing appropriately with data limitations. The Transportation Command also shifted the evaluation focus from the individual pilot programs to specific features from the three programs, such as full replacement value for loss and damage and the screening process for carrier participation. Survey and Analytical Data Support Command’s Recommendations: Our work indicated that the Transportation Command’s analysis of data collected from the three pilot programs supports the three recommendations that DOD included in its report to Congress. The Transportation Command’s analysis of household goods shipment data from the pilot programs showed that the average amount of time that service members and DOD spend to settle claims and recover costs from carriers fell dramatically in all three pilot programs. In comparison with the current program’s 146-day average, it took only 30 days, on average, to settle a claim under the Reengineered Personal Property Program and the Full Service Moving Project and fewer than 14 days under the Navy’s program. Survey results indicated that full replacement (rather than depreciated) value, direct claims settlements, and anticipated improvements in the claims process accounted for the highest increases in satisfaction. Based on experiences during the pilot programs, DOD believes that direct claims settlement between service members and carriers should reduce claims costs DOD currently incurs. Under the current program, DOD must collect from the carriers after it has paid the service members’ claims. DOD expects that this step will be eliminated in most instances because it is anticipated that service members will be resolving most of their claims directly with their carriers. The Transportation Command’s analysis of process improvement data, interviews and observations during site visits, and survey results from the pilot programs supported DOD’s recommendation to use performance- based service contracts to improve the quality of services that the moving industry provides to the military. The process of prescreening carriers desiring to participate in the pilot programs on the basis of their financial viability and past performance helped to eliminate poor performers. Furthermore, the pilot programs’ use of post-move surveys allowed them to get immediate and continuous feedback on the carriers’ performance and to use this information to distribute future work to those carriers with the highest performance ratings and best value. In addition, two of the pilot programs reduced the amount of paperwork associated with soliciting proposals and approving carriers. Finally, the Transportation Command’s review and observations of two of the pilot programs’ Web-based data management systems supported DOD’s recommendation to overhaul the current personal property program’s computer system (the Transportation Operational Personal Property Standard System). The Transportation Command found that the Reengineered Personal Property Program’s data management system significantly improved communications between the various DOD offices and the moving industry. The system gave real-time access to shipment records to DOD’s personal property shipment offices, certifying officers, prepayment auditors, military service headquarters, and military service claims offices and finance centers, as well as moving industry participants. Similar results occurred with the Full Service Moving Project’s Best Value Distribution Database system, but the military services terminated their participation in this pilot program before the system’s full potential could be demonstrated. While the shipments included in the evaluation do not represent all the shipment types managed annually by DOD, we believe that the evaluation results provide sufficient information to allow DOD to initiate actions to improve its current personal property program. Ability to Implement New Program within Cost Estimates Reported to Congress Is Uncertain: Our review found that the estimates DOD reported to Congress might understate the total initial cost for implementing the information technology improvements recommendation and contain a questionable adjustment for costs associated with the claims and contracting process recommendations. Also, DOD did not quantify the risk associated with implementing these latter recommendations within its projected 13 percent increase over the current program’s cost. Therefore, the ability to implement changes to the existing program within the cost estimates reported to Congress is uncertain. Based on our discussions with Military Traffic Management Command officials and review of available documents, we concluded that the total initial cost to implement the information technology improvements recommendation will more likely be $7 million rather than the $4 million to $6 million estimate that DOD previously reported to Congress. In its response to a draft of this report, DOD maintained that the costs to implement a new Web-based data management system would fall within its initial cost estimate of $4 million to $6 million. DOD’s projected cost estimate includes $5 million for development and implementation of the new system and $500,000 each for user training and system verification and validation testing. At a minimum, based on these projected cost estimates, the initial cost to implement the information technology improvements recommendation would more likely be $6 million. While we concur with the premise of two of the three adjustments used to develop the 13 percent cost increase to implement the remaining recommendations, we are less assured in the extent to which the projected savings related to the third adjustment may occur. We found that the first two adjustments were based on historical data. However, we question the rationale DOD used to develop the third adjustment, as the savings associated with this adjustment are based on assumed cost reductions resulting from changes in program operations. Also, these reductions lack the same quality of evidentiary support as DOD provided for the other two adjustments. DOD believes it took a conservative approach in developing the savings in each of the three adjustments; therefore, it assumes that the proposed changes to claims and the contracting process can be achieved within the 13 percent increase over the current program’s costs. Due to the long-standing problems with this program and the high pilot program costs that contributed to the military services’ early termination of participating in one of the pilot programs, we believe that by quantifying the risk associated with this projection, DOD could provide the military services and Congress information needed to develop and review future budget requests for this program. Further, without carefully monitoring costs during the implementation process and assessing costs and benefits from a period succeeding full implementation of the recommendations, DOD will not have the information needed to determine if anticipated improvements in the program are being achieved at a reasonable cost. Currently, DOD is beginning planning efforts to implement the recommendations. These efforts do not include monitoring and evaluating costs and benefits during the implementation phase and post implementation of the recommendations in a new program. Costs to Implement Information Technology Improvements Vary: The information DOD has provided on costs to implement the information technology improvements recommendation varies. Information provided during our review indicated that the total initial cost to improve the current data management system would be higher than the $4 million to $6 million DOD included in its report to Congress. DOD worked with the contractor who developed the Reengineered Personal Property Program’s Web-based data management system to develop an estimate of the cost to expand the capabilities tested during the pilot program. Also included in this estimate were funds to provide training for users of the new system. Based on our discussions with officials from the Military Traffic Management Command and our review of available documents, we concluded that these costs would more likely be $6 million, as the data management system development cost was projected to be $5 million with an additional $1 million for user training. The need for this training as part of a new personal property program was identified during DOD’s evaluation of the pilot programs. We increased our overall projections for the cost of the new system to $7 million when we learned that DOD planned to continue spending at least another $1 million annually for independent verification and validation testing and contractor support. This latter expense was identified to us during discussions following DOD’s submission of its report to Congress. In its response to a draft of this report, DOD maintained that the costs to implement a new Web-based data management system would fall within its initial cost estimate of $4 million to $6 million. It projected a cost of $5 million for system development and implementation and $500,000 each for user training and initial system validation. At a minimum, based on these projected cost estimates, the initial cost to implement the information technology improvements recommendation would more likely be $6 million. Because we did not assess the sufficiency of DOD’s original estimates of $1 million each for training and validation testing, we are unable to assess the impact of the reduction on the improvements in information technology across DOD. Based on our discussion with DOD officials, we learned that the plan is to implement this recommendation regardless of the status of the other two recommendations because managers and users of the program need more reliable information to manage the program’s shipments and their costs. Funds to implement this recommendation would come from the military services’ operations and maintenance accounts. Soundness of Estimates to Develop Cost for Changing Claims and Contracting Processes Varies: The soundness of the three adjustments the Military Traffic Management Command used to develop its estimated 13 percent increase over the current program costs to implement the remaining recommendations--the claims process and performance-based service contracts--varies. We found that two of these adjustments are based on reasonable assumptions and are supported by historical experience and by data. The savings associated with the third adjustment are based on assumed cost reductions resulting from changes in program operations and lack the same quality of evidentiary support as DOD provided for the other two adjustments. Therefore, we are less assured in the extent to which the savings associated with this adjustment may occur. Finally, we found that in its report to Congress, DOD did not quantify the risk of achieving these recommendations within the projected 13 percent increase. This information is important to the military services as they develop their military personnel and operations and maintenance budget requests and to Congress as it assesses the reasonableness of these requests. Estimate Includes a Questionable Cost Adjustment: In developing the 13 percent estimate, the Military Traffic Management Command determined that three adjustments to the average costs for the pilot programs were required to develop the cost for the full rollout of a new personal property program. The first two adjustments (i.e., reducing the average weight of shipments and reducing costs to adjust for a mix of small and large businesses) were made to offset differences between the pilot programs’ shipments and those more typically managed across DOD. The third adjustment was made to reduce the pilot programs’ costs to reflect anticipated savings based on economies of scale.[Footnote 8] In developing these adjustments, the Military Traffic Management Command worked with a contractor and consulted with officials from the military services and moving industry associations. While we believe that the shipment weight and small business mix adjustments are reasonable, we question the extent to which the economies of scale or program efficiencies adjustment may be achieved. For the weight adjustment, the Military Traffic Management Command determined that the average weights of moves in the two pilot program areas were higher than those experienced in typical departmentwide moves. As a result, the Military Traffic Management Command reduced the pilot programs’ average weights to reflect the lower, more typical weights to be used in calculating a total cost for a departmentwide program. This adjustment resulted in a 12 percent drop in average costs. We found the approach of using historical data to more accurately reflect the typical shipment weights to be reasonable. Next, the Military Traffic Management Command further lowered the pilot programs’ average costs because the pilot programs had higher small business participation rates than the departmentwide average, and small businesses are typically more expensive than large businesses. Small businesses accounted for 48 percent of the cost of all moves under the Reengineered Personal Property Program and 73 percent under the Full Service Moving Project. In addition, small businesses were 14 percent more expensive per shipment in the Reengineered Personal Property Program and 74 percent more expensive in the Full Service Moving Project than what each pilot program paid to large businesses. In developing its departmentwide estimate, the Military Traffic Management Command used a small business participation target rate of 30 percent. This 30 percent target rate is higher than the Small Business Administration’s 23 percent goal for government agencies conducting business with this industry. On the basis of this lower participation rate, the Military Traffic Management Command reduced the pilot programs’ average costs further by 8 percent. We agree that this adjustment in costs based on differences in the pilot programs’ small business participation rate and the new 30 percent goal is a reasonable way to reflect the differences between the pilot programs’ costs and the departmentwide-projected costs. We found that the third adjustment that the Military Traffic Management Command made--to reduce the cost of departmentwide shipments because of economies of scale or program efficiencies--was not adequately supported based on either historical experience or data that DOD later provided. The Military Traffic Management Command reduced the pilot programs’ average costs by 5 percent on the assumption that: * the pilot programs’ shipments involved only a limited number of providers; : * the pilot programs only included a limited number of shipments while the current program manages over 600,000 shipments annually; : * more accurate and timely management data that includes service member counseling, reduced losses, and storage and indirect costs will result in a more efficient program; and : * overhead and operating costs will be spread due to a larger volume of shipments. : While recognizing that some changes may result from these anticipated program efficiencies, the effect of these changes on potential cost savings is uncertain at this time. The Military Traffic Management Command did not provide the same level of evidentiary support that it provided on the other two adjustments. Further, we believe that only time will determine if DOD’s assumption for this adjustment, in particular, proves to be correct. DOD Did Not Quantify the Risk for Its Cost Estimate: We found that DOD has not provided a level of assurance to the military services and Congress that its projected 13 percent increase over the current program’s cost can be achieved. Quantifying the risk associated with this projection could provide the military services assurance of the viability of the projected 13 percent increase as they prepare budgets to support the increased cost for this program. Congress could also use this information as it reviews DOD’s requests for additional funds to implement changes in this program. The need for this type of information is further supported based on the long-standing problems associated with the current program and the fact that shipment and storage costs under the pilot programs were significantly higher than those that DOD estimated it would have paid under its current program in the same geographical areas. These costs ranged from 31 to 32 percent higher under the Reengineered Personal Property Program and from 51 to 54 percent higher under the Full Service Moving Project.[Footnote 9] These higher-than-anticipated costs contributed to the military services’ decision to terminate their participation in the Full Service Moving Project before its test period ended. While DOD did not quantify the risk, per se, it believes a conservative approach was taken in developing the savings in each of the three adjustments. As a result, DOD assumes that the proposed changes to the claims and contracting processes can be achieved with its projected increase of 13 percent over the current program’s budget. We still believe that the Military Traffic Management Command could have quantified the risk and provided this additional information to the military services and Congress as additional assurance of the likelihood of implementing the two recommendations within its projected 13 percent increase. The need for this information is further supported based on the long-standing problems DOD has experienced in this program, the fact that the military services terminated participation in one of the pilot programs due to the high cost increases, and the need to determine whether the proposed additional funds from military personnel and operations and maintenance accounts will be sufficient to implement the recommendations. In addition to the information that could be gained from quantifying the risk of its cost projection, we believe that only by careful monitoring during the implementation phase will DOD be able to ensure that the proposed changes are being achieved within an acceptable and a predefined range. Further, while we believe that the evaluation results support implementing plans to enhance the current program, it should be noted that the pilot programs’ shipments included in the evaluation were not typical of all types of shipments managed annually. Therefore, DOD was precluded from projecting the extent to which the recommended improvements can be achieved DOD-wide. Unless a subsequent evaluation is undertaken after the recommendations have been implemented, DOD will not be able to assess the extent to which the projected benefits are being achieved for military personnel, their families, and DOD, and whether the benefits are being achieved at a reasonable cost. Selecting an evaluation period to include the peak-moving season would also provide DOD with the information its needs to determine if the proposed changes can be achieved during the summer, when the demand for moving services by DOD and the private sector is at its highest. Conclusions: The three recommendations DOD developed from its evaluation of the current and pilot programs, if implemented successfully, could enhance the quality of life for relocating service members and their families; reduce claims-related costs to DOD; and resolve problems related to the reliability of management information on the status of shipments and on the quantity, types, and costs of shipments that DOD and the military services manage annually. Delaying implementation of the recommendations only prolongs problems military personnel, their families, and DOD experience under the current program. DOD has not quantified the risk associated with achieving its projected 13 percent increase over the current program’s costs to implement the claims process and performance-based service contract recommendations. Without quantifying the risk, the military services and Congress cannot be assured that these recommendations can be achieved within this estimate or whether additional funding or trade-offs may be needed. Further, without careful monitoring during the implementation phase, DOD will not be able to ensure that the proposed changes are being achieved within an acceptable and a predefined range. Because the pilot programs’ shipments included in the evaluation were not typical of all types of shipments managed annually, it was not possible for DOD to project the extent to which the recommended improvements can be achieved departmentwide. Without evaluating the program following implementation of the recommendations, DOD will be unable to assess the extent to which the projected benefits for military personnel, their families, and DOD are being achieved and, if so, whether they are being achieved within a reasonable cost. Also, if DOD does not select an evaluation period that includes the peak-moving season, it will not have the information needed to determine if the proposed changes can be achieved in the summer, when the demand for moving services is at its highest. Recommendations for Executive Action: To improve the personal property program for military personnel, their families, and program administrators, we recommend that the Secretary of Defense direct the Commander, U.S. Transportation Command, to: * initiate actions to implement the three recommendations contained in DOD’s report to Congress within budget constraints, : * provide the military services and Congress additional information to quantify the risk associated with achieving the projected 13 percent cost estimate before the claims process and performance-based service contracts recommendations are implemented to provide the military services with information needed for budgeting purposes, : * monitor costs for all recommendations during the implementation phase to ensure that the proposed changes are being achieved within an acceptable and a predefined range, and : * assess the effects of the three recommendations on the personal property program (to be carried out after the summertime peak-moving season once the recommendations have been implemented) to determine whether the anticipated improvements in the program are being achieved at a reasonable cost. Agency Comments and Our Evaluation: In commenting on a draft of this report, DOD concurred with three of our four recommendations. For the first of these recommendations, DOD stated that it is developing a plan to implement those recommendations it reported to Congress and anticipates its recommendations will be implemented by the end of the first quarter of fiscal year 2006, assuming the military services receive the additional funds needed to fund program enhancements. In response to our recommendation to monitor costs during the implementation phase, DOD stated that rolling out the new program will require monitoring of costs to determine if the moving industry partners are submitting bids that will allow DOD to enhance this program within the projected 13 percent cost increase. Further, DOD plans to include a process to conduct a rate reasonableness analysis upon receipt of the rates. For rates found to be outside the range of reasonableness, carriers will be given one opportunity to resubmit their rates. DOD plans to only use those rates determined to be reasonable in the new program. DOD also plans to include metrics, target/benchmark performance indicators, and a methodology for data collection in an updated program of action and milestone plan. For our recommendation, i.e., assess the effects of the three DOD recommendations on the personal property program to determine whether the anticipated improvements in the program are being achieved at a reasonable cost, DOD plans to collect data needed to determine if anticipated improvements have been achieved on a continuing basis. DOD plans to use customer satisfaction surveys in developing carrier performance ratings, which will be established quarterly, with the exception of the peak season, when performance ratings will be established monthly. If properly implemented, we believe the proposed DOD actions will sufficiently address these recommendations. DOD partially concurred with the remaining recommendation, i.e., provide the military services and Congress with additional information to quantify the risk associated with achieving the projected 13 percent cost estimate to provide the military services with information needed for budgeting purposes. DOD continues to believe that the 5 percent reduction it made to pilot programs’ average costs to adjust for economies of scale/program efficiencies was reasonable and very conservative and that the program can be implemented within the projected 13 percent increase over current program costs. DOD also reported that one of the military services validated the 13 percent cost increase following our audit. Further, DOD stated that it did not see value added in providing the military services or Congress a formal risk assessment but will continue to work with the military services as execution progresses to make sure they have all information required for budget purposes. Additionally, while not part of this recommendation, DOD also said it did not concur with our finding that the cost estimate for implementing its information technology improvements recommendation would be $7 million. In reviewing the response, we found that DOD still did not provide any data to support its assumption of a 5 percent cost savings from economies of scale/program efficiencies. DOD stated that the new program will be about 200 times larger than the pilot programs and that the resulting increase in volume will lower the cost per unit, a standard and accepted law of economics. While we agree that the cost may decrease, it may also increase or remain unchanged. Moreover, the cost may decrease by less than 5 percent. Without specific data showing the per move costs will decrease as the scale of operations increase, we continue to question the basis for DOD’s assumption of a 5 percent reduction. We believe that the validation effort completed by one of the military services, along with the calculations and assumptions DOD used in developing the 13 percent cost estimate, does not provide the military services and Congress with information needed to reliably develop and review budget requests to fund enhancements to the current program. We continue to believe that DOD needs to qualify this estimate with a measure of the risk associated with implementing its recommendations. Without providing the range of possible cost increases and the risk regarding the likelihood of achieving this 13 percent projection within that range, DOD may encounter a repetition of its experience with one of the pilot programs, which had to be terminated because actual costs exceeded projected costs. Absent this risk information, the military services will have to wait until after the transportation providers submit their bids in order to learn whether the recommendations can be implemented within the 13 percent projection. Should the bids result in costs that exceed this estimate, DOD and the military services will need to make adjustments to ensure that the recommendations are implemented within funding limits. Therefore, we continue to believe that our recommendation has merit. Our finding that the implementation of the information technology improvements recommendation would likely cost $7 million rather than the $4 million to $6 million that DOD projected was based on information we received from DOD during the audit. Specifically, we calculated that the costs to develop and implement the new system would be about $5 million and that training for users of the enhanced system would cost an additional $1 million. DOD had identified the need for this training during its evaluation of the pilot programs. After DOD submitted its report to Congress, it identified another potential cost- -an additional $1 million for independent verification and validation testing of the system. Our $7 million estimated included all three of these cost elements. In its response to a draft of this report, DOD maintained that its costs estimate would fall within its initial cost estimate of $4 million to $6 million, including $5 million for system development and implementation and an additional $500,000 each for user training and system verification and validation testing. At a minimum, based on these projected cost estimates, the initial cost to implement the information technology improvements recommendation would more likely be $6 million. However, since we did not originally assess the sufficiency of the $1 million estimates for training and testing, we are unable to assess what impact DOD’s reduction for these costs to $500,000 would have on the implementation of the system across DOD. We have reflected DOD’s changes in the body of our report. DOD’s comments are reprinted in appendix III. DOD also provided technical comments, and we revised our report to reflect them where appropriate. We performed our review from April 2002 through February 2003 in accordance with generally accepted government auditing standards. Appendix I contains the scope and methodology for this report. DOD’s comments are reprinted in their entirety in appendix III. We are sending copies of this report to the appropriate congressional committees; the Secretary of Defense; the Commander, U.S. Transportation Command; and the Director, Office of Management and Budget. We will also make copies available to others upon request. In addition, the report will be made available at no charge on the GAO Web site at http://www.gao.gov. Please contact me at (202) 512-8365 or Lawson Gist, Jr., at (202) 512- 4478 if you or your staff have any questions concerning this report. Key contributors to this assignment were Robert L. Self, Jacqueline S. McColl, Arthur L. James, Jr., Charles W. Perdue, and Nancy L. Benco. William M. Solis Director Defense Capabilities and Management: Signed by William M. Solis: List of Congressional Committees: The Honorable John Warner Chairman The Honorable Carl Levin Ranking Minority Member Committee on Armed Services United States Senate: The Honorable Ted Stevens Chairman The Honorable Daniel K. Inouye Ranking Minority Member Subcommittee on Defense Committee on Appropriations United States Senate: The Honorable Duncan Hunter Chairman The Honorable Ike Skelton Ranking Minority Member Committee on Armed Services House of Representatives: The Honorable Joel Hefley Chairman The Honorable Solomon P. Ortiz Ranking Minority Member Subcommittee on Readiness Committee on Armed Services House of Representatives: The Honorable Jerry Lewis Chairman The Honorable John P. Murtha Ranking Minority Member Subcommittee on Defense Committee on Appropriations House of Representatives: [End of section] Appendix I: Scope and Methodology: To assess the extent to which the recommendations in the Department of Defense’s (DOD) November 2002 report to Congress addressed major problems in the personal property program, we took the following steps: * To identify the major problems facing the current personal property program, we reviewed DOD and GAO reports addressing this program. These reports identified problems associated with quality of service and claims. We also conducted interviews with personal property program officials and their contractors to gain an understanding of the current data management system’s limitations and the long-standing problems involving the lack of reliable information on shipments and their costs. : * To determine whether the proposed recommendations in DOD’s report to Congress addressed the major problems of the current program, we tracked the recommendations back to the U.S. Transportation Command’s report on its evaluation results and assessed the extent to which the recommendations are linked to and have the potential to address problems. : To assess whether the recommendations in DOD’s report to Congress were supported by DOD’s evaluation findings and should be implemented, we took the following steps: * To determine if the Transportation Command developed a methodologically sound evaluation plan, we assessed the command’s efforts against the findings and recommendations contained in our report[Footnote 10] on the Army’s Hunter Pilot Program results and against professional standards[Footnote 11] we would use if conducting a comparable evaluation. These sources addressed issues such as (1) seeking advice in designing a methodologically sound evaluation plan, (2) developing the evaluation plan prior to testing, (3) identifying factors to be assessed and the data required for analyses to develop findings and recommendations, (4) limiting quality of life surveys to only one for each participant to preclude survey “fatigue,” and (5) conducting simultaneous testing of the pilot and current programs. * To determine if the Transportation Command implemented an effective evaluation strategy during the data collection phase of its evaluation, we reviewed the pilot programs’ efforts to collect data for the four factors as prescribed in the Transportation Command’s evaluation plan. We also assessed the adjustments the Transportation Command made in its evaluation strategy to address issues that could affect the soundness of the results. An example of the issues addressed included developing a constructed cost methodology to provide better estimates of what DOD would have paid under the current program for shipments made by the pilot programs. : * To assess the Transportation Command’s development of findings and recommendations to improve the current personal property program, we reviewed the evaluation techniques (quantitative and qualitative analyses) used to analyze data collected for the four factors. Further, we assessed the extent to which the Transportation Command adjusted the evaluation techniques to make up for differences in the way that the pilot programs provided data for the evaluation. : To assess the methodology that DOD used to develop cost estimates for implementing the recommendations, we took the following steps: * To determine the reliability of the cost estimates for the pilot programs and for the proposed recommendations, we reviewed the cost projection methodologies used by the Transportation Command and by the Military Traffic Management Command. : * To determine the reliability of pilot program shipment-related costs used in the report, we reviewed the data collection efforts used by each pilot program for the transportation and storage of household goods included in the Transportation Command’s evaluation. Further, we reviewed the constructed cost methodology used to develop the estimates of what DOD would have paid to make comparable shipments under the current program in the pilot programs’ test areas. : * To determine the reasonableness of the assumptions and sources of data used to develop cost estimates for implementing recommendations for the personal property program, we met with officials from the Military Traffic Management Command and their contractor to discuss the methodology. We also reviewed the contents of their briefing on the cost estimate work for implementing changes to the claims process and performance-based service contracts and additional information the Military Traffic Management Command provided on the costs to implement information technology improvements. : We did not make an assessment of whether the anticipated benefits to be derived from implementing the three recommendations would warrant the additional costs DOD projects will be required to fund these improvements. Furthermore, we did not independently test the reliability of data DOD extracted from its data system to develop costs. We found that the department placed proper caveats on their use of such data, and in the case of comparing pilot programs’ shipment costs to current program costs, developed a constructed cost methodology to address current program data management system weaknesses. During this and prior[Footnote 12] reviews of DOD’s evaluation efforts, we met with officials and obtained documents from the Office of the Assistant Deputy Under Secretary of Defense (Transportation Policy), Washington, D.C.; the U.S. Transportation Command, Scott Air Force Base, Illinois; the Military Traffic Management Command, Alexandria, Virginia; the Department of Defense Inspector General, Full Service Moving Project, and Hay Group (Transportation Command Contractor), Arlington, Virginia; American Management Systems (Transportation Command contractor), PricewaterhouseCoopers (Military Traffic Management Command contractor), and Systems Research and Applications (Military Traffic Management Command contractor), Fairfax, Virginia; Logistics Management Institute (Military Traffic Management Command contractor), McLean, Virginia; the Navy’s Service Member Arranged Move Pilot Program, Mechanicsburg, Pennsylvania; The Gallup Organization (Full Service Moving Project contractor), Omaha and Lincoln, Nebraska; and Parsifal Corporation (Military Management Traffic Command contractor), Palm Bay, Florida. In addition to these agency meetings and documents, we drew upon information contained in a testimony statement, in reports, and in status briefings resulting from our prior reviews of this program. Our work for this review was performed from April 2002 through February 2003 in accordance with generally accepted government auditing standards. : [End of section] Appendix II: Overview of Current Personal Property Program and Pilot Programs: The Transportation Command evaluated three pilot programs to assess alternative approaches that might address long-standing problems with its current personal property program. The following tables provide features of the current program and the three pilot programs. As the tables show, the pilot programs had several features that provided enhancements to military personnel and their families and to DOD that are not offered by the current program. Table 1 compares claims-related features. Specifically, the pilot programs provided full replacement value rather than depreciated value for loss and damage and guaranteed claims settlement with 45 to 60 days of filing the claims. Table 1: Claims-Related Features of the Current Personal Property Program and Pilot Programs: Program features: Loss and damage claims. Program features: Reimbursements: Current personal property program: [Empty]; Military Traffic Management Command’s Reengineered Personal Property Program: [Empty]; Department of Defense’s Full Service Moving Project: [Empty]; Navy’s Service Member Arranged Move Pilot Program: [Empty]. Program features: --Basis for valuing property; Current personal property program: Depreciated value; Military Traffic Management Command’s Reengineered Personal Property Program: Full replacement value; Department of Defense’s Full Service Moving Project: Full replacement value; Navy’s Service Member Arranged Move Pilot Program: Full replacement value. Program features: --Maximum dollar value per move; Current personal property program: $40,000; Military Traffic Management Command’s Reengineered Personal Property Program: $63,000; Department of Defense’s Full Service Moving Project: $75,000; Navy’s Service Member Arranged Move Pilot Program: $72,000. Program features: Guaranteed claims settlement; Current personal property program: Not specified; Military Traffic Management Command’s Reengineered Personal Property Program: Within 60 days; Department of Defense’s Full Service Moving Project: Within 45 days; Navy’s Service Member Arranged Move Pilot Program: Within 60 days. Sources: DOD (data); GAO (analysis). [End of table] Table 2 compares the quality of service-related features. Some of the comparable features included emphasizing performance over cost in selecting transportation providers and prescreening of transportation providers. Table 2: Quality of Service-Related Features of the Current Personal Property Program and Pilot Programs: Program features: Counseling and arranging shipment services. Program features: Single relocation coordinator; Current personal property program: Counseling and arranging shipment services: No; Military Traffic Management Command’s Reengineered Personal Property Program: Counseling and arranging shipment services: No; Department of Defense’s Full Service Moving Project: Counseling and arranging shipment services: Yes; Navy’s Service Member Arranged Move Pilot Program: Counseling and arranging shipment services: Yes. Program features: Program management and counseling services provider; Current personal property program: Counseling and arranging shipment services: Installation personal property shipping office; Military Traffic Management Command’s Reengineered Personal Property Program: Counseling and arranging shipment services: Installation personal property shipping office; Department of Defense’s Full Service Moving Project: Counseling and arranging shipment services: Private-sector move managers; Navy’s Service Member Arranged Move Pilot Program: Counseling and arranging shipment services: Installation personal property shipping office. Program features: Screening process for transportation providers. Program features: Program emphasis in selecting transportation providers; Current personal property program: Counseling and arranging shipment services: Lowest cost; Military Traffic Management Command’s Reengineered Personal Property Program: Counseling and arranging shipment services: Performance; Department of Defense’s Full Service Moving Project: Counseling and arranging shipment services: Performance; Navy’s Service Member Arranged Move Pilot Program: Counseling and arranging shipment services: Performance. Program features: Prescreening of transportation providers; Current personal property program: Counseling and arranging shipment services: Very limited; Military Traffic Management Command’s Reengineered Personal Property Program: Counseling and arranging shipment services: Yes; Department of Defense’s Full Service Moving Project: Counseling and arranging shipment services: Yes; Navy’s Service Member Arranged Move Pilot Program: Counseling and arranging shipment services: Yes. Program features: Customer satisfaction surveys conducted and method; Current personal property program: Counseling and arranging shipment services: Surveys are not conducted; Military Traffic Management Command’s Reengineered Personal Property Program: Counseling and arranging shipment services: Survey company calls member; Department of Defense’s Full Service Moving Project: Counseling and arranging shipment services: Survey company calls member; Navy’s Service Member Arranged Move Pilot Program: Counseling and arranging shipment services: Service member returns survey via mail. Program features: Visibility of shipments during relocation process. Program features: Methods used to increase service members’ visibility of shipments during move process; Current personal property program: Counseling and arranging shipment services: None; Military Traffic Management Command’s Reengineered Personal Property Program: Counseling and arranging shipment services: Toll-free number to transportation provider; Department of Defense’s Full Service Moving Project: Counseling and arranging shipment services: Toll-free number to move manager and transportation provider; Navy’s Service Member Arranged Move Pilot Program: Counseling and arranging shipment services: Toll-free number to move coordinator and transportation provider, and member has pager. Sources: DOD (data); GAO (analysis). [End of table] Table 3 compares data reliability-related features. As noted, only one of the pilot programs had a data management system that provided reliable information to track individual shipments in transit and provide overall data on shipments and their associated costs. Table 3: Data Reliability-Related Features of the Current Personal Property Program and Pilot Programs: Program features: Availability and reliability of data on household goods shipments. Program features: Reliable data management system to track individual shipments in transit and to provide overall data on shipments and associated costs; Current personal property program: Availability and reliability of data on household goods shipments: No (current system is not designed to track shipments nor provide reliable shipment and cost data); Military Traffic Management Command’s Reengineered Personal Property Program: Availability and reliability of data on household goods shipments: Yes; Department of Defense’s Full Service Moving Project: Availability and reliability of data on household goods shipments: Not determined (developed but not fully operational-data management system needed additional refinement); Navy’s Service Member Arranged Move Pilot Program: Availability and reliability of data on household goods shipments: Not determined (developed a database by end of pilot program but was not fully implemented or evaluated). Sources: DOD (data); GAO (analysis). [End of table] Additional information on the current program and on each pilot program and its unique features follows. Current Program: The current DOD personal property program, valued at over $1.7 billion annually, moves more than 600,000 shipments each year for military personnel and their families from the military services, Defense agencies, and the Coast Guard. DOD is the moving industry’s single largest customer. Managed centrally by the headquarters office of the Military Traffic Management Command and administered locally by about 200 military and DOD transportation offices around the world, this program relies on over 1,200 domestic commercial carriers and more than 150 forwarders for international traffic to provide moving and storage services. Loss and Damage Claims: When loss and damage occur, military personnel can submit claims to their respective military service claims office. Based on depreciated values, the reimbursement rate is $1.25 per pound multiplied by the shipment weight, with a maximum amount of $40,000 per move. Military personnel have up to 2 years after receiving their shipments to file claims but must submit notice of loss and damage within 70 days of delivery. The current program does not have a specified time period in which the claims are to be settled. Counseling and Arranging Shipment Services: The current program provides counseling services and arranges the shipment and storage of household goods and unaccompanied baggage through government representatives, who are available to assist military personnel and their families at the origin and destination points of their moves. The current program does not have a real-time tracking system for shipments nor does it provide a single point of contact to manage the entire moving process; therefore, military personnel may interact with several people at the origin and destination offices during their relocation. Screening and Shipment Distribution Process for Transportation Providers: The current system is not designed to select transportation providers on the basis of quality service; rather, transportation providers offering a minimally acceptable level of quality are generally selected based on the lowest rates. The program uses the Total Quality Assurance Program to develop quality scores for each transportation provider. Each local military installation distributes its traffic using a traffic distribution roster. Transportation providers are placed on the rosters for each channel (origin and destination areas) by order of rate level and quality score. Transportation providers who participate in the domestic part of the current program submit their rates as a percentage of the government tariff, which is nearly 20 years old. The providers who participate in the international part of the current program submit single factor or fixed rates per hundredweight of the shipments. The current program does not use customer satisfaction surveys as a means to evaluate transportation provider performance. To remain in the program, a provider must maintain a minimally acceptable level of quality-a 90 percent score. Three factors are measured: on-time pickup, on-time delivery, and reported loss and damage to determine if points should be deducted from transportation providers and allocation of shipments should be reduced or terminated. Visibility of Shipments during the Relocation Process: The current program does not provide service members with real-time visibility of shipments during the relocation process. : Availability and Reliability of Data on Household Goods Shipments: Personnel at origin and destination personal property shipping offices enter information on shipments to their respective Transportation Operational Personal Property Standard Systems. However, data in these individual systems does not include all shipments that occur during the year, and the systems are not accessible to all parties involved in the relocation process. Destination personal property shipment offices are forwarded information on shipments via the current system; however, payment data on these shipments is maintained in a separate system. In addition to not providing information on all aspects of individual shipments, the current program’s data management system does not provide DOD and the military services information about the types of shipments and related costs for planning and budgeting purposes. The following are examples of the current system’s limitations: * the format of the system makes compiling data from multiple sites difficult; * not all data is captured promptly; and: * not all data and costs are captured/updated in the system. Funding Sources: Under the current program, the military services reimburse carriers and forwarders for shipment-related costs from military personnel accounts. Personal property shipment office expenses and claims filed with the government are funded from the military services’ operations and maintenance accounts. Military Traffic Management Command’s Reengineered Personal Property Program: Sponsored by the Military Traffic Management Command, the Reengineered Personal Property Program included outbound shipments for military and Coast Guard personnel departing from installations located in North Carolina, South Carolina, and Florida (excluding Tyndall Air Force Base). The pilot program ran concurrently with the existing program at these installations. The pilot program’s goal was to include 50 percent of eligible moves from the above installations to continental United States and European locations. The remaining shipments were to be moved under the existing program. The Reengineered Personal Property Program was initiated in January 1999 and operated for 12 months before data was submitted to the Transportation Command for evaluation. Loss and Damage Claims: Reimbursement for loss and damage claims was increased from depreciated value to full replacement value, and the dollar amounts per move increased from $40,000 under the current program to $63,000 under the Reengineered Personal Property Program. Additionally, the pilot program provided direct claims settlement between military personnel and their transportation providers and a requirement that transportation providers settle claims within 60 days of receiving claims forms from military personnel. Counseling and Arranging Shipment Services: Like the current program, the Reengineered Personal Property Program relied on personnel in the personal property shipping offices to provide counseling services and arrange for shipment and storage of household goods and unaccompanied baggage. A central contact point in these offices was not designated to manage the entire moving process; therefore, military personnel may have interacted with several people at the origin and destination offices during the relocation process. However, to improve customer service, the program’s Pilot Transportation Operational Personal Property Standard System provided real-time worldwide tracing capability. Screening and Shipment Distribution Process for Transportation Providers: Greater emphasis was placed on performance in awarding shipments to transportation providers. Evaluation of financial status, elimination of high-risk companies, and consideration of providers’ past performance, rather than lowest bid, played the dominant role in selecting initial transportation providers to participate in this pilot program. Transportation providers who participated in the pilot program submitted their bids for various origin and destination routes as a discount from the commercial tariff. Prices were fixed for a year, with no provision for rate increases during the contract period. Awards were made only to transportation providers whose offers conformed to the solicitation and represented the best overall value to the government. The Military Traffic Management Command evaluated company performance quarterly and compliance with terms and conditions of the contracts annually. Subsequent performance reviews were conducted based on customer satisfaction surveys and claims data. After transportation providers received their minimum guarantee of business for the year ($25,000), future awards were offered to the best performers. Feedback was provided monthly to transportation providers, and those that became poor performers were no longer offered household goods and unaccompanied baggage shipments. Visibility of Shipments during the Relocation Process: The Reengineered Personal Property Program provided the transportation provider’s toll-free number to military personnel to enhance visibility over their shipments throughout the relocation process. Availability and Reliability of Data on Household Goods Shipments: The Reengineered Personal Property Program implemented its central, Web-based Pilot Transportation Operational Personal Property Standard System in part to address problems associated with visibility of and availability of information on shipments during the relocation process. The pilot program’s data management system provided real-time access to both shipment and payment records. Access to the various modules of the system was granted to personal property shipment office personal at origin and destination locations, transportation providers, invoice certifying officers, prepayment auditors, military service headquarters, and military service claims offices and finance centers, based on each party’s need for the information. The system’s design allowed for entry of current address and telephone numbers of military personnel to improve the process of delivering household goods to a new residence. Data reliability was enhanced under the Reengineered Personal Property Program, but one problem noted during the evaluation was the need for military personnel to ensure that their contact information (phone number and address) was current during the relocation process. This had an effect on deliveries of household goods and the quality of life survey contractor’s ability to reach military personnel to ascertain their opinions about their relocation experience. In addition to providing information on all aspects of individual shipments, the Reengineered Personal Property Program’s data management system demonstrated the potential to provide DOD and the military services with information about the types of shipments and related costs managed annually for planning and budgeting purposes. Other Reengineered Personal Property Program Features: The Reengineered Personal Property Program achieved stronger transportation provider commitment with long-term contracts, and it used contractor support to conduct quality of life surveys with military personnel moving under the pilot program and to perform audits of each invoice submitted by transportation providers. Funding Sources: Like the current program, the military services reimbursed carriers and forwarders for shipment-related costs from their military personnel accounts. Transportation office expenses and any claims filed with the government were funded from the services’ operations and maintenance accounts. The Department of Defense’s Full Service Moving Project: Sponsored by the Office of the Assistant Deputy Under Secretary of Defense (Transportation Policy), the Full Service Moving Project included outbound shipments for military and Coast Guard personnel and DOD civilian departing from locations in the National Capital Region, Georgia (excluding Robins Air Force Base), and Minot Air Force Base, North Dakota. The pilot program’s goal was to include 90 percent of the moves from these locations to continental United States and to European and Asian-Pacific locations. The remaining shipments were to be moved under the current program. The Full Service Moving Project began in January 2001 and continued until its early termination in September 2001. Due to continuing delays in implementing this pilot program and DOD’s decision to terminate the pilot program in September 2001, the Full Service Moving Project had limited operational experience before submitting data to the Transportation Command. Loss and Damage Claims: Reimbursement for loss and damage claims was increased from depreciated value to full replacement value, with the dollar amounts increasing from $40,000 per move under the current program to $75,000 per move under the Full Service Moving Project. Additionally, the pilot program provided for direct claims settlement between military personnel and their transportation providers and a requirement that the responsible party (transportation providers or move managers) settle claims within 45 days of receiving claim forms from military personnel. Counseling and Arranging Shipment Services: Unlike the current program and other pilot programs, the Full Service Moving Project tested the use of private-sector relocation companies (move managers) for outsourcing traditional transportation services (counseling and arranging for the shipment and storage of household goods and unaccompanied baggage) performed by origin and destination personal property shipping offices. The pilot program’s goal was to provide a single point of contact (move manager) for military personnel and transportation providers to contact throughout the relocation process. Screening and Shipment Distribution Process for Transportation Providers: The Full Service Moving Project made major changes to the existing transportation provider approval, rate solicitation, and traffic distribution processes. The pilot program emphasized best value and placed more weight on performance (70 percent) than cost (30 percent) in determining which transportation providers would be awarded shipments. The pilot program contracted with a financial services company to conduct financial and performance assessments of transportation providers and move manager companies that wanted to participate in the pilot program. For approved transportation providers, rates were established for a 1-year cycle. The providers submitted their rates as a discount from the commercial tariff for domestic shipments and negotiated single rate factors for the overseas locations. Approved move managers were awarded 2-year contracts with 1- year options. The move management companies competitively bid their fees as flat rates, depending on whether they were responsible for claims settlement or the transportation provider carried this liability. Also, different fees were established for domestic and international shipments. The Full Service Moving Project used survey data from all personnel participating in the pilot program to determine future percentages of shipments that would be allocated to the transportation providers. The pilot program also planned to use survey data on move manager performance to determine future participation in the pilot program and incentive payments. The Full Service Moving Project’s Web-based Best Value Distribution Database maintained the transportation providers’ quality and cost scores based on survey information and costs associated with prior shipments. Move managers used this data to assign future shipments. However, in some instances (i.e., for group moves, when meeting small business requirements, when there was a lack of transportation provider capacity to handle shipments offered, for multiple shipments to a single transportation provider, and for international shipments to areas without an established rate), move managers were told to deviate from the information provided by the data management system. Visibility of Shipments during the Relocation Process: One of the goals of incorporating move managers into the relocation process was to provide real-time information to military personnel and to transportation providers regarding the status of household goods shipments. The move managers, unlike the current program’s personal property shipping office personnel, were responsible for the entire relocation process from the point of origin in establishing entitlements, arranging for transportation providers, and handling other personnel-related issues, to the destination in overseeing deliveries, approving storage, and either settling claims or assisting military personnel with issues involving settling claims with the transportation providers if the liability fell with the providers. Working with both military personnel and transportation providers, the move managers used contact information to keep military personnel informed of their shipments’ status and to coordinate the delivery of the shipments at the destination. Additionally, as part of the pilot program, all participants were provided a toll-free number to maintain visibility over their shipments throughout the process. Availability and Reliability of Data on Household Goods Shipments: In addition to move managers, the Full Service Moving Project’s Web- based Best Value Distribution Database was implemented to address problems associated with visibility of shipments during the relocation process. The pilot program’s data management system had access to both shipment and payment records via interface with US Bank’s PowerTrack and the move managers’ systems. Access to the pilot program’s data management system was granted to move managers, invoice certifying officers, military service headquarters, and military service claims offices and finance centers, based on each party’s requirements. Move managers were responsible for keeping the status of the shipments current in the pilot program’s data management system. However, the move managers did not always update this information in the system. Further, the ability of the move manager to contact the service member was directly affected by the information provided by the member. The Full Service Moving Project’s Web-based Best Value Distribution Database was anticipated to provide DOD and the military services information on the types of shipments and related costs managed annually for planning and budgeting purposes. Unlike the Reengineered Personal Property Program where various parties in the relocation process entered data into that pilot program’s data management system, the majority of the data in the Full Service Moving Project’s data management system was predicated on the move managers gathering and entering the information. Other Full Service Moving Project Features: The Full Service Moving Project achieved stronger transportation provider commitment with long-term contracts and faster payment of invoices; it offered binding cost estimates for shipments; it used contractor support to conduct quality of life surveys with military personnel moving under the pilot program and to perform audits of each invoice submitted by the transportation providers; and it offered optional relocation referral assistance for activities such as the purchase and sale of service members’ residences. Move managers were required to perform prepayment audits and business rules were established for an automatic payment method. Payment methodology was predicated on the move manager entering the expected invoice into PowerTrack and the transportation provider submitting a notice of delivery and invoice. Payment timeliness was also driven by the timeliness of documentation submitted by the transportation providers. On some invoices, the contracting representative had to review and certify payment in PowerTrack. This occurred when the match showed a difference of more than $1.00. Funding Sources: For this pilot program, the military services reimbursed carriers and forwarders for shipment-related costs from military personnel accounts. These accounts were also used to fund move manager expenses. Any claims that might have been filed with the government would have been funded from the military services’ operations and maintenance accounts. Navy’s Service Member Arranged Move Pilot Program: Sponsored by the Navy, the Service Member Arranged Move Pilot Program included only domestic outbound intrastate and interstate shipments for Navy personnel moving from its installations located at Puget Sound, Washington; San Diego, California; Norfolk, Virginia; New London, Connecticut; and Whidbey Island, Washington. One of this program’s objectives was to offer Navy military personnel a set of moving choices to meet their specific needs. This pilot program was one of three choices offered. Military personnel moving from the above locations could choose to move under the current personal property program, move their own household goods, or participate in the pilot program. The pilot program was initiated in April 1997 and began operations in January 1998. Because the Navy decided not to scope the Service Member Arranged Move Pilot Program comparable to other pilot programs (i.e., operational at multiple military services) and the pilot program did not provide data as outlined by the Transportation Command’s evaluation plan, its inclusion in the Transportation Command’s evaluation was limited to a qualitative assessment. Loss and Damage Claims: Reimbursement for loss and damage claims was increased from depreciated value to full replacement value, with the dollar amounts per move increasing from $40,000 under the current program to $72,000 under the Service Member Arranged Move Pilot Program. Additionally, the pilot program provided direct claims settlement between military personnel and their transportation providers and a requirement that transportation providers settle claims within 60 days of receiving claims forms from military personnel. Counseling and Arranging Shipment Services: Like the current program, the Service Member Arranged Move Pilot Program also relied on personnel in the personal property shipping offices to provide counseling services and arrange shipment and storage of household goods and unaccompanied baggage. However, the shipping office personnel at the origin installations participating in this pilot program served as the single point of contact coordinating the service members’ moves and remained available throughout the move to handle all issues, including claims. Unlike those participating in the current program and other pilot programs, service members participating in this pilot program identified the transportation provider they desired to handle their household goods shipments after they completed their reviews of participating providers’ vendor quality books (containing provider information and marketing materials) and of surveys completed by previous pilot program participants. The personal property office coordinator assigned to the service member had to concur with the member’s request, and the coordinator made actual arrangements with the carrier. Screening and Shipment Distribution Process for Transportation Providers: Staff in the program management office and personal property shipping offices participating in the Service Member Arranged Move Pilot Program initially screened transportation providers that wished to participate in the pilot program based on providers’ performance rather than low cost. Letters of agreement were adopted to streamline the contracting process and improve the quality of the move for Navy personnel. According to pilot program officials, these letters of agreement provided commercial best practices and enabled lessons learned from prior pilot program efforts and industry to be incorporated into the Navy pilot program. Actual contract awards were made on a case-by-case basis based on the best value decision for each move. Transportation providers used commercial tariffs in developing their bids for each move. Transportation providers approved to participate in the pilot program submitted their bids for various origin and destination channels using commercial tariffs. Bids were rejected if they did not fall within acceptable percentage discounts. Feedback was provided monthly to transportation providers, and those that became poor performers were no longer offered household goods and unaccompanied baggage shipments. Subsequently, service members who were planning their upcoming moves relied on information contained in a nine-question survey that other service members had completed after their moves and claims process ended. Service members who volunteered to participate in this pilot program had to manually review carrier books, which included documents provided by the carriers and prior surveys completed by service members who had been moved by the carriers. According to pilot program officials, six carriers were terminated or canceled from the pilot program-four for providing poor service and two for price gouging. Visibility of Shipments during Relocation Process: The Service Member Arranged Move Pilot Program relied upon the shipping office personnel, who served as the single point of contact coordinating the service members’ moves, to maintain visibility of shipments during the relocation process. In addition, the pilot program provided both the personal property shipping office’s and transportation provider’s toll-free numbers, as well as a pager to service members to enhance the members’ visibility of their shipments during the relocation process. Availability and Reliability of Data on Household Goods Shipments: The Service Member Arranged Move Pilot Program did not initially develop an alternative data management system to capture data on shipments and payment records. By the end of the pilot program, the Navy had developed a database to capture shipment data; however, the system was not fully implemented or evaluated. Unlike the other pilot programs, the Navy pilot program used local personal property program personnel rather than third parties to review all invoices for payment. Navy personnel who participated in the pilot program completed their own surveys, mailing the paper forms to their respective personal property program offices. Other Service Member Arranged Move Pilot Program Features: The Service Member Arranged Move Pilot Program was designed to offer all shipments to small businesses, to provide direct claims settlements between Navy personnel and the transportation providers, to make faster payments to transportation providers through government purchase cards, and to establish a stronger commitment from transportation providers by offering long-term contracts. Funding Sources: Like the current program, the Navy reimbursed carriers and forwarders for shipment-related costs from its military personnel account. Personal property shipment office expenses were funded from the Navy’s operations and maintenance account. While information on claims filed with the government was not provided, under this pilot program such expenses would also be funded from the operations and maintenance account. [End of section] Appendix III: Comments from the Department of Defense: DEPUTY UNDER SECRETARY OF DEFENSE FOR LOGISTICS AND MATERIEL READINESS 3500 DEFENSE PENTAGON WASHINGTON, DC 20301-3500: MAR 25 2003: Mr. William M. Solis: Director, Defense Capabilities and Management U.S. General Accounting Office: Washington, D.C. 20548: Dear Mr. Solis: This is the Department of Defense (DoD) response to the GAO draft report, GAO-03-367, “DEFENSE TRANSPORTATATION: Monitoring Costs and Benefits Needed While Implementing a New Program for Moving Household Goods’, dated February 24, 2003 (GAO Code 350161). The Department has reviewed the subject GAO draft report and offers the attached comments. Sincerely, Allen W. Beckett, Principal Assistant: Signed by Allen W. Beckett: GAO-03-367/GAO CODE 350161: “DEFENSE TRANSPORTATION: MONITORING COSTS AND BENEFITS NEEDED WHILE IMPLEMENTING A NEW PROGRAM FOR MOVING HOUSEHOLD GOODS”: DEPARTMENT OF DEFENSE COMMENTS TO THE RECOMMENDATIONS: RECOMMENDATION 1: The GAO recommended that the Secretary of Defense direct the Commander, U.S. TRANSCOM, to initiate actions to implement the three recommendations contained in DOD’s report to Congress within budget constraints. (Page 21/Draft Report): DoD RESPONSE: DoD concurs with this recommendation. DoD is currently developing a plan to implement the recommendations. The Department expects new program implementation beginning with electronic payment process in 4QFY03, with remaining changes beginning in 1QFY05, continuing through 1QFY06 assuming the Military Services receive the additional funds needed to fund program enhancements. RECOMMENDATION 2: The GAO recommended that the Secretary of Defense direct the Commander, U.S. TRANSCOM, to provide the Military Services and Congress additional information to quantify the risk associated with achieving the 13 percent projected cost estimate before the claims process and performance-based service contracts recommendations are implemented to provide the Military Services with information needed for budgeting purposes. (Page 21/Draft Report): DoD RESPONSE: DoD partially concurs with this recommendation. The Department continues to believe that the factors used in developing its projected cost estimate for the claims process and performance-based service contract recommendations are supportable and the new program can be implemented within the projected 13 percent increase over current program costs. The cost analysis was based on historical data and assumptions only where data was not available to provide analytical information. The GAO report emphasizes a lack of confidence in the adjustment made to the averaged pilot costs for economies of scale/ program efficiencies. The 5 percent reduction to the averaged pilot costs for economies of scale is not only reasonable, but also very conservative. The shipment costs in the pilots were based on a new set of business practices with only 3,000-5,000 shipments annually spread among the participating transportation providers. The current and new program encompasses approximately 600,000 shipments annually, about 200 times larger scale of operations than the pilots. The larger volume resulting in a lower cost per unit concept is a standard and accepted law of economics. While the GAO reports the basis of the 5 percent reduction for economies of scale was not based on either historical experience or data provided by DoD, they did not declare the methodology flawed. GAO concurs with the methodology used for estimating costs for the pilot programs and for two of the three cost reductions used to estimate the future program cost increase. We are confident in all three. We have presented the methodology to several of the Military Services. One Military Service performed their own analysis and validated the 13 percent. The methodology evaluated the “unproductive” cost drivers (shipment weight, small/large carrier mix, economies of scale) in the pilots and compensated for them using very conservative estimates. DoD does not see value added in providing the Military Services or Congress a formal risk assessment but will continue to work with the Military Services as execution progresses to make sure they have all information required for budget purposes. The Department non-concurs with the GAO finding that the cost estimate for implementing technology improvements will be $7M. The Department maintains that the cost to implement a new web-based system will fall between $4-6M. We projected costs of $5 million for development and implementation. Training costs and approximately two months of costs for independent verification and validation (IV&V) testing are estimated to be $500,000 each respectively. RECOMMENDATION 3: The GAO recommended that the Secretary of Defense direct the Commander, U.S. TRANSCOM, to monitor costs for all recommendations during the implementation phase to ensure that the proposed changes are being achieved within an acceptable and predefined range. (Page 21-22/Draft Report): DoD RESPONSE: DoD concurs with this recommendation. Rolling out the new program will require monitoring of costs to determine if the moving industry partners are submitting bids that will allow the Department to enhance this program within the 13 percent cost increase. The Department intends to establish program requirements that will focus more on quality of services (weighted best value score: 70 percent performance and 30 percent cost). To address the issue of monitoring costs, the plan includes a process to conduct a rate reasonableness analysis upon receipt of the rates. Those rates found to be outside the range of reasonableness will be rejected, and the carries will be given one opportunity to resubmit their rates-a function of controlling and evaluating costs. Only rates determined to be reasonable will be available for use in the program. For the first time, the Department will have one database with all cost data elements required by the Department’s functional and financial communities for cost evaluation at the shipment level. The establishment of metrics, target/benchmark performance indicators, and methodology for data collection will be included in the updated program plan of action and milestone document that was not available at the time of the GAO audit. RECOMMENDATION 4: The GAO recommended that the Secretary of Defense direct the Commander, U.S. TRANSCOM, to assess the effects of the three recommendations on the personal property program (to be carried out after the summertime peak moving season once the recommendations have been implemented) to determine whether the anticipated improvements in the program are being achieved at a reasonable cost. (Page 22/Draft Report): DoD RESPONSE: DoD concurs with this recommendation. The Department will have the information needed to determine if anticipated improvements have been achieved on a continuing basis. A customer satisfaction survey will be implemented to assess improvements in the program and identify how well the carriers are performing. This approach allows the relocating member to access carrier performance and report customer satisfaction. Also, since the surveys will be completed upon shipment delivery, timely performance data will be available. Carrier performance ratings will be established quarterly, with the exception of the peak season, when performance ratings with be established monthly. [End of section] Related GAO Products: Defense Transportation: Final Evaluation Plan Is Needed to Assess Alternatives to the Current Personal Property Program. GAO/NSIAD-00- 217R. Washington, D.C.: September 27, 2000. Defense Transportation: The Army’s Hunter Pilot Project Is Inconclusive but Provides Lessons Learned. GAO/NSIAD-99-129. Washington, D.C.: June 23, 1999. Defense Transportation: Plan Needed for Evaluating the Navy Personal Property Pilot. GAO/NSIAD-99-138. Washington, D.C.: June 23, 1999. Defense Transportation: Efforts to Improve DOD’s Personal Property Program. GAO/T-NSIAD-99-106. Washington, D.C.: March 18, 1999. Defense Transportation: The Army’s Hunter Pilot Project to Outsource Relocation Services. GAO/NSIAD-98-149. Washington, D.C.: June 10, 1998. Defense Transportation: Reengineering the DOD Personal Property Program. GAO/NSIAD-97-49. Washington, D.C.: November 27, 1996. FOOTNOTES [1] The Deputy Under Secretary of Defense (Logistics) tasked the U.S. Transportation Command to evaluate alternatives and develop recommendations to enhance the department’s current personal property program. [2] The three pilots included in the evaluation are the Military Traffic Management Command’s Reengineered Personal Property Program, the Department of Defense’s Full Service Moving Project, and the Navy’s Service Member Arranged Move Pilot Program. [3] U.S. General Accounting Office, Defense Transportation: Efforts to Improve DOD’s Personal Property Program, GAO/T-NSIAD-99-106 (Washington, D.C.: Mar. 18, 1999). [4] U.S. General Accounting Office, Defense Transportation: Final Evaluation Plan Is Needed to Assess Alternatives to the Current Personal Property Program, GAO/NSIAD-00-217R (Washington, D.C.: Sept. 27, 2000). [5] DOD reported that the 146 day recovery time is due to the fact that all current DOD contracts for shipment and storage of household goods give the carriers 120 days from receipt of the government’s demand to pay, deny, or make a final written offer on the claims. While many claims are settled in less that 120 days, mailing time and negotiations to resolve disputes result in a slightly higher average settlement time. [6] For example, a 10,000-pound shipment would have a maximum carrier liability for loss and damage of $12,500. Service members can make shipments exceeding their weight allowance by paying the extra shipping and storage costs for the overage in weight. However, the government’s liability is limited to $40,000 per move regardless of the excess weight shipped and stored. [7] U.S. General Accounting Office, Defense Transportation: The Army’s Hunter Pilot Project Is Inconclusive but Provides Lessons Learned, GAO/ NSIAD-99-129 (Washington, D.C.: June 23, 1999). [8] In later discussions, department officials stated that “economies of scale” should be changed to “program efficiencies” to reflect a more efficient program with timely and accurate management data; member counseling; and reduced loss, storage, and indirect costs. [9] We believe that the methodology and data used to develop these cost estimates are sound. [10] GAO/NSIAD-99-129. [11] U.S. General Accounting Office, Developing and Using Questionnaires, GAO/PEMD-10.1.7 (Washington, D.C.: Oct. 1993) and U.S. General Accounting Office, Using Structured Interviewing Techniques, GAO/PEMD-10.1.5 (Washington, D.C.: June 1991). [12] Our prior reviews focused on the department’s efforts (1) to develop a methodologically sound evaluation plan and (2) to collect data according to the plan for future analysis and development of recommendations for an improved departmentwide program. GAO’s Mission: The General Accounting Office, the investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO’s Web site ( www.gao.gov ) contains abstracts and full-text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics. Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select “Subscribe to daily E-mail alert for newly released products” under the GAO Reports heading. Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov Automated answering system: (800) 424-5454 or (202) 512-7470: Public Affairs: Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S. General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C. 20548: