The Army's Hunter Pilot Project Is Inconclusive but Provides Lessons Learned
NSIAD-99-129: Published: Jun 23, 1999. Publicly Released: Jun 23, 1999.
- Full Report:
Pursuant to a legislative requirement, GAO provided information on the lessons learned from the Army's personal property program, the Hunter Pilot Project, focusing on the: (1) Army's evaluation methodology of the Hunter Pilot, including the validity of data and reported results; and (2) status of all ongoing and planned pilot projects and the adequacy of the Department of Defense's (DOD) plans to evaluate the pilot projects.
GAO noted that: (1) although the Army reported that the Hunter pilot was a success, GAO found that most of the results of its evaluation were inconclusive; (2) while the Hunter pilot provides services and benefits that were not previously available during the moving process, GAO was unable to validate all reported results of the Army's evaluation of this pilot because of weaknesses in the evaluation methodology and data; (3) specifically, because of the methods used in conducting the customer surveys, GAO could not confirm that customer satisfaction improved; (4) also, due to weaknesses in the Army's methodology and data reliability, GAO could not validate the extent to which pilot costs exceeded baseline costs; (5) however, GAO was able to confirm that 33 percent of the pilot shipments were awarded to small business carriers and agents; (6) further, lessons learned by the Army in developing an evaluation plan, conducting the pilot test, and evaluating results can provide useful information to DOD as it conducts and assesses other pilot efforts; (7) the Navy and the Military Traffic Management Command each have a personal property pilot project underway, and DOD is proposing a fourth pilot to test different approaches to improve its personal property program; (8) as a result, DOD will be running multiple pilots concurrently, with different goals, objectives, and expected outcomes; (9) U.S. Transportation Command (USTRANSCOM) is tasked with evaluating the results of the pilots and using that information to recommend a redesigned Department-wide relocation program; (10) however, DOD has not yet determined how many approaches will ultimately be tested and the milestones for completing the pilots' evaluation and implementing an improved process, nor has it assured itself that a methodologically sound evaluation process is in place to execute this process; (11) improving DOD's personal property program has been a slow, complex process; (12) DOD and the services have spent a large amount of time and effort to dramatically change the quality of services military personnel receive; and (13) GAO supports pilots as a tool to test different approaches.
Recommendation for Executive Action
Status: Closed - Implemented
Comments: USTRANSCOM has developed a comprehensive strategy for evaluating the pilot programs and the current DOD-wide personal property program. In its May 5, 2000 evaluation plan, the Command identified the four factors (quality of life, total cost, small business participation, and process improvements) for which it will collect and analyze data in developing recommendations for a new program. At this time, all sections of the plan have been completed with the exception of cost. USTRANSCOM anticipates that the cost issue will be resolved this fall, which will allow the Command to complete its evaluation plan. USTRANSCOM provided us a timeline showing that the data collection and analysis of all pilot and current program data and formulation of a recommendation for the new DOD-wide program should be included by January 30, 2002. USTRANSCOM also contracted with American Management Systems (AMS) to provide assistance in developing the evaluation plan, collect and analyze data from pilot projects and the current program, and develop a recommendation for a new personal property program. GAO is working with AMS and USTRANSCOM as this process continues and will continue to monitor their efforts.
Recommendation: The Secretary of Defense should direct the Commander in Chief, USTRANSCOM, to develop a comprehensive strategy for evaluating each of the pilot approaches that includes: (1) tests of a finite number of approaches; (2) timelines for completing the analysis and for implementing a new personal property process; and (3) milestones for finalizing a methodologically sound evaluation plan, including specific criteria for assessing each pilot's attributes and validating the baseline indirect costs to be used for each pilot program. To facilitate this process and to enhance the quality of the assessment, the Secretary of Defense should seek expert methodological advice before finalizing the evaluation plan.
Agency Affected: Department of Defense