Transportation Research:

Actions Needed to Improve Coordination and Evaluation of Research

GAO-03-500: Published: May 1, 2003. Publicly Released: May 1, 2003.

Additional Materials:

Contact:

Katherine A. Siggerud
(202) 512-6570
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

The Research and Special Programs Administration (RSPA) within the Department of Transportation (DOT) is responsible for coordinating and ensuring the evaluation of DOT research programs to promote the efficient use of departmental research funds, which in fiscal year 2002 totaled over $1 billion. RSPA is also responsible for conducting multimodal research that cuts across different modes of transportation. The House Committee on Appropriations directed GAO to examine RSPA's coordination and evaluation of research within DOT and the status of its own multimodal research.

RSPA has met some, but not all, legislative and DOT requirements pertaining to the coordination of departmental research efforts. For example, while RSPA develops an annual plan and meets monthly with other DOT research officials, RSPA does not review the status of all DOT research activities. Thus, it cannot determine whether duplication of research efforts within DOT does or does not occur. Additionally, RSPA has not developed standards against which to measure its performance in coordinating research within DOT. Moreover, RSPA has not fully met all legislative and DOT requirements to measure research results and oversee research evaluations across DOT. RSPA officials cited a lack of ready information on DOT research activities budget constraints and a lack of authority over other DOT agencies as reasons why they served primarily an information-sharing role, rather than as an overseer and manager of the coordination and evaluation processes. Since 1999, RSPA has budgeted $37 million to conduct four major research programs with applicability to more than one mode of transportation--for example, using technology to improve energy efficiency and reduce emissions and transportation dependence on petroleum. According to the Associate Administrator for Innovation, Research, and Education, RSPA's current multimodal research programs are scheduled for completion by the end of fiscal year 2004 and have had a variety of positive results. However, RSPA does not have an evaluation process to systematically evaluate the results of its multimodal research programs.

Status Legend:

More Info
  • Review Pending-GAO has not yet assessed implementation status.
  • Open-Actions to satisfy the intent of the recommendation have not been taken or are being planned, or actions that partially satisfy the intent of the recommendation have been taken.
  • Closed-implemented-Actions that satisfy the intent of the recommendation have been taken.
  • Closed-not implemented-While the intent of the recommendation has not been satisfied, time or circumstances have rendered the recommendation invalid.
    • Review Pending
    • Open
    • Closed - implemented
    • Closed - not implemented

    Recommendations for Executive Action

    Recommendation: To better meet legislation and DOT requirements for coordinating and evaluating transportation research within the department, the Secretary of Transportation should, in conjunction with the RSPA Administrator, work with DOT operating administrations to include in the department's annual research plan a summary of all research program evaluations conducted and a schedule of future evaluations.

    Agency Affected: Department of Transportation

    Status: Closed - Implemented

    Comments: In 2005, RSPA published a summary of all research program evaluations in that it published the results of its review of the operating administrations' application of the Office of Management and Budget's Research and Development Investment Criteria in its Fiscal Year 2005 Annual Research, Development, and Technology Plan. The results of RITA's fiscal year 2005 and 2006 reviews, and a schedule of RITA's planned fiscal year 2007 reviews, is included in the "Research, Development and Technology Annual Funding Fiscal Years 2006-2008, A Report to Congress", expected to be delivered to Congress in mid-September 2007. This report to Congress also includes summaries of research program evaluations conducted by modal research advisory committees, the Transportation Research Board, and key modal stakeholders in fiscal years 2006 and 2007.

    Recommendation: To better meet legislation and DOT requirements for coordinating and evaluating transportation research within the department, the Secretary of Transportation should, in conjunction with the RSPA Administrator, work with DOT operating administrations to develop a strategy to ensure that the results of all DOT's transportation research activities are evaluated according to established best practices. This strategy should include estimates of the costs for ensuring that evaluations are completed. Once the strategy has been developed and implemented, the results of these efforts should be incorporated in the department's annual research plan and reported to the Congress on an annual basis.

    Agency Affected: Department of Transportation

    Status: Closed - Implemented

    Comments: RITA (formerly RSPA) has developed strategy to ensure that the results of all DOT's research activities are evaluated according to established best practices. RITA's strategy includes three primary mechanisms, including (1) ensuring systematic application of the Office of Management and Budget's Research and Development Investment Criteria (relevance, quality, and performance) and the Program Assessment Rating Tool by the operating administrations; (2) annual internal program reviews with self-reporting by the operating administrations; and (3) documenting the operating administration's external stakeholder coordination and review. This strategy, documented in the agency's 2006-2010 Research, Development, and Technology Strategic Plan, upholds the technical quality of DOT's research activities and helps to ensure that research activities are addressing critical needs. The results of the fiscal year 2005 and 2006 reviews are included in the "Research, Development and Technology Annual Funding Fiscal Years 2006-2008: A Report to Congress", expected to be delivered to Congress in mid-September 2007. According to RITA, the fiscal year 2007 reviews have been focused on how well the operating administrations are implementing best practices, including external stakeholder involvement, merit review of competitive proposals, independent expert review, research performance measures, and external research coordination. RITA will report the results of its fiscal year 2007 reviews to the Planning Council and will publish the results in the next Research, Development and Technology Annual Funding Report to Congress. RITA has not estimated costs for these evaluations and a RITA official told us they do not have plans to do so since many of the evaluations have already been completed.

    Recommendation: To better meet legislation and DOT requirements for coordinating and evaluating transportation research within the department, the Secretary of Transportation should, in conjunction with the RSPA Administrator, work with DOT operating administrations to develop and apply quantifiable performance measures to assess the effectiveness of research coordination efforts (once a strategy for review has been developed and implemented), and document the results of these efforts in the department's annual research plan. These measures could include the number or research projects identified as possible candidates for joint effort or elimination and/or the associated reduction in the department's research spending.

    Agency Affected: Department of Transportation

    Status: Closed - Not Implemented

    Comments: DOT and RSPA did not concur with this recommendation in 2003, citing their views that: (1) the most useful and effective performance measures focus on results, while coordination is a process; (2) there are no performance measures capable of quantifying effective coordination; and (3) existing coordination processes have been effective in preventing unnecessary research duplication. RITA (formerly RSPA) officials also told us that they did not develop performance measures to assess the effectiveness of research coordination effort because they do not believe that a metric approach is well suited for assessing the effects of the coordination process.

    Recommendation: To better meet legislation and DOT requirements for coordinating and evaluating transportation research within the department, the Secretary of Transportation should, in conjunction with the RSPA Administrator, work with DOT operating administrations to develop a strategy for reviewing all DOT research projects to identify areas of unnecessary research duplication, overlap, and opportunities for joint efforts. The strategy should address time frames for implementing this review as well as discuss the extent to which finalizing the development and implementation of a DOT-wide research tracking system database could serve to facilitate this process. Once this strategy has been developed and implemented, the results of this effort should be incorporated in the department's annual research plan and reported to the Congress on an annual basis.

    Agency Affected: Department of Transportation

    Status: Closed - Implemented

    Comments: RITA (formerly RSPA) has developed strategy to review all of DOT research programs to identify areas of unnecessary research duplication overlap and opportunities for joint efforts. RITA's strategy consists of on-going internal reviews of all of DOT's research programs, conducted by the Program Review Working Group. The objectives of these internal reviews are to continuously improve transportation research management and performance; identify, share, and learn best practices; identify opportunities for leveraging resources and for crossmodal research initiatives; and prevent unnecessary research duplication. RITA's Program Review Working Group conducts these reviews by (1) convening meetings where officials from each of the operating administrations share information about areas of on-going or planned research, looking for opportunities for joint effort, and (2) conducting annual reviews of each operating administrations research plans, looking for research duplication, among other things. Plans for a database, which could assist in identifying areas of research duplication and opportunities for joint efforts are progressing, in that according to a RITA official, funding has been identified and the testing of a preliminary system is anticipated for Spring 2008, pending final approval. The results of RITA's coordination activities are discussed during Planning Team and Planning Council meetings and have been reported in the 2006-2010 Research, Development, and Technology Strategic Plan. According to the Strategic Plan, in fiscal year 2005 and again in fiscal year 2006, the Program Review Working Group found no research duplication and identified a number of areas for crossmodal collaboration, including geospatial technologies, freight capacity, security, alternative energy technologies, and advanced materials and sensors. A RITA official told us that eight working groups have been formed with representation from the relevant operating administrations to foster collaboration in these different areas. In addition to the Strategic Plan, the results of the coordination efforts are also included in the Research, Development and Technology Annual Funding Fiscal Years 2006-2008, A Report to Congress, expected to be delivered to Congress in mid-September 2007.

    Recommendation: The Secretary of Transportation should direct RSPA's Administrator to document RSPA's process for systematically evaluating the results of its own multimodal research programs, and apply this process to any future multimodal research programs that RSPA conducts.

    Agency Affected: Department of Transportation

    Status: Closed - Implemented

    Comments: RITA has established a systematic process for evaluating all of DOT's research, including RITA's multi-modal research programs. This process consists of reviews to identify, share, and apply best practices, identify opportunities for leveraging resources, and advance cross-modal research. RITA oversees DOT's Intermodal Advanced Research Working Groups which have identified 8 cross-modal research areas for collaboration with operating administrations including: congestion reduction; energy efficiency/alternative fuels; enhanced safety data; human factors; nanotechnology; positioning, navigation and timing; safe transport in an aging society, and system resilience/global logistics. RITA has developed an evaluation strategy to ensure that these cross-modal activities, as well as other DOT research activities, are evaluated according to best practices, including systematic application of OMB investment criteria, external stakeholder involvement, merit review of competitive proposals, independent expert review, development of research performance measures, and external research coordination. RITA officials told us that they are continuing to pursue the development and implementation of a more consistent process to be applied across all program areas, which they expect to complete by the end of fiscal year 2008.

    Jul 23, 2014

    Jun 25, 2014

    Jun 24, 2014

    Jun 18, 2014

    Jun 11, 2014

    May 30, 2014

    May 28, 2014

    Looking for more? Browse all our products here