This is the accessible text file for GAO report number GAO-03-640T entitled 'Highway Research: DOT's Actions to Implement Best Practices for Setting Research Agendas and Evaluating Outcomes' which was released on April 10, 2003. This text file was formatted by the U.S. General Accounting Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products’ accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. Testimony: Before the Committee on Science, Subcommittee on Environment, Technology and Standards, House of Representatives: United States General Accounting Office: GAO: For Release on Delivery Expected at 10:00 a.m. EDT: Thursday, April 10, 2003: Highway Research: DOT’s Actions to Implement Best Practices for Setting Research Agendas and Evaluating Outcomes: Statement of Katherine Siggerud, Acting Director Physical Infrastructure Team: Highway Research: GAO-03-640T: This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. GAO Highlights: Highlights of GAO-03-640T, a report to House Committee on Science, Subcommittee on Environment, Technology and Standards Why GAO Did This Study: Improvement and innovation based on highway research have long been important to the highway system. The Federal Highway Administration (FHWA) is the primary federal agency involved in highway research. Throughout the past decade, FHWA received hundreds of millions of dollars for its surface transportation research program, including nearly half of the Department of Transportation’s approximate $1 billion budget for research in fiscal year 2002. Given the expectations of highway research and the level of resources dedicated to it, it is important to know that FHWA is conducting high quality research that is relevant and useful. In May 2002, GAO issued a report on these issues and made recommendations to FHWA, which the agency agreed with, aimed at improving its processes for setting research agendas and evaluating its research efforts. GAO was asked to testify on (1) best practices for developing research agendas and evaluating research outcomes for federal research programs; (2) how FHWA’s processes for developing research agendas align with these best practices; and (3) how FHWA’s processes for evaluating research outcomes align with these best practices. What GAO Found: Leading organizations, federal agencies, and experts that conduct scientific and engineering research use best practices designed to ensure that research objectives are related to the areas of greatest interest to research users and that research is evaluated according to these objectives. Of the specific best practices recommended by experts—such as the Committee on Science, Engineering, and Public Policy and the National Science Foundation—GAO identified the following practices as particularly relevant for FHWA: (1) developing research agendas in consultation with external stakeholders to identify high- value research and (2) using a systematic approach to evaluate research through such techniques as peer review. FHWA’s processes for developing its research agendas do not always consistently include stakeholder involvement. External stakeholder involvement is important for FHWA because its research is to be used by others that manage and construct transportation systems. FHWA acknowledges that its approach for developing research agendas lacks a systematic process to ensure that external stakeholders are involved. In response to GAO’s recommendation, FHWA has drafted plans that take the necessary steps toward developing a systematic process for involving external stakeholders. While the plans appear responsive to GAO’s recommendation, as shown in the table below, GAO cannot evaluate their effectiveness until they are implemented. FHWA does not have a systematic process that incorporates techniques such as peer review for evaluating research outcomes. Instead, the agency primarily uses a “success story” approach to communicate about those research projects that have positive impacts. As a result, it is unclear the extent to which all research projects have achieved their objectives. FHWA acknowledges that it must do more to measure the performance of its research program, however, it is still in the process of developing a framework for this purpose. While FHWA’s initial plans appear responsive to GAO’s recommendation, GAO cannot evaluate their effectiveness until they are implemented. www.gao.gov/cgi-bin/getrpt? GAO-03-640T. To view the full report, including the scope and methodology, click on the link above. For more information, contact Katherine Siggerud at (202) 512-2834 or siggerudk@gao.gov [End of section] Mr. Chairman and Members of the Committee: We appreciate the opportunity to testify today on the Federal Highway Administration’s (FHWA) surface transportation research and technology program. Change, improvement, and innovation based on highway research have long been important to the highway system. While this research is a shared responsibility among FHWA, state departments of transportation, and private organizations, we focused on FHWA’s important leadership role as the primary federal agency involved in highway research. Throughout the past decade, FHWA has received hundreds of millions of dollars for its surface transportation research and technology program, including nearly half of the Department of Transportation’s (DOT) approximate $1 billion budget for research, development, and technology in fiscal year 2002. Given the important expectations of highway research and the significant level of resources dedicated to it, it is important for the Congress and the American people to know that the agency is conducting research that is relevant and useful to stakeholders and that is of high quality. In May 2002 we issued a report on these issues and made recommendations to FHWA, which the agency agreed with, aimed at improving its processes for setting research agendas and evaluating its research efforts.[Footnote 1] As it considers reauthorizing FHWA’s research and technology program, Congress will be making decisions about the structure of the program. Accordingly, my testimony today will discuss (1) best practices for developing research agendas and evaluating research outcomes for federal research programs; (2) the extent to which FHWA’s processes for developing research agendas align with the best practices for similar federal research programs; and (3) the extent to which FHWA’s processes for evaluating research outcomes align with these best practices. My statement is based in part on our May 2002 report, which focused primarily on those activities funded by the surface transportation research and technology deployment funding categories identified in the Transportation Equity Act for the 21st Century. In developing that report, we held discussions with FHWA officials and reviewed relevant program documents, legislation, and publications on best practices in federal research from the Transportation Research Board, the Committee on Science, Engineering, and Public Policy and others. In preparing for this hearing, we also updated FHWA’s activities in response to our findings and recommendations. In summary: * Leading organizations that conduct scientific and engineering research, other federal agencies with research programs, and experts in research and technology have identified and use best practices designed to ensure that research objectives are related to the areas of greatest interest and concern to research users and that research is evaluated according to these objectives. Specific best practices in these areas used in other federal research programs or recommended by experts--such as the Committee on Science, Engineering, and Public Policy[Footnote 2], the Environmental Protection Agency, the National Science Foundation, and the Office of Management and Budget--include: (1) developing research agendas in consultation with external stakeholders to identify high-value research and (2) using a systematic approach to evaluate ongoing and completed research through such techniques as peer review. : * As we reported last year, FHWA’s processes for developing research agendas for its research and technology program do not always consistently include stakeholder involvement. External stakeholder involvement is important for FHWA because its research is expected to be used by others, such as state departments of transportation, which manage and construct transportation systems. FHWA acknowledges that its approach for developing research agendas lacks a consistent, transparent, and systematic process to ensure that external stakeholders are involved. Instead, the agency expects each program office to determine how or whether to involve external stakeholders in the agenda setting process. As a result, this approach is used inconsistently. To improve its program and in response to our recommendations, FHWA has drafted plans that seem to take the necessary steps toward developing a systematic process for involving external stakeholders in the agenda setting process. FHWA’s plans have not been finalized, and we cannot comment on the potential effectiveness of these plans. : * We reported last year that FHWA does not have a systematic process that incorporates techniques such as peer review for evaluating research outcomes. Instead, the agency primarily uses a “success story” approach to evaluate and communicate its research outcomes. While this approach illustrates some benefits of the agency’s research, it cannot be used as the primary method to evaluate the outcomes of the research against intended results because these stories represent only a fraction of the program’s completed research projects. As a result, it is unclear whether the organization is selecting research projects that have the highest potential value, or the extent to which these projects have achieved their objectives. We recommended that FHWA develop a systematic approach to evaluating its research program, and noted peer review as a best practice for doing so. In response, FHWA agreed that the agency must do a better job to measure the performance of its research and technology program. However, currently it is still in the process of developing, defining, and adopting a framework for measuring performance. Therefore, we cannot yet comment on FHWA’s efforts to evaluate research outcomes. Background: FHWA is the DOT agency responsible for federal highway programs-- including distributing billions of dollars in federal highway funds to the states--and developing federal policy regarding the nation’s highways. The agency provides technical assistance to improve the quality of the transportation network, conducts transportation research, and disseminates research results throughout the country. FHWA’s program offices conduct these activities through its Research and Technology Program, which includes “research” (conducting research activities), “development” (developing practical applications or prototypes of research findings), and “technology” (communicating research and development knowledge and products to users). FHWA maintains a highway research facility in McLean, Virginia. This facility, known as the Turner-Fairbank Highway Research Center, has over 24 indoor and outdoor laboratories and support facilities. Approximately 300 federal employees, on-site contract employees, and students are currently engaged in transportation research at the center. FHWA’s research and technology program is based on the research and technology needs of each of its program offices such as the Offices of Infrastructure, Safety, or Policy. Each of the program offices is responsible for identifying research needs, formulating strategies to address transportation problems, and setting goals for research and technology activities that support the agency’s strategic goals. (See Appendix I for examples of research that these offices undertake.) One program office that is located at FHWA’s research facility provides support for administering the overall program and conducts some of the research. The agency’s leadership team, consisting of the associate administrators of the program offices and other FHWA offices, provides periodic oversight of the overall program. In 2002 FHWA appointed the Director of its Office of Research, Development, and Technology as the focal point for achieving the agency’s national performance objective of increasing the effectiveness of all FHWA program offices, as well as its partners and stakeholders, in determining research priorities and deploying technologies and innovation. In addition to the research activities within FHWA, the agency collaborates with other DOT agencies to conduct research and technology activities. For example, FHWA works with DOT’s Research and Special Programs Administration to coordinate efforts to support key research identified in the department’s strategic plan.[Footnote 3] Other nonfederal research and technology organizations also conduct research funded by FHWA related to highways and bridges. Among these are state research and technology programs that address technical questions associated with the planning, design, construction, rehabilitation, and maintenance of highways. In addition, the National Cooperative Highway Research Program conducts research on acute problems related to highway planning, design, construction, operation, and maintenance that are common to most states. Private organizations, including companies that design and construct highways and supply highway-related products, national associations of industry components, and engineering associations active in construction and highway transportation, also conduct or sponsor individual programs. Universities receive funding for research on surface transportation from FHWA, the states, and the private sector. Research Community Promotes Use of Best Practices for Developing Research Agendas and Evaluating Research Outcomes: Leading organizations that conduct scientific and engineering research, other federal agencies with research programs, and experts in research and technology have identified and use best practices for developing research agendas and evaluating research outcomes. Although the uncertain nature of research outcomes over time makes it difficult to set specific, measurable program goals and evaluate results, the best practices we identified are designed to ensure that the research objectives are related to the areas of greatest interest and concern to research users and that research is evaluated according to these objectives. These practices include (1) developing research agendas through the involvement of external stakeholders and (2) evaluation of research using techniques such as expert review of the quality of research outcomes. Developing Research Agendas Through the Involvement of External Stakeholders: External stakeholder involvement is particularly important for FHWA because its research is expected to improve the construction, safety, and operation of transportation systems that are primarily managed by others, such as state departments of transportation. According to the Transportation Research Board’s Research and Technology Coordinating Committee,[Footnote 4] research has to be closely connected to its stakeholders to help ensure relevance and program support, and stakeholders are more likely to promote the use of research results if they are involved in the research process from the start.[Footnote 5] The committee also identified merit review of research proposals by independent technical experts based on technical criteria as being necessary to help ensure the most effective use of federal research funds. In 1999, we reported that other federal science agencies--such as the Environmental Protection Agency and the National Science Foundation--used such reviews to varying degrees to assess the merits of competitive and noncompetitive research proposals.[Footnote 6] In April 2002, the Office of Management and Budget issued investment criteria for federal research and technology program budgets that urge these agencies to put into place processes to assure the relevance, quality and performance of their programs. For example, the guidance requires these programs to have agendas that are assessed prospectively and retrospectively through external review to ensure that funds are being expended on quality research efforts. Evaluation of Research Using Systematic Approach to Review the Quality of Research Outcomes: The Committee on Science, Engineering, and Public Policy reported in 1999 that federal agencies that support research in science and engineering have been challenged to find the most useful and effective ways to evaluate the performance and results of the research programs they support. Nevertheless, the committee found that research programs, no matter what their character and goals, can be evaluated meaningfully on a regular basis and in accordance with the Government Performance and Results Act. Similarly, in April 2002 the Office of Management and Budget issued investment criteria for federal research and technology program budgets that require these programs to define appropriate outcome measures and milestones that can be used to track progress toward goals and assess whether funding should be enhanced or redirected. In addition, program quality should be assessed periodically in relation to these criteria through retrospective expert review. The Committee on Science, Engineering, and Public Policy also emphasized that the evaluation methods must match the type of research and its objectives, and it concluded that expert or peer review is a particularly effective means to evaluate federally funded research. Peer review is a process that includes an independent assessment of the technical and scientific merit or quality of research by peers with essential subject area expertise and perspective equal to that of the researchers. Peer review does not require that the final impact of the research be known. In 1999, we reported that federal agencies, such as the Department of Agriculture, the National Institutes of Health, and the Department of Energy, use peer review to help them (1) determine whether to continue or renew research projects, (2) evaluate the results of research prior to publication of those results, and (3) evaluate the performance of programs and scientists.[Footnote 7] In its 1999 report, the Committee on Science, Engineering, and Public Policy also stated that expert review is widely used to evaluate: (1) the quality of current research as compared with other work being conducted in the field, (2) the relevance of research to the agency’s goals and mission, and (3) whether the research is at the “cutting edge.”: External Stakeholders’ Involvement in Developing FHWA’s Research Agendas Has Been Limited: Although FHWA engages external stakeholders in elements of its research and technology program, the agency currently does not follow the best practice of engaging external stakeholders on a consistent and transparent basis in setting its research agendas. The agency expects each program office to determine how or whether to involve external stakeholders in the agenda setting process. As we reported in May 2002, FHWA acknowledges that its approach to preparing research agendas is inconsistent and that the associate administrators of FHWA’s program offices primarily use input from the agency’s program offices, resource centers, and division offices.[Footnote 8] Although agency officials told us that resource center and division office staff provide the associate administrators with input based on their interactions with external stakeholders, to the extent that external stakeholder input into developing research agendas occurs, it is usually ad hoc and provided through technical committees and professional societies. For example, the agency’s agenda for environmental research was developed with input from both internal sources (including DOT’s and FHWA’s strategic plans and staff) and external sources (including the Transportation Research Board’s reports on environmental research needs and clean air, environmental justice leaders, planners, civil rights advocates, and legal experts). In our May 2002 report we recommended that FHWA develop a systematic approach for obtaining input from external stakeholders in determining its research and technology program’s agendas. FHWA concurred with our recommendation and has taken steps to develop such an approach. FHWA formed a planning group consisting of internal stakeholders as well as representatives from the Research and Special Programs Administration and the Pennsylvania Department of Transportation to determine how to implement our recommendation. This planning group prepared a report analyzing the approaches that four other federal agencies are taking to involve external stakeholders in setting their research and technology program agendas. Using the lessons learned from reviewing these other agencies’ activities, FHWA has drafted a Corporate Master Plan for Research and Deployment of Technology & Innovation. Under the draft plan, the agency would be required to establish specific steps for including external stakeholders in the agenda setting process for all areas of research throughout the agency’s research and technology program by fiscal year 2004. In drafting this plan, FHWA officials obtained input from internal stakeholders as well as external stakeholders, including state departments of transportation, academia, consultants, and members of the Transportation Research Board. It appears that FHWA has committed to taking the necessary steps to adopt the best practice of developing a systematic process for involving external stakeholders in the agenda setting process. The draft plan invites external stakeholders to assist FHWA with such activities as providing focus and direction to the research and technology program and setting the program’s agendas and priorities. However, because FHWA’s plan has not been finalized, we cannot comment on its potential effectiveness in involving external stakeholders. FHWA Lacks a Systematic Approach to Evaluating Research Outcomes: As we reported last year, FHWA does not have an agency wide systematic process to evaluate whether its research projects are achieving intended results that uses such techniques as peer review. Although the agency’s program offices may use methods such as obtaining feedback from customers and evaluating outputs or outcomes versus milestones, they all use success stories as the primary method to evaluate and communicate research outcomes. According to agency officials, success stories are examples of research results adopted or implemented by such stakeholders as state departments of transportation. These officials told us that success stories can document the financial returns on investment and nonmonetary benefits of research and technology efforts. However, we raised concerns that success stories are selective and do not cover the breadth of FHWA’s research and technology program. In 2001, the Transportation Research Board’s Research and Technology Coordinating Committee concluded that peer or expert review is an appropriate way to evaluate FHWA’s surface transportation research and technology program. Therefore, the committee recommended a variety of actions, including a systematic evaluation of outcomes by panels of external stakeholders and technical experts to help ensure the maximum return on investment in research. Agency officials told us that increased stakeholder involvement and peer review will require significant additional expenditures for the program. However, a Transportation Research Board official told us that the cost of obtaining expert assistance could be relatively low because the time needed to provide input would be minimal and could be provided by such inexpensive methods as electronic mail. In our May 2002 report, we recommended that FHWA develop a systematic process for evaluating significant ongoing and completed research that incorporates peer review or other best practices in use at federal agencies that conduct research.[Footnote 9] While FHWA has concurred that the agency must measure the performance of its research and technology program, it has not developed, defined or adopted a framework for measuring performance. FHWA’s report on efforts of other federal agencies that conduct research, discussed above, analyzed the approaches that four other federal agencies are taking to evaluate their research and technology programs using these best practices. According to FHWA’s assistant director for Research, Technology, and Innovation Deployment, the agency is using the results of this report to develop its own systematic approach for evaluating its research and technology program. However, this official noted that FHWA has been challenged to find the most useful and effective ways to evaluate the performance and results of the agency’s research and technology program. According to FHWA’s draft Corporate Master Plan for Research and Deployment of Technology & Innovation, FHWA is committed to developing a systematic method of evaluating its research and technology program that includes the use of a merit review panel. This panel would conduct evaluations and reviews in collaboration with representatives from FHWA staff, technical experts, peers, special interest groups, senior management, and contracting officers. According to the draft plan, these merit reviews would be conducted on a periodic basis for program-level and agency-level evaluations, while merit reviews at the project level would depend on the project’s size and complexity. FHWA is still in the process of developing, defining, and adopting a framework for measuring performance. Therefore, we cannot yet comment on how well FHWA’s efforts to evaluate research outcomes will follow established best practices. Mr. Chairman, this concludes my prepared statement. I would be pleased to answer any questions that you or Members of the Committee may have. Contacts and Acknowledgments: For further information on this testimony, please contact Katherine Siggerud at (202) 512-2834 or siggerudk@gao.gov. Deena Richart made key contributions to this testimony. [End of section] Appendix I: Roles of Program Offices in FHWA’s Research and Technology Program: FHWA’s research and technology program is based on the research and technology needs of each of its program offices such as the Offices of Infrastructure, Safety, and Policy. Each of the program offices is responsible for identifying research needs, formulating strategies to address transportation problems, and setting goals for research and technology activities that support the agency’s strategic goals. (See table 1.): Table 1: Roles of Program Offices in Research and Technology: Program office name: Federal Lands Highway; Role in research and technology: Development of applied research and technology applicable to transportation systems serving federal lands.; Examples of research and technology projects: Road Surface Analyzer (ROSAN) measurement of pavement smoothness. Program office name: Infrastructure; Role in research and technology: Development of research and technology in the areas of highway construction and physical maintenance, pavements, and structures.; Examples of research and technology projects: Long-term pavement performance.; Concrete research and technology.; Innovative bridge technology. Program office name: Operations; Role in research and technology: Development of research and technology program plans for the Intelligent Transportation Systems program, as well as operation of the transportation system and management of freight transportation.; Examples of research and technology projects: Research into advanced traffic simulation modeling.; Prediction tools and research into advanced, adaptive traffic signal control strategies.; Analysis of critical intermodal freight corridors and facilities.; Work zone best practices guide and program support. Program office name: Planning and Environment; Role in research and technology: Development of research and technology in the areas of planning, environment, and property acquisition.; Examples of research and technology projects: Workshops, synthesis materials, and case studies of state consultation practices with rural officials.; Statewide planning and travel forecasting training.; Research on the contribution of transportation to air pollution and on strategies to reduce transportation effects.; Highway noise barrier design handbook. Program office name: Policy; Role in research and technology: Development of analytical tools and data systems for policy development and studies; conducting analysis and studies to support the formulation of transportation policy and legislative initiatives; and preparation of major reports to Congress on highway policy issues.; Examples of research and technology projects: National personal transportation survey.; Highway cost allocation study.; Production of biennial report, “Status of the Nation’s Highways, Bridges, and Transit: Condition and Performance.”. Program office name: Safety; Role in research and technology: Leading in development of research and technology activities in the areas of Intersections; Pedestrian and Bicyclist Safety; Roadside Safety; Run- Off-Road Safety; and Speed Management.; Examples of research and technology projects: Interactive highway safety design model for two-lane roads.; Pedestrian safety countermeasure selection system.; ; Education and community programs for pedestrian/bicyclist safety.; Analysis of intersection safety issues.; Red-light running prevention.; Speed limit setting and enforcement.; Variable speed limits. Program office name: Research, Development, and Technology; Role in research and technology: Support of all other business units in the development and delivery of new technologies.; Examples of research and technology projects: Research activities to support Infrastructure, Operations, and Safety units. Source: GAO’s presentation of information provided by FHWA. [End of table] FOOTNOTES [1] Highway Research: Systematic Selection and Evaluation Processes Needed for Research Program (GAO-02-573, May 2002). [2] Committee on Science, Engineering, and Public Policy, Evaluating Federal Research Programs: Research and the Government Performance and Results Act (Washington, DC: Feb. 1999). The Committee on Science, Engineering, and Public Policy is a joint committee of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. [3] As required by the Transportation Equity Act for the 21ST Century, DOT annually develops the department wide “Research, Development, and Technology Plan.” This plan, drafted by the Research and Special Programs Administration and funded in part by FHWA, provides program- level detail on the directions that DOT’s research will take. This plan is used by the individual operating administrations, such as FHWA and the Research and Special Programs Administration, as a resource document to develop their subsequent program proposals for inclusion in their administration budgets. [4] The Research and Technology Coordinating Committee was convened in 1991 by the Transportation Research Board of the National Academies to provide a continuing, independent assessment of FHWA’s research and technology program. FHWA provides funding for the committee. [5] Transportation Research Board, The Federal Role in Highway Research and Technology (Washington, D.C.: National Academy Press, 2001), p. 76. For surface transportation research, potential stakeholders include state and local highway agencies that own and operate the nation’s highways; highway users; the companies that furnish the products, services, and equipment needed to build, operate, and maintain the highway system; and the people and communities that benefit from and are affected by the system. [6] Federal Research: Peer Review Practices at Federal Science Agencies Vary (GAO/RCED-99-99, Mar. 1999), p. 2. [7] GAO/RCED-99-99. [8] FHWA has 4 resource centers throughout the country, and division offices in each state, Puerto Rico and District of Columbia. [9] GAO-02-573.