Public Transportation:

FTA's Triennial Review Program Has Improved, But Assessments of Grantees' Performance Could Be Enhanced

GAO-09-603: Published: Jun 30, 2009. Publicly Released: Jul 30, 2009.

Additional Materials:

Contact:

Phillip R. Herr
(202) 512-8509
HerrP@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

The Federal Transit Administration (FTA) oversees about $5.5 billion in federal funds each year to transit agencies serving urban areas (grantee), in part through its triennial review program, which evaluates grantee adherence to federal requirements at least once every 3 years. GAO recommended in a 1998 oversight report that FTA improve the program. The subcommittee requested that GAO review this program. GAO identified (1) the extent to which triennial reviews indicate that grantees met applicable federal requirements from fiscal years 2000 through 2008; (2) the strengths and weaknesses of the triennial review process; and (3) FTA's performance measures for the triennial review and the extent to which they meet key attributes of successful performance measures. GAO addressed these objectives by analyzing oversight data on 424 grantees that had three triennial reviews, reviewing triennial review reports and guidance, assessing FTA's performance measures; and interviewing FTA headquarters and regional officials, contractors who conduct the reviews, and grantees.

GAO's analysis of FTA's triennial review oversight data found that over two-thirds of the 424 grantees analyzed have not consistently improved overall performance in terms of meeting more federal requirements from fiscal years 2000 through 2008. Fifty-one percent of grantees had mixed results in meeting requirements and 17 percent consistently met fewer requirements; while 31 percent consistently met more requirements--one of the goals of the triennial review program. Executives from three grantees that met most requirements attributed their performance to, among other things, having job descriptions that link employee responsibilities to the triennial review--a practice they said contributed to a culture of accountability. During the same time, grantees had the greatest number of findings in 5 of 23 triennial review areas, including the procurement and drug and alcohol testing areas. While FTA helps grantees address findings, additional efforts to identify the underlying causes and the severity of findings could further benefit grantees. FTA's triennial review program uses some strong management practices--having a well-defined process, using an information system to monitor grantees, and issuing reports timely. Still, two areas could be strengthened. First, while FTA is legislatively required to conduct a complete review of grantees' adherence to federal requirements at least once every 3 years, GAO identified a few instances where documentation does not clearly show that FTA reviewed all requirement areas. For example, 10 triennial review reports for 2008 showed that the drug and alcohol program area was "not reviewed." FTA's practice is to review all areas, regardless of documentation, but because FTA's guidance is not clear about how to document the review of areas where FTA has conducted a related special review in the prior two years, a few grantees may not be reviewed for 5 years. FTA plans to revise its guidance to avoid ambiguity. Second, FTA is aware of the burden oversight reviews place on grantees and works to limit this burden. However, in a limited number of cases, FTA did not coordinate its special oversight reviews with the triennial review schedule, which may place undue burden on a few grantees receiving multiple oversight reviews in the same fiscal year. FTA's two timeliness performance measures for assessing the triennial review program--(1) closing 80 percent of grantees' deficient findings within 30 days of their due date and (2) issuing 95 percent of the final triennial review reports within 30 days of completing a review--meet some, but not all key attributes of successful measures. Although both measures link throughout the organization, have measurable targets, are clearly stated, and do not overlap, the "close findings" measure does not meet the objectivity and reliability attribute. For example, data inaccuracies in past "close findings" data raised questions about the reliability of the measure. Also, both measures do not assess the core program activity to evaluate grantees' performance or governmentwide priorities, such as the quality of the triennial review program, and thus, as a whole, are not balanced, making it difficult for managers to not overemphasize one priority at the expense of others.

Status Legend:

More Info
  • Review Pending-GAO has not yet assessed implementation status.
  • Open-Actions to satisfy the intent of the recommendation have not been taken or are being planned, or actions that partially satisfy the intent of the recommendation have been taken.
  • Closed-implemented-Actions that satisfy the intent of the recommendation have been taken.
  • Closed-not implemented-While the intent of the recommendation has not been satisfied, time or circumstances have rendered the recommendation invalid.
    • Review Pending
    • Open
    • Closed - implemented
    • Closed - not implemented

    Recommendations for Executive Action

    Recommendation: To support the triennial review program's goal of having grantees consistently meet more federal requirements, the Secretary of Transportation should direct the FTA Administrator to meet standards set forth in previous GAO work, based on the Government Performance and Results Act of 1993, and develop performance measures to assess: (1) the outcomes of the triennial review program, such as a method for evaluating improvements in grantee performance in meeting more federal requirements over time (2) and the quality of the triennial review through improved contractor oversight, testing, or inspection.

    Agency Affected: Department of Transportation

    Status: Open

    Comments: Federal Transit Administration (FTA) responded to our recommendations stating that it plans to improve the quality of oversight reviews. Actions aimed to improve quality are: (1) increasing oversight of contractors by emphasizing regional and headquarters' participation in all on-site triennial reviews and (2) increasing annual oversight conference participation and quarterly review workshops that focus on improving performance measures. These activities were to be specified in new procedures coming out in 2011.

    Recommendation: To support the triennial review program's goal of having grantees consistently meet more federal requirements, the Secretary of Transportation should direct the FTA Administrator to improve the objectivity and reliability of the "Close Findings Timely" performance measure by: (1) developing an internal controls process to verify grantees have satisfactorily resolved findings before closing them (2) continuing to emphasize the use of OTRAK across all regions and developing a process to close out triennial reviews in OTRAK upon grantees' completion of corrective actions.

    Agency Affected: Department of Transportation

    Status: Open

    Comments: Federal Transit Administration (FTA) is working to strengthen the internal control process for closing out triennial review findings. Specifically, it is drafting operating procedures indicating that (1) the regional offices have the primary responsibility to provide the necessary technical assistance to grantees to close-out findings as well as ensure the findings are not closed-out prematurely and (2) program managers have a secondary responsibility to verify whether the findings were closed out properly. The procedures were to be issued in October 2010.

    Recommendation: To support the triennial review program's goal of having grantees consistently meet more federal requirements, the Secretary of Transportation should direct the FTA Administrator to strengthen the triennial review process by ensuring that the Oversight Review Council, or another body, carries out the responsibilities specified in FTA Order 5400.1 to be actively involved in coordinating oversight review schedules in order to increase FTA's oversight coverage of grantees while, at the same time, minimizing the undue burden of multiple oversight reviews on grantees.

    Agency Affected: Department of Transportation

    Status: Closed - Implemented

    Comments: In June 2009, we reported that the Federal Transit Administration's (FTA) Triennial Review Program, which evaluates grantees' adherence to federal requirements at least once every 3 years, could be strengthened. FTA Order 5400.1 established an oversight review council and gave the council the responsibility to develop and coordinate oversight review schedules of various oversight reviews to avoid unnecessary duplication and overlap. However, the council was not fulfilling its responsibility as oversight coordinator. In a limited number of cases, for example, the council did not coordinate the schedules of FTA's special oversight reviews with the triennial review, which may have placed an undue burden on a few grantees receiving multiple oversight reviews in the same fiscal year. Therefore, we recommended that the FTA Administrator help ensure that the council, or another body, carries out the responsibilities of the oversight review coordinator, as specified in the FTA order. FTA responded to our recommendation by creating a publicly-available, oversight master calendar that allows FTA staff, contractors, and others to know when special oversight reviews, such as procurement, civil rights, or financial management reviews, have occurred and coordinate the scheduling of future oversight reviews. Furthermore, the current calendar shows special oversight reviews in the upcoming 6-months as well as reviews within in the past year or more. As a result, FTA could increase its oversight coverage of grantees and minimize the undue burden of uncoordinated multiple oversight reviews on some grantees.

    Recommendation: To support the triennial review program's goal of having grantees consistently meet more federal requirements, the Secretary of Transportation should direct the FTA Administrator to help ensure that a complete performance review and evaluation of each grantee is conducted at least once every three years, in accordance with the statutory requirement, revise the contractors' guide to remove the "not reviewed" category, require an updated evaluation of areas reviewed in the prior two years of the triennial review, and describe how contractors should document the triennial review reports with updates to other reviews, such as special oversight reviews conducted in the prior two years.

    Agency Affected: Department of Transportation

    Status: Closed - Implemented

    Comments: In June 2009, we reported that the Federal Transit Administration's (FTA) Triennial Review Program could be strengthened. Federal law requires FTA to review and evaluate completely grantees' adherence to federal requirements at least once every 3 years. However, guidance to contractors conducting the reviews provided flexibility to not review some areas in certain circumstances and documentation did not clearly show that grantees were receiving a complete review once every 3 years. For example, 10 triennial review reports for 2008 showed that the drug and alcohol program area was "not reviewed" because these grantees had received special drug and alcohol reviews in fiscal year 2006. Because this area was not reviewed in 2008, these grantees may not be reviewed in this area for 5 years (this area was last reviewed during the 2006 special oversight review and will not be reviewed again until the 2011 triennial review). Therefore, we recommended that the FTA Administrator help ensure that a complete performance review and evaluation of each grantee is conducted at least once every three years, in accordance with the statutory requirement, by revising the contractors' guide to remove the "not reviewed" category, requiring an updated evaluation of areas reviewed in the prior two years of the triennial review, and describing how contractors should document the triennial review reports with updates to other reviews, such as special oversight reviews conducted in the prior two years. In response, FTA revised its 2010 triennial review guidance, which states for all areas with special oversight reviews that even if such a special oversight review is scheduled for the current fiscal year or has been recently conducted, all questions are still to be reviewed. Proper evaluation and documentation provides future reviewers with a more transparent audit trail of findings and gives FTA managers the confidence that grantees were completely reviewed and evaluated in all areas at least once every three years.

    Recommendation: To support the triennial review program's goal of having grantees consistently meet more federal requirements, the Secretary of Transportation should direct the FTA Administrator to further analyze and use Oversight Tracking System (OTRAK) data on the triennial review to help grantees improve and sustain their performance. As part of this analysis, FTA should consider and identify, when necessary: (1) high performing grantees and their management practices, among other things; (2) the areas in which problem findings occur, and the underlying causes of these findings; and (3) the level of severity for each finding.

    Agency Affected: Department of Transportation

    Status: Closed - Implemented

    Comments: GAO reported in June 2009 on the Federal Transit Administration's (FTA) Triennial Review Program--an evaluation of grantees' adherence to federal requirements at least once every 3 years. Our analysis of FTA's oversight tracking data found that a few grantees consistently met nearly all federal requirements while all grantees reviewed generally faced problems in 5 requirement areas, such as procurement and maintenance requirements. We reported that while FTA analyzed some data to track and monitor oversight activities, it did not fully analyze oversight tracking data to identify, for example, high performing grantees along with management practices (i.e., best practices) that contributed to their high performance. FTA agreed that sharing best practices among grantees contributes to improved performance since grantees learn from other grantees, including high performing grantees. FTA also did not analyze oversight tracking data to identify problem areas and their underlying causes. Identifying the problem areas and their common underlying causes could provide FTA with valuable information to help many grantees find solutions to correct their findings and improve grantees' long-term performance. Therefore, we recommended that FTA further analyze and use oversight tracking data on the triennial review to, for example, help grantees address common problem areas and sustain their performance by applying best practices learned from high performing grantees. Starting in 2010, FTA began analyzing oversight data to identify the overall top 10 problem areas and the causes for each of these problem areas-information that is now available on its public website and has been discussing these problem areas with grantees during triennial review workshops each year. FTA has also begun sharing best practices among all grantees by publishing these best practices in the fiscal year 2011 and 2012 triennial review preparation workbooks. As a result, these actions will provide FTA with valuable opportunities to increase the triennial review program's effectiveness in helping grantees improve their performance.

    Recommendation: To support the triennial review program's goal of having grantees consistently meet more federal requirements, the Secretary of Transportation should direct the FTA to ensure that additional outcome and quality performance measures are balanced with the existing measures for assessing the timeliness of the triennial review program so that one priority is not emphasized at the expense of others.

    Agency Affected: Department of Transportation

    Status: Open

    Comments: Federal Transit Administration (FTA) anticipates adding a performance measurement, called reduction in repeat findings, that relates to assessing the quality of the review. FTA officials told us that this new performance measurement is noncontroversial and was to be discussed at a June 2012 Oversight Meeting.

    Jul 31, 2014

    Jul 23, 2014

    Jun 25, 2014

    Jun 24, 2014

    Jun 18, 2014

    Jun 11, 2014

    May 30, 2014

    Looking for more? Browse all our products here