Public Transportation:

FTA's Triennial Review Program Has Improved, But Assessments of Grantees' Performance Could Be Enhanced

GAO-09-603: Published: Jun 30, 2009. Publicly Released: Jul 30, 2009.

Additional Materials:

Contact:

Phillip R. Herr
(202) 512-8509
HerrP@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

The Federal Transit Administration (FTA) oversees about $5.5 billion in federal funds each year to transit agencies serving urban areas (grantee), in part through its triennial review program, which evaluates grantee adherence to federal requirements at least once every 3 years. GAO recommended in a 1998 oversight report that FTA improve the program. The subcommittee requested that GAO review this program. GAO identified (1) the extent to which triennial reviews indicate that grantees met applicable federal requirements from fiscal years 2000 through 2008; (2) the strengths and weaknesses of the triennial review process; and (3) FTA's performance measures for the triennial review and the extent to which they meet key attributes of successful performance measures. GAO addressed these objectives by analyzing oversight data on 424 grantees that had three triennial reviews, reviewing triennial review reports and guidance, assessing FTA's performance measures; and interviewing FTA headquarters and regional officials, contractors who conduct the reviews, and grantees.

GAO's analysis of FTA's triennial review oversight data found that over two-thirds of the 424 grantees analyzed have not consistently improved overall performance in terms of meeting more federal requirements from fiscal years 2000 through 2008. Fifty-one percent of grantees had mixed results in meeting requirements and 17 percent consistently met fewer requirements; while 31 percent consistently met more requirements--one of the goals of the triennial review program. Executives from three grantees that met most requirements attributed their performance to, among other things, having job descriptions that link employee responsibilities to the triennial review--a practice they said contributed to a culture of accountability. During the same time, grantees had the greatest number of findings in 5 of 23 triennial review areas, including the procurement and drug and alcohol testing areas. While FTA helps grantees address findings, additional efforts to identify the underlying causes and the severity of findings could further benefit grantees. FTA's triennial review program uses some strong management practices--having a well-defined process, using an information system to monitor grantees, and issuing reports timely. Still, two areas could be strengthened. First, while FTA is legislatively required to conduct a complete review of grantees' adherence to federal requirements at least once every 3 years, GAO identified a few instances where documentation does not clearly show that FTA reviewed all requirement areas. For example, 10 triennial review reports for 2008 showed that the drug and alcohol program area was "not reviewed." FTA's practice is to review all areas, regardless of documentation, but because FTA's guidance is not clear about how to document the review of areas where FTA has conducted a related special review in the prior two years, a few grantees may not be reviewed for 5 years. FTA plans to revise its guidance to avoid ambiguity. Second, FTA is aware of the burden oversight reviews place on grantees and works to limit this burden. However, in a limited number of cases, FTA did not coordinate its special oversight reviews with the triennial review schedule, which may place undue burden on a few grantees receiving multiple oversight reviews in the same fiscal year. FTA's two timeliness performance measures for assessing the triennial review program--(1) closing 80 percent of grantees' deficient findings within 30 days of their due date and (2) issuing 95 percent of the final triennial review reports within 30 days of completing a review--meet some, but not all key attributes of successful measures. Although both measures link throughout the organization, have measurable targets, are clearly stated, and do not overlap, the "close findings" measure does not meet the objectivity and reliability attribute. For example, data inaccuracies in past "close findings" data raised questions about the reliability of the measure. Also, both measures do not assess the core program activity to evaluate grantees' performance or governmentwide priorities, such as the quality of the triennial review program, and thus, as a whole, are not balanced, making it difficult for managers to not overemphasize one priority at the expense of others.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Comments: GAO reported in June 2009 on the Federal Transit Administration's (FTA) Triennial Review Program--an evaluation of grantees' adherence to federal requirements at least once every 3 years. Our analysis of FTA's oversight tracking data found that a few grantees consistently met nearly all federal requirements while all grantees reviewed generally faced problems in 5 requirement areas, such as procurement and maintenance requirements. We reported that while FTA analyzed some data to track and monitor oversight activities, it did not fully analyze oversight tracking data to identify, for example, high performing grantees along with management practices (i.e., best practices) that contributed to their high performance. FTA agreed that sharing best practices among grantees contributes to improved performance since grantees learn from other grantees, including high performing grantees. FTA also did not analyze oversight tracking data to identify problem areas and their underlying causes. Identifying the problem areas and their common underlying causes could provide FTA with valuable information to help many grantees find solutions to correct their findings and improve grantees' long-term performance. Therefore, we recommended that FTA further analyze and use oversight tracking data on the triennial review to, for example, help grantees address common problem areas and sustain their performance by applying best practices learned from high performing grantees. Starting in 2010, FTA began analyzing oversight data to identify the overall top 10 problem areas and the causes for each of these problem areas-information that is now available on its public website and has been discussing these problem areas with grantees during triennial review workshops each year. FTA has also begun sharing best practices among all grantees by publishing these best practices in the fiscal year 2011 and 2012 triennial review preparation workbooks. As a result, these actions will provide FTA with valuable opportunities to increase the triennial review program's effectiveness in helping grantees improve their performance.

    Recommendation: To support the triennial review program's goal of having grantees consistently meet more federal requirements, the Secretary of Transportation should direct the FTA Administrator to further analyze and use Oversight Tracking System (OTRAK) data on the triennial review to help grantees improve and sustain their performance. As part of this analysis, FTA should consider and identify, when necessary: (1) high performing grantees and their management practices, among other things; (2) the areas in which problem findings occur, and the underlying causes of these findings; and (3) the level of severity for each finding.

    Agency Affected: Department of Transportation

  2. Status: Closed - Implemented

    Comments: In June 2009, we reported that the Federal Transit Administration's (FTA) Triennial Review Program could be strengthened. Federal law requires FTA to review and evaluate completely grantees' adherence to federal requirements at least once every 3 years. However, guidance to contractors conducting the reviews provided flexibility to not review some areas in certain circumstances and documentation did not clearly show that grantees were receiving a complete review once every 3 years. For example, 10 triennial review reports for 2008 showed that the drug and alcohol program area was "not reviewed" because these grantees had received special drug and alcohol reviews in fiscal year 2006. Because this area was not reviewed in 2008, these grantees may not be reviewed in this area for 5 years (this area was last reviewed during the 2006 special oversight review and will not be reviewed again until the 2011 triennial review). Therefore, we recommended that the FTA Administrator help ensure that a complete performance review and evaluation of each grantee is conducted at least once every three years, in accordance with the statutory requirement, by revising the contractors' guide to remove the "not reviewed" category, requiring an updated evaluation of areas reviewed in the prior two years of the triennial review, and describing how contractors should document the triennial review reports with updates to other reviews, such as special oversight reviews conducted in the prior two years. In response, FTA revised its 2010 triennial review guidance, which states for all areas with special oversight reviews that even if such a special oversight review is scheduled for the current fiscal year or has been recently conducted, all questions are still to be reviewed. Proper evaluation and documentation provides future reviewers with a more transparent audit trail of findings and gives FTA managers the confidence that grantees were completely reviewed and evaluated in all areas at least once every three years.

    Recommendation: To support the triennial review program's goal of having grantees consistently meet more federal requirements, the Secretary of Transportation should direct the FTA Administrator to help ensure that a complete performance review and evaluation of each grantee is conducted at least once every three years, in accordance with the statutory requirement, revise the contractors' guide to remove the "not reviewed" category, require an updated evaluation of areas reviewed in the prior two years of the triennial review, and describe how contractors should document the triennial review reports with updates to other reviews, such as special oversight reviews conducted in the prior two years.

    Agency Affected: Department of Transportation

  3. Status: Closed - Implemented

    Comments: In June 2009, we reported that the Federal Transit Administration's (FTA) Triennial Review Program, which evaluates grantees' adherence to federal requirements at least once every 3 years, could be strengthened. FTA Order 5400.1 established an oversight review council and gave the council the responsibility to develop and coordinate oversight review schedules of various oversight reviews to avoid unnecessary duplication and overlap. However, the council was not fulfilling its responsibility as oversight coordinator. In a limited number of cases, for example, the council did not coordinate the schedules of FTA's special oversight reviews with the triennial review, which may have placed an undue burden on a few grantees receiving multiple oversight reviews in the same fiscal year. Therefore, we recommended that the FTA Administrator help ensure that the council, or another body, carries out the responsibilities of the oversight review coordinator, as specified in the FTA order. FTA responded to our recommendation by creating a publicly-available, oversight master calendar that allows FTA staff, contractors, and others to know when special oversight reviews, such as procurement, civil rights, or financial management reviews, have occurred and coordinate the scheduling of future oversight reviews. Furthermore, the current calendar shows special oversight reviews in the upcoming 6-months as well as reviews within in the past year or more. As a result, FTA could increase its oversight coverage of grantees and minimize the undue burden of uncoordinated multiple oversight reviews on some grantees.

    Recommendation: To support the triennial review program's goal of having grantees consistently meet more federal requirements, the Secretary of Transportation should direct the FTA Administrator to strengthen the triennial review process by ensuring that the Oversight Review Council, or another body, carries out the responsibilities specified in FTA Order 5400.1 to be actively involved in coordinating oversight review schedules in order to increase FTA's oversight coverage of grantees while, at the same time, minimizing the undue burden of multiple oversight reviews on grantees.

    Agency Affected: Department of Transportation

  4. Status: Closed - Implemented

    Comments: The Federal Transit Administration (FTA) oversees grants to transit agencies serving urban areas (grantee) in part through its triennial review program. The reviews provide FTA with an opportunity to evaluate urban area grantees' use of federal funds in areas such as procurement, financial management, and drug and alcohol programs. FTA's reviews can result in findings of deficiency- findings that represent areas for which the grantee is not adhering to federal requirements. In June 2009, GAO reported that within the triennial review program its "Close Findings Timely" performance measure links throughout the organization, had measurable targets, were clearly stated and did not overlap. However, the Close Findings measure did not meet the objectivity and reliability attribute. Specifically, GAO's analysis of the Close Findings measure found that its objectivity was limited because decisions to close findings were done largely at the discretion of regional staff, in consultation with grantees, and, when appropriate, headquarters officials. GAO found regional staff may use different criteria for when to close a deficient finding. The lack of criteria reduces the transparency of FTA's performance in closing findings. For example, an FTA official stated that the Closing Findings Timely performance measure could result in the unintended consequence of prematurely closing findings. Given these findings, the triennial review program could be improved to minimize the possibility for manipulation and increase the reliability of closed finding data. Therefore, GAO recommended that FTA should improve the objectivity and reliability with which FTA ensured deficient findings were satisfactorily closed. This would help reduce the incidents of non-compliance. In 2020, GAO confirmed that FTA developed a Quality Assurance Program for its reviews of grantees. Among other things, this improved review program is designed to promote consistency in implementation of the program. As part of quality monitoring, FTA will determine the percent of findings closed only after sufficient evidence of resolution has been provided. The program will also ensure that interpretation of statutes, regulations, guidance, issuance of findings, and application of corrective actions is consistent. This approach will help FTA ensure that grantees have satisfactorily resolved findings before closing them. Thus, reducing incidents of continuing or repeated non-compliance FTA is better positioned to determine the validity of reported results and make appropriate decisions.

    Recommendation: To support the triennial review program's goal of having grantees consistently meet more federal requirements, the Secretary of Transportation should direct the FTA Administrator to improve the objectivity and reliability of the "Close Findings Timely" performance measure by: (1) developing an internal controls process to verify grantees have satisfactorily resolved findings before closing them (2) continuing to emphasize the use of OTRAK across all regions and developing a process to close out triennial reviews in OTRAK upon grantees' completion of corrective actions.

    Agency Affected: Department of Transportation

  5. Status: Closed - Implemented

    Comments: FTA oversees grants to transit agencies serving urban areas (grantee) in part through its triennial review program. The reviews provide FTA with an opportunity to evaluate urban area grantees' use of federal funds in areas such as procurement, financial management, and drug and alcohol programs. In June 2009, GAO reported that FTA lacked the performance measures for evaluating the outcome and quality of the triennial review program. The performance measures assess the output of the triennial review program. Outcome performance measures-such as grantees' performance over time-can help evaluate the results of the triennial review programs. Without such a measure, FTA was limited in the extent to which it could effectively ascertain and document the extent to which the triennial review program was improving grantees' performance in meeting federal requirements and FTA managers had limited information on program performance, making the possibility of achieving program goals less likely. Therefore, GAO recommended that FTA should develop a method for evaluating improvements in grantee performance in meeting more federal requirements over time. In 2020, GAO confirmed that FTA developed a Quality Assurance Program for its reviews of grantees. Among other things, this improved review program is designed to promote grantee compliance. For example, as part of quality monitoring, FTA will determine the percent of grantees that demonstrate an increase or decrease in compliance with federal requirements from the prior review. Further, FTA will look at the percent of recipients that have repeat findings from the previous review. The program will also use its oversight reviews to promote recipient understanding of federal requirements and thus long-term compliance. As a result of this program, FTA is better positioned to provide decision makers the essential information they need determine areas for improvement and increase the likelihood of achieving program goals, such as increasing the number of grantees that meet federal requirements.

    Recommendation: To support the triennial review program's goal of having grantees consistently meet more federal requirements, the Secretary of Transportation should direct the FTA Administrator to meet standards set forth in previous GAO work, based on the Government Performance and Results Act of 1993, and develop performance measures to assess: (1) the outcomes of the triennial review program, such as a method for evaluating improvements in grantee performance in meeting more federal requirements over time (2) and the quality of the triennial review through improved contractor oversight, testing, or inspection.

    Agency Affected: Department of Transportation

  6. Status: Open

    Comments: Federal Transit Administration (FTA) anticipates adding a performance measurement, called reduction in repeat findings, that relates to assessing the quality of the review. This has been added to FTA's new Quality Review program focused on improving its Triennial Review program.

    Recommendation: To support the triennial review program's goal of having grantees consistently meet more federal requirements, the Secretary of Transportation should direct the FTA to ensure that additional outcome and quality performance measures are balanced with the existing measures for assessing the timeliness of the triennial review program so that one priority is not emphasized at the expense of others.

    Agency Affected: Department of Transportation

 

Explore the full database of GAO's Open Recommendations »

Sep 30, 2020

Sep 8, 2020

Sep 2, 2020

Aug 20, 2020

Aug 18, 2020

Aug 10, 2020

Jul 27, 2020

Jul 16, 2020

Looking for more? Browse all our products here