Motor Carriers:

Better Information Needed to Assess Effectiveness and Efficiency of Safety Interventions

GAO-17-49: Published: Oct 27, 2016. Publicly Released: Oct 27, 2016.

Additional Materials:

Contact:

Susan Fleming
(202) 512-2834
flemings@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

What GAO Found

In July 2010, the Federal Motor Carrier Safety Administration (FMCSA) chose to delay nationwide implementation of two of the eight interventions that FMCSA uses to address motor carrier safety concerns under its Compliance, Safety, Accountability (CSA) program. This delay is linked to continuing delays in developing software needed to support the two interventions, offsite investigations and the use of cooperative safety plans. The software under development is intended to help FMCSA overcome some of the information challenges it faces due to its reliance on legacy information systems. FMCSA estimates that the software development project will be completed by April 2017.

FMCSA has conducted evaluations of the effectiveness and efficiency outcomes it established for the CSA program. However, GAO identified several limitations in FMCSA's approaches that impact the usefulness of the evaluations:

Intervention effectiveness: FMCSA has developed a statistical model to annually evaluate the combined effectiveness of interventions. Although the model has some key strengths, such as accounting for a broad range of external factors, GAO identified a number of design and methodology limitations that reduce the usefulness of its results. For example, the model does not include an assessment of individual intervention types. Without this type of specific information, FMCSA is hampered in its ability to identify the circumstances under which different types of interventions are effective. Similarly, these types of limitations affect FMCSA's ability to accurately draw conclusions about intervention effectiveness across all intervention types.

Intervention efficiency: To assess the efficiency of CSA interventions, FMCSA has relied on a study that it sponsored and that was published in 2011. This study estimated the average cost of conducting interventions in four states from October 2008 through May 2009. However, FMCSA has not taken steps to update its cost estimates for interventions since the 2011 evaluation, despite changes since that time in the resources needed to conduct CSA interventions; nor has it taken steps to develop additional information that is representative of the costs in other states. Without current cost estimates that are representative of all states, FMCSA cannot appropriately assess the efficiency of its interventions.

FMCSA has taken some actions to improve the effectiveness and efficiency of CSA interventions, but lacks measures to monitor progress. In April 2014, FMCSA established a working group to assess CSA interventions and make recommendations for improvement. As of April 2016, the group had made 20 recommendations, of which 12 had been implemented. However, GAO found that while FMCSA has established some performance measures for its effectiveness outcome that are appropriate, it has not established similar measures for its efficiency outcome. FMCSA headquarters officials told GAO that effectiveness and efficiency are complementary outcomes that FMCSA strives to balance. Without a complete set of measures for both outcomes, FMCSA lacks benchmarks needed to regularly measure progress to achieve these outcomes.

Why GAO Did This Study

As part of its mission to reduce crashes and fatalities involving large commercial trucks and buses, FMCSA seeks to use a data-driven approach to identify the highest-risk motor carriers and address safety problems by applying a range of eight CSA program enforcement tools, called interventions, ranging from warning letters to placing carriers out of service.

A provision in a Senate report requires GAO to periodically assess FMCSA's implementation of the CSA program. This report examines the extent to which FMCSA has (1) implemented CSA interventions, (2) evaluated the effectiveness and efficiency of CSA interventions, and (3) monitored progress toward achieving outcomes.

GAO reviewed FMCSA data and documentation on all eight CSA intervention types from fiscal years 2010–2015, including FMCSA's strategic planning documents, guidance, and program evaluations. GAO interviewed industry stakeholders and FMCSA officials in headquarters, in each of FMCSA's service centers, and in eight states selected for their participation in FMCSA's CSA pilot test, location, and program size, among other factors.

What GAO Recommends

GAO recommends that FMCSA evaluate the effectiveness of individual intervention types, update cost estimates so that they are current and representative of all states, and establish complete performance measures.

The Department of Transportation concurred with all of GAO's recommendations.

For more information, contact Susan Fleming at (202) 512-2834 or flemings@gao.gov.

Recommendations for Executive Action

  1. Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.

    Recommendation: To determine whether CSA interventions influence motor carrier safety performance, the Secretary of Transportation should direct the FMCSA Administrator to identify and implement, as appropriate, methods to evaluate the effectiveness of individual intervention types or common intervention patterns to obtain more complete, appropriate, and accurate information on the effectiveness of interventions in improving motor carrier safety performance. In identifying and implementing appropriate methods, FMCSA should incorporate accepted practices for designing program effectiveness evaluations, including practices that would enable FMCSA to more confidently attribute changes in carriers' safety behavior to CSA interventions.

    Agency Affected: Department of Transportation

  2. Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.

    Recommendation: To understand the efficiency of CSA interventions the Secretary of Transportation should direct the FMCSA Administrator to update FMCSA's cost estimates to determine the resources currently used to conduct individual intervention types and ensure FMCSA has cost information that is representative of all states.

    Agency Affected: Department of Transportation

  3. Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.

    Recommendation: To enable FMCSA management to monitor the agency's progress in achieving its effectiveness and efficiency outcomes for CSA interventions and balance priorities, the Secretary of Transportation should direct the FMCSA Administrator to establish and use performance measures to regularly monitor progress toward both FMCSA's effectiveness outcome and its efficiency outcome.

    Agency Affected: Department of Transportation

 

Explore the full database of GAO's Open Recommendations »

Dec 4, 2017

Nov 30, 2017

Nov 15, 2017

Nov 2, 2017

Oct 31, 2017

Oct 26, 2017

Oct 5, 2017

Sep 20, 2017

Sep 7, 2017

Aug 28, 2017

Looking for more? Browse all our products here