Motor Carriers: Better Information Needed to Assess Effectiveness and Efficiency of Safety Interventions
Highlights
What GAO Found
In July 2010, the Federal Motor Carrier Safety Administration (FMCSA) chose to delay nationwide implementation of two of the eight interventions that FMCSA uses to address motor carrier safety concerns under its Compliance, Safety, Accountability (CSA) program. This delay is linked to continuing delays in developing software needed to support the two interventions, offsite investigations and the use of cooperative safety plans. The software under development is intended to help FMCSA overcome some of the information challenges it faces due to its reliance on legacy information systems. FMCSA estimates that the software development project will be completed by April 2017.
FMCSA has conducted evaluations of the effectiveness and efficiency outcomes it established for the CSA program. However, GAO identified several limitations in FMCSA's approaches that impact the usefulness of the evaluations:
Intervention effectiveness: FMCSA has developed a statistical model to annually evaluate the combined effectiveness of interventions. Although the model has some key strengths, such as accounting for a broad range of external factors, GAO identified a number of design and methodology limitations that reduce the usefulness of its results. For example, the model does not include an assessment of individual intervention types. Without this type of specific information, FMCSA is hampered in its ability to identify the circumstances under which different types of interventions are effective. Similarly, these types of limitations affect FMCSA's ability to accurately draw conclusions about intervention effectiveness across all intervention types.
Intervention efficiency: To assess the efficiency of CSA interventions, FMCSA has relied on a study that it sponsored and that was published in 2011. This study estimated the average cost of conducting interventions in four states from October 2008 through May 2009. However, FMCSA has not taken steps to update its cost estimates for interventions since the 2011 evaluation, despite changes since that time in the resources needed to conduct CSA interventions; nor has it taken steps to develop additional information that is representative of the costs in other states. Without current cost estimates that are representative of all states, FMCSA cannot appropriately assess the efficiency of its interventions.
FMCSA has taken some actions to improve the effectiveness and efficiency of CSA interventions, but lacks measures to monitor progress. In April 2014, FMCSA established a working group to assess CSA interventions and make recommendations for improvement. As of April 2016, the group had made 20 recommendations, of which 12 had been implemented. However, GAO found that while FMCSA has established some performance measures for its effectiveness outcome that are appropriate, it has not established similar measures for its efficiency outcome. FMCSA headquarters officials told GAO that effectiveness and efficiency are complementary outcomes that FMCSA strives to balance. Without a complete set of measures for both outcomes, FMCSA lacks benchmarks needed to regularly measure progress to achieve these outcomes.
Why GAO Did This Study
As part of its mission to reduce crashes and fatalities involving large commercial trucks and buses, FMCSA seeks to use a data-driven approach to identify the highest-risk motor carriers and address safety problems by applying a range of eight CSA program enforcement tools, called interventions, ranging from warning letters to placing carriers out of service.
A provision in a Senate report requires GAO to periodically assess FMCSA's implementation of the CSA program. This report examines the extent to which FMCSA has (1) implemented CSA interventions, (2) evaluated the effectiveness and efficiency of CSA interventions, and (3) monitored progress toward achieving outcomes.
GAO reviewed FMCSA data and documentation on all eight CSA intervention types from fiscal years 2010–2015, including FMCSA's strategic planning documents, guidance, and program evaluations. GAO interviewed industry stakeholders and FMCSA officials in headquarters, in each of FMCSA's service centers, and in eight states selected for their participation in FMCSA's CSA pilot test, location, and program size, among other factors.
Recommendations
GAO recommends that FMCSA evaluate the effectiveness of individual intervention types, update cost estimates so that they are current and representative of all states, and establish complete performance measures.
The Department of Transportation concurred with all of GAO's recommendations.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Transportation | To determine whether CSA interventions influence motor carrier safety performance, the Secretary of Transportation should direct the FMCSA Administrator to identify and implement, as appropriate, methods to evaluate the effectiveness of individual intervention types or common intervention patterns to obtain more complete, appropriate, and accurate information on the effectiveness of interventions in improving motor carrier safety performance. In identifying and implementing appropriate methods, FMCSA should incorporate accepted practices for designing program effectiveness evaluations, including practices that would enable FMCSA to more confidently attribute changes in carriers' safety behavior to CSA interventions. | The Federal Motor Carrier Safety Administration's (FMCSA) Compliance, Safety, Accountability (CSA) program is intended to reduce the number of crashes involving large commercial trucks and buses. FMCSA seeks to achieve this goal by using a data-driven approach to identify and intervene with the highest-risk motor carriers and address safety problems by using a range of eight enforcement tools, called interventions. FMCSA applies a range of intervention types-from warning letters to placing carriers out of service-based on the type and severity of the safety problems identified for each specific carrier. Additionally, FMCSA identified improved effectiveness as a strategic outcome for CSA... interventions. In 2016, we reported that FMCSA conducted regular evaluations of its CSA interventions to determine how effective these interventions were in improving carrier safety and reducing crashes, injuries, and fatalities. To evaluate the effectiveness of CSA interventions FMCSA developed a statistical model intended to annually evaluate the combined effect of all of its interventions. However, the model did not produce sufficiently complete, appropriate, or accurate information because of design and methodological limitations. For example, the model did not include an assessment of individual intervention types which was inconsistent with FMCSA's design and implementation of CSA interventions, which intends to apply specific intervention types to better address the safety problems unique to individual carriers. As a result, FMCSA lacked quality information needed to estimate how each intervention type affects motor carrier safety performance and address carriers' specific safety problems. Finally, accepted practices for designing evaluations explain that quality evaluations should draw conclusions commensurate with the power of the design. For example, because FMCSA did not evaluate the separate effect of warning letters, it lacked specific analytical evidence to support its conclusion that "the warning letter in and of itself can be an effective tool for improving motor carrier safety". To address these limitations, we recommended that FMCSA identify and implement methods to evaluate the effectiveness of individual intervention types and, in doing so, incorporate accepted practices for designing program effectiveness evaluations. In 2020, we confirmed that FMCSA revised its statistical model to measure the effectiveness of individual intervention types in reducing motor carrier crashes and fatalities. FCMSA also identified and clearly noted some limitations of its model. By taking these steps, FCMSA is better positioned to more accurately evaluate how effectively individual interventions improved motor carriers' safety performance.
View More |
| Department of Transportation | To understand the efficiency of CSA interventions the Secretary of Transportation should direct the FMCSA Administrator to update FMCSA's cost estimates to determine the resources currently used to conduct individual intervention types and ensure FMCSA has cost information that is representative of all states. | The Federal Motor Carrier Safety Administration (FMCSA) seeks to use a data-driven approach to identify the highest-risk motor carriers and address safety problems by applying a range of enforcement tools, called intervention, under its Compliance, Safety, Accountability (CSA) program. FMCSA uses a range of eight intervention types-such as sending warning letters, conducting investigations, or issuing civil penalties-to enforce compliance with safety regulations. In 2016, GAO reported that FMCSA relied on a 2011 study to estimate the cost and assess the efficiency of conducting CSA interventions in four states, which did not represent costs in all states. However, FMCSA had not taken...
|
| Department of Transportation | To enable FMCSA management to monitor the agency's progress in achieving its effectiveness and efficiency outcomes for CSA interventions and balance priorities, the Secretary of Transportation should direct the FMCSA Administrator to establish and use performance measures to regularly monitor progress toward both FMCSA's effectiveness outcome and its efficiency outcome. | The Federal Motor Carrier Safety Administration's (FMCSA) Compliance, Safety, Accountability (CSA) program is intended to reduce the number of crashes involving large commercial trucks and buses. FMCSA seeks to achieve this goal by using a data-driven approach to identify and intervene with the highest-risk motor carriers and address safety problems by using a range of eight enforcement tools, called interventions. FMCSA applies a range of intervention types based on the type and severity of the safety problems identified for each specific carrier. FMCSA identified improved effectiveness and efficiency - which FMCSA considered to include the number of carriers reached through...
|