Skip to main content

Motor Carriers: Better Information Needed to Assess Effectiveness and Efficiency of Safety Interventions

GAO-17-49 Published: Oct 27, 2016. Publicly Released: Oct 27, 2016.
Jump To:
Skip to Highlights

Highlights

What GAO Found

In July 2010, the Federal Motor Carrier Safety Administration (FMCSA) chose to delay nationwide implementation of two of the eight interventions that FMCSA uses to address motor carrier safety concerns under its Compliance, Safety, Accountability (CSA) program. This delay is linked to continuing delays in developing software needed to support the two interventions, offsite investigations and the use of cooperative safety plans. The software under development is intended to help FMCSA overcome some of the information challenges it faces due to its reliance on legacy information systems. FMCSA estimates that the software development project will be completed by April 2017.

FMCSA has conducted evaluations of the effectiveness and efficiency outcomes it established for the CSA program. However, GAO identified several limitations in FMCSA's approaches that impact the usefulness of the evaluations:

Intervention effectiveness: FMCSA has developed a statistical model to annually evaluate the combined effectiveness of interventions. Although the model has some key strengths, such as accounting for a broad range of external factors, GAO identified a number of design and methodology limitations that reduce the usefulness of its results. For example, the model does not include an assessment of individual intervention types. Without this type of specific information, FMCSA is hampered in its ability to identify the circumstances under which different types of interventions are effective. Similarly, these types of limitations affect FMCSA's ability to accurately draw conclusions about intervention effectiveness across all intervention types.

Intervention efficiency: To assess the efficiency of CSA interventions, FMCSA has relied on a study that it sponsored and that was published in 2011. This study estimated the average cost of conducting interventions in four states from October 2008 through May 2009. However, FMCSA has not taken steps to update its cost estimates for interventions since the 2011 evaluation, despite changes since that time in the resources needed to conduct CSA interventions; nor has it taken steps to develop additional information that is representative of the costs in other states. Without current cost estimates that are representative of all states, FMCSA cannot appropriately assess the efficiency of its interventions.

FMCSA has taken some actions to improve the effectiveness and efficiency of CSA interventions, but lacks measures to monitor progress. In April 2014, FMCSA established a working group to assess CSA interventions and make recommendations for improvement. As of April 2016, the group had made 20 recommendations, of which 12 had been implemented. However, GAO found that while FMCSA has established some performance measures for its effectiveness outcome that are appropriate, it has not established similar measures for its efficiency outcome. FMCSA headquarters officials told GAO that effectiveness and efficiency are complementary outcomes that FMCSA strives to balance. Without a complete set of measures for both outcomes, FMCSA lacks benchmarks needed to regularly measure progress to achieve these outcomes.

Why GAO Did This Study

As part of its mission to reduce crashes and fatalities involving large commercial trucks and buses, FMCSA seeks to use a data-driven approach to identify the highest-risk motor carriers and address safety problems by applying a range of eight CSA program enforcement tools, called interventions, ranging from warning letters to placing carriers out of service.

A provision in a Senate report requires GAO to periodically assess FMCSA's implementation of the CSA program. This report examines the extent to which FMCSA has (1) implemented CSA interventions, (2) evaluated the effectiveness and efficiency of CSA interventions, and (3) monitored progress toward achieving outcomes.

GAO reviewed FMCSA data and documentation on all eight CSA intervention types from fiscal years 2010–2015, including FMCSA's strategic planning documents, guidance, and program evaluations. GAO interviewed industry stakeholders and FMCSA officials in headquarters, in each of FMCSA's service centers, and in eight states selected for their participation in FMCSA's CSA pilot test, location, and program size, among other factors.

Recommendations

GAO recommends that FMCSA evaluate the effectiveness of individual intervention types, update cost estimates so that they are current and representative of all states, and establish complete performance measures.

The Department of Transportation concurred with all of GAO's recommendations.

Recommendations for Executive Action

Agency Affected Recommendation Status
Department of Transportation To determine whether CSA interventions influence motor carrier safety performance, the Secretary of Transportation should direct the FMCSA Administrator to identify and implement, as appropriate, methods to evaluate the effectiveness of individual intervention types or common intervention patterns to obtain more complete, appropriate, and accurate information on the effectiveness of interventions in improving motor carrier safety performance. In identifying and implementing appropriate methods, FMCSA should incorporate accepted practices for designing program effectiveness evaluations, including practices that would enable FMCSA to more confidently attribute changes in carriers' safety behavior to CSA interventions.
Closed – Implemented
The Federal Motor Carrier Safety Administration's (FMCSA) Compliance, Safety, Accountability (CSA) program is intended to reduce the number of crashes involving large commercial trucks and buses. FMCSA seeks to achieve this goal by using a data-driven approach to identify and intervene with the highest-risk motor carriers and address safety problems by using a range of eight enforcement tools, called interventions. FMCSA applies a range of intervention types-from warning letters to placing carriers out of service-based on the type and severity of the safety problems identified for each specific carrier. Additionally, FMCSA identified improved effectiveness as a strategic outcome for CSA interventions. In 2016, we reported that FMCSA conducted regular evaluations of its CSA interventions to determine how effective these interventions were in improving carrier safety and reducing crashes, injuries, and fatalities. To evaluate the effectiveness of CSA interventions FMCSA developed a statistical model intended to annually evaluate the combined effect of all of its interventions. However, the model did not produce sufficiently complete, appropriate, or accurate information because of design and methodological limitations. For example, the model did not include an assessment of individual intervention types which was inconsistent with FMCSA's design and implementation of CSA interventions, which intends to apply specific intervention types to better address the safety problems unique to individual carriers. As a result, FMCSA lacked quality information needed to estimate how each intervention type affects motor carrier safety performance and address carriers' specific safety problems. Finally, accepted practices for designing evaluations explain that quality evaluations should draw conclusions commensurate with the power of the design. For example, because FMCSA did not evaluate the separate effect of warning letters, it lacked specific analytical evidence to support its conclusion that "the warning letter in and of itself can be an effective tool for improving motor carrier safety". To address these limitations, we recommended that FMCSA identify and implement methods to evaluate the effectiveness of individual intervention types and, in doing so, incorporate accepted practices for designing program effectiveness evaluations. In 2020, we confirmed that FMCSA revised its statistical model to measure the effectiveness of individual intervention types in reducing motor carrier crashes and fatalities. FCMSA also identified and clearly noted some limitations of its model. By taking these steps, FCMSA is better positioned to more accurately evaluate how effectively individual interventions improved motor carriers' safety performance.
Department of Transportation To understand the efficiency of CSA interventions the Secretary of Transportation should direct the FMCSA Administrator to update FMCSA's cost estimates to determine the resources currently used to conduct individual intervention types and ensure FMCSA has cost information that is representative of all states.
Closed – Implemented
The Federal Motor Carrier Safety Administration (FMCSA) seeks to use a data-driven approach to identify the highest-risk motor carriers and address safety problems by applying a range of enforcement tools, called intervention, under its Compliance, Safety, Accountability (CSA) program. FMCSA uses a range of eight intervention types-such as sending warning letters, conducting investigations, or issuing civil penalties-to enforce compliance with safety regulations. In 2016, GAO reported that FMCSA relied on a 2011 study to estimate the cost and assess the efficiency of conducting CSA interventions in four states, which did not represent costs in all states. However, FMCSA had not taken steps to update its cost estimates for interventions since the 2011 study, despite changes since that time in the resources needed to conduct CSA interventions; nor had it taken steps to develop additional information that was representative of costs in other states. GAO reported that without current cost estimates that are representative of all states, FMCSA could not appropriately assess the efficiency of its interventions. As a result, GAO recommended that FMCSA update its cost estimates to determine the resources currently used to conduct individual intervention types and ensure FMCSA has cost information that is representative of all states. In 2020, GAO confirmed that FMCSA completed a study with data it collected from division offices on the travel, labor, and other costs associated with conducting individual onsite and offsite investigations. FMCSA received updated cost data for onsite comprehensive investigations from all 50 states. Additionally, FMCSA received data on conducting offsite investigations from 9 of the 10 states that provide the requisite training for conducting such investigations. By updating these cost estimates, FMCSA has information it needs to understand the most efficient methods of conducting CSA interventions in all states and can better understand the relationship between these costs and intervention effectiveness.
Department of Transportation To enable FMCSA management to monitor the agency's progress in achieving its effectiveness and efficiency outcomes for CSA interventions and balance priorities, the Secretary of Transportation should direct the FMCSA Administrator to establish and use performance measures to regularly monitor progress toward both FMCSA's effectiveness outcome and its efficiency outcome.
Closed – Implemented
The Federal Motor Carrier Safety Administration's (FMCSA) Compliance, Safety, Accountability (CSA) program is intended to reduce the number of crashes involving large commercial trucks and buses. FMCSA seeks to achieve this goal by using a data-driven approach to identify and intervene with the highest-risk motor carriers and address safety problems by using a range of eight enforcement tools, called interventions. FMCSA applies a range of intervention types based on the type and severity of the safety problems identified for each specific carrier. FMCSA identified improved effectiveness and efficiency - which FMCSA considered to include the number of carriers reached through interventions and the resources required to complete interventions - as strategic outcomes of CSA interventions. In 2016, we reported that while FMCSA had established and monitored some performance measures for its effectiveness outcome for interventions, FMCSA had not established measures to monitor progress toward achieving its efficiency outcome for interventions. Leading practices for performance management state that agencies should express outcomes in a measurable form and establish a set of performance measures that help monitor progress toward achieving each outcome. Because FMCSA did not have a complete set of measures that reflected both the effectiveness and efficiency outcomes for CSA interventions, FMCSA lacked information needed to balance these priorities and guide management decisions about its application of interventions. As a result, we recommended that FMCSA establish and use performance measures to monitor progress toward both FMCSA's effectiveness and efficiency outcomes. In 2020, we confirmed that FMCSA completed a study of the cost of some CSA intervention types in which it identified total, average, and median costs for each intervention type as overall measures to monitor progress. In this study, FMCSA applied these measures to multiple circumstances, including geographic areas and level of enforcement activity. Further, in its October 2020 revised statistical model which measured the effectiveness of individual intervention types, FMCSA reinforced its established performance measure for intervention effectiveness, crash rates. In this revised model, FMCSA also included a performance measure for the amount of carriers reached through each intervention type, a key dimension of efficiency. In addition, FMCSA officials told us that FMCSA management had not taken any specific actions as a result of the efficiency measures it identified due to changes in operating procedures resulting from the Coronavirus Disease 2019. However, officials said that as FMCSA returns to normal operations, it will continue to monitor and make adjustments to the CSA intervention program. Because it has established performance measures for both its effectiveness and efficiency outcomes, FMCSA is better positioned to monitor progress and make any needed adjustments to CSA interventions to achieve the agency's desired outcomes.

Full Report

Office of Public Affairs

Topics

Agency evaluationsCommuter transportationCost analysisCost estimatesHighway safetyInformation systemsMotor carriersMotor carrier safetyPerformance measuresProgram evaluationRisk factorsRisk managementSoftwareTransportation industryTransportation safetyTransportation securityTrucking operationsBusesTrucksData reliabilityVehicle safety