Federal Motor Carrier Safety:
Modifying the Compliance, Safety, Accountability Program Would Improve the Ability to Identify High Risk Carriers
GAO-14-114: Published: Feb 3, 2014. Publicly Released: Feb 3, 2014.
What GAO Found
The Federal Motor Carrier Safety Administration's (FMCSA) Compliance, Safety, Accountability (CSA) program has helped the agency contact or investigate more motor carrier companies that own commercial trucks and buses and has provided a range of safety benefits to safety officials, law enforcement, and the industry than the previous approach, SafeStat. Specifically, from fiscal year 2007 to fiscal year 2012, FMCSA more than doubled its number of annual interventions, largely by sending warning letters to riskier carriers.
A key component of CSA--the Safety Measurement System (SMS)--uses carrier performance data collected from roadside inspections or crash investigations to identify high risk carriers for intervention by analyzing relative safety scores in various categories, including Unsafe Driving and Vehicle Maintenance. FMCSA faces at least two challenges in reliably assessing safety risk for the majority of carriers. First, for SMS to be effective in identifying carriers more likely to crash, the violations that FMCSA uses to calculate SMS scores should have a strong predictive relationship with crashes. However, based on GAO's analysis of available information, most regulations used to calculate SMS scores are not violated often enough to strongly associate them with crash risk for individual carriers. Second, most carriers lack sufficient safety performance data to ensure that FMCSA can reliably compare them with other carriers. To produce an SMS score, FMCSA calculates violation rates for each carrier and then compares these rates to other carriers. Most carriers operate few vehicles and are inspected infrequently, providing insufficient information to produce reliable SMS scores. FMCSA acknowledges that violation rates are less precise for carriers with little information, but its methods do not fully address this limitation. For example, FMCSA requires a minimum level of information for a carrier to receive an SMS score; however, this requirement is not strong enough to produce sufficiently reliable scores. As a result, GAO found that FMCSA identified many carriers as high risk that were not later involved in a crash, potentially causing FMCSA to miss opportunities to intervene with carriers that were involved in crashes.
FMCSA's methodology is limited because of insufficient information, which reduces the precision of SMS scores. GAO found that by scoring only carriers with more information, FMCSA could better identify high risk carriers likely to be involved in crashes. This illustrative approach involves trade-offs; it would assign SMS scores to fewer carriers, but these scores would generally be more reliable and thus more useful in targeting FMCSA's scarce resources.
In addition to using SMS scores to prioritize carriers for intervention, FMCSA reports these scores publicly and is considering using a carrier's performance information to determine its fitness to operate. Given the limitations with safety performance information, determining the appropriate amount of information needed to assess a carrier requires consideration of how reliable and precise the scores need to be for the purposes for which they are used. Ultimately, the mission of FMCSA is to reduce crashes, injuries, and fatalities. GAO continues to believe a data-driven, risk-based approach holds promise; however, revising the SMS methodology would help FMCSA better focus intervention resources where they can have the greatest impact on achieving this goal.
Why GAO Did This Study
From 2009 to 2012, large commercial trucks and buses have averaged about 125,000 crashes per year, with about 78,000 injuries and over 4,100 fatalities. In 2010, FMCSA replaced its tool for identifying the riskiest carriers--SafeStat--with the CSA program. CSA is intended to reduce the number of motor carrier crashes by better targeting the highest risk carriers using information from roadside inspections and crash investigations. CSA includes SMS, a data-driven approach for identifying motor carriers at risk of causing a crash.
GAO was directed by the Consolidated Appropriations Act of 2012 to monitor the implementation of CSA. This report examines the effectiveness of the CSA program in assessing safety risk for motor carriers. GAO spoke with FMCSA officials and stakeholders to understand SMS. Using FMCSA's data, GAO replicated FMCSA's method for calculating SMS scores and assessed the effect of changes--such as stronger data-sufficiency standards--on the scores. GAO also evaluated SMS's ability to predict crashes.
What GAO Recommends
GAO recommends that FMCSA revise the SMS methodology to better account for limitations in drawing comparisons of safety performance information across carriers. In addition, determination of a carrier's fitness to operate should account for limitations in available performance information. In response to comments from the Department of Transportation (USDOT), GAO clarified one of the recommendations. USDOT agreed to consider the recommendations.
For more information, contact Susan Fleming at (202) 512-2834 or email@example.com.
Recommendations for Executive Action
Comments: The Department has not concurred with the recommendation, does not agree with our conclusions, and has requested that we close the recommendation as not implemented. For example, the Federal Motor Carrier Safety Administration (FMCSA) believes that implementing GAO's recommendation would result in a prioritization tool that only provides meaningful information about large carriers with little, if any, practical applications for assessing the safety performance of small and medium carriers that are involved in the majority of commercial motor vehicle related crashes. However, many small and medium carriers are not inspected frequently enough for FMCSA to reliably assess their crash risk using the current SMS methodology. If FMCSA continues using its current system, it may devote significant intervention resources targeting carriers that do not pose the greatest risk. FMCSA acknowledges that more data and observations would improve SMS from a statistical confidence interval perspective, which the Agency will continue to work towards. In addition, the National Academies of Sciences (NAS) conducted a congressionally-mandated evaluation of SMS, including the system's data accuracy and sufficiency, and in 2017 made several recommendations to improve FMCSA's ability to identify high-risk carriers more effectively. FMCSA is currently developing an action plan in response to the NAS recommendations but has not yet provided timeframes for completing or implementing the plan. GAO continues to believe our evidence was sufficient to support this recommendation and that absent FMSCA action, the agency will continue to lack reliable data to assess carriers that pose the greatest safety risks. GAO will review any SMS methodology revisions FMCSA implements to better account for limitations in comparing safety performance across carriers.
Recommendation: To improve the CSA program, the Secretary of Transportation should direct the FMCSA Administrator to revise the SMS methodology to better account for limitations in drawing comparisons of safety performance information across carriers; in doing so, the Secretary of Transportation should direct the FMCSA Administrator to conduct a formal analysis that specifically identifies: (1) limitations in the data used to calculate SMS scores including variability in the carrier population and the quality and quantity of data available for carrier safety performance assessments, and (2) limitations in the resulting SMS scores including their precision, confidence, and reliability for the purposes for which they are used.
Agency Affected: Department of Transportation
Comments: The Federal Motor Carrier Safety Administration (FMCSA) agreed with the basic principles that GAO addressed in this area, but disagreed with GAO's characterization of FMCSA's proposed Safety Fitness Determination (SFD) rule. In January 2016, FMCSA issued a notice of proposed rulemaking (NPRM), which proposed a revised methodology for issuance of a safety fitness determination for motor carriers. Specifically, the new methodology would have determined when a motor carrier is not fit to operate commercial motor vehicles in or affecting interstate commerce based on the carrier's on-road safety data; an investigation; or a combination of both. However, in March 2017 FMCSA withdrew the NPRM. The Department plans, in part, to review the National Academies of Sciences (NAS) congressionally-mandated evaluation of SMS to determine whether further rulemaking action is necessary to revise the safety fitness determination process.
Recommendation: To improve the CSA program, the Secretary of Transportation should direct the FMCSA Administrator to ensure that any determination of a carrier's fitness to operate properly accounts for limitations we have identified regarding safety performance information.
Agency Affected: Department of Transportation