Federal Motor Carrier Safety:
Modifying the Compliance, Safety, Accountability Program Would Improve the Ability to Identify High Risk Carriers
GAO-14-114: Published: Feb 3, 2014. Publicly Released: Feb 3, 2014.
What GAO Found
The Federal Motor Carrier Safety Administration's (FMCSA) Compliance, Safety, Accountability (CSA) program has helped the agency contact or investigate more motor carrier companies that own commercial trucks and buses and has provided a range of safety benefits to safety officials, law enforcement, and the industry than the previous approach, SafeStat. Specifically, from fiscal year 2007 to fiscal year 2012, FMCSA more than doubled its number of annual interventions, largely by sending warning letters to riskier carriers.
A key component of CSA--the Safety Measurement System (SMS)--uses carrier performance data collected from roadside inspections or crash investigations to identify high risk carriers for intervention by analyzing relative safety scores in various categories, including Unsafe Driving and Vehicle Maintenance. FMCSA faces at least two challenges in reliably assessing safety risk for the majority of carriers. First, for SMS to be effective in identifying carriers more likely to crash, the violations that FMCSA uses to calculate SMS scores should have a strong predictive relationship with crashes. However, based on GAO's analysis of available information, most regulations used to calculate SMS scores are not violated often enough to strongly associate them with crash risk for individual carriers. Second, most carriers lack sufficient safety performance data to ensure that FMCSA can reliably compare them with other carriers. To produce an SMS score, FMCSA calculates violation rates for each carrier and then compares these rates to other carriers. Most carriers operate few vehicles and are inspected infrequently, providing insufficient information to produce reliable SMS scores. FMCSA acknowledges that violation rates are less precise for carriers with little information, but its methods do not fully address this limitation. For example, FMCSA requires a minimum level of information for a carrier to receive an SMS score; however, this requirement is not strong enough to produce sufficiently reliable scores. As a result, GAO found that FMCSA identified many carriers as high risk that were not later involved in a crash, potentially causing FMCSA to miss opportunities to intervene with carriers that were involved in crashes.
FMCSA's methodology is limited because of insufficient information, which reduces the precision of SMS scores. GAO found that by scoring only carriers with more information, FMCSA could better identify high risk carriers likely to be involved in crashes. This illustrative approach involves trade-offs; it would assign SMS scores to fewer carriers, but these scores would generally be more reliable and thus more useful in targeting FMCSA's scarce resources.
In addition to using SMS scores to prioritize carriers for intervention, FMCSA reports these scores publicly and is considering using a carrier's performance information to determine its fitness to operate. Given the limitations with safety performance information, determining the appropriate amount of information needed to assess a carrier requires consideration of how reliable and precise the scores need to be for the purposes for which they are used. Ultimately, the mission of FMCSA is to reduce crashes, injuries, and fatalities. GAO continues to believe a data-driven, risk-based approach holds promise; however, revising the SMS methodology would help FMCSA better focus intervention resources where they can have the greatest impact on achieving this goal.
Why GAO Did This Study
From 2009 to 2012, large commercial trucks and buses have averaged about 125,000 crashes per year, with about 78,000 injuries and over 4,100 fatalities. In 2010, FMCSA replaced its tool for identifying the riskiest carriers--SafeStat--with the CSA program. CSA is intended to reduce the number of motor carrier crashes by better targeting the highest risk carriers using information from roadside inspections and crash investigations. CSA includes SMS, a data-driven approach for identifying motor carriers at risk of causing a crash.
GAO was directed by the Consolidated Appropriations Act of 2012 to monitor the implementation of CSA. This report examines the effectiveness of the CSA program in assessing safety risk for motor carriers. GAO spoke with FMCSA officials and stakeholders to understand SMS. Using FMCSA's data, GAO replicated FMCSA's method for calculating SMS scores and assessed the effect of changes--such as stronger data-sufficiency standards--on the scores. GAO also evaluated SMS's ability to predict crashes.
What GAO Recommends
GAO recommends that FMCSA revise the SMS methodology to better account for limitations in drawing comparisons of safety performance information across carriers. In addition, determination of a carrier's fitness to operate should account for limitations in available performance information. In response to comments from the Department of Transportation (USDOT), GAO clarified one of the recommendations. USDOT agreed to consider the recommendations.
For more information, contact Susan Fleming at (202) 512-2834 or firstname.lastname@example.org.
Recommendations for Executive Action
Comments: FMCSA did not agree with our recommendation, disputing the methodology and conclusions in our report. However, we continue to believe that addressing Safety Measurement System (SMS) methodology limitations has merit and could help the agency better target FMCSA's resources to the carriers that pose the highest risk of crashing. For example, we found FMCSA requires a minimum level of information for a carrier to receive an SMS score; however, this requirement is not strong enough to produce sufficiently reliable scores. As a result, FMCSA identified many carriers as high risk that were not later involved in a crash, potentially causing FMCSA to miss opportunities to intervene with higher risk carriers. To fully implement this recommendation, FMCSA should revise SMS methodology to account for data limitations that limit comparisons so that the FMCSA is better positioned to identify and mitigate carriers that pose the greatest safety risks.
Recommendation: To improve the CSA program, the Secretary of Transportation should direct the FMCSA Administrator to revise the SMS methodology to better account for limitations in drawing comparisons of safety performance information across carriers; in doing so, the Secretary of Transportation should direct the FMCSA Administrator to conduct a formal analysis that specifically identifies: (1) limitations in the data used to calculate SMS scores including variability in the carrier population and the quality and quantity of data available for carrier safety performance assessments, and (2) limitations in the resulting SMS scores including their precision, confidence, and reliability for the purposes for which they are used.
Agency Affected: Department of Transportation
Comments: The Federal Motor Carrier Safety Administration (FMCSA) agreed with the basic principles that GAO addressed in this area, but disagreed with GAO's characterization of FMCSA's proposed Safety Fitness Determination (SFD) rule. In January 2016, FMCSA issued a notice of proposed rulemaking (NPRM), which proposed a revised methodology for issuance of a safety fitness determination for motor carriers. Specifically, the new methodology would have determined when a motor carrier is not fit to operate commercial motor vehicles in or affecting interstate commerce based on the carrier's on-road safety data; an investigation; or a combination of both. However, in March 2017 FMCSA withdrew the NPRM. The Department plans, in part, to review the National Academies of Sciences (NAS) congressionally-mandated evaluation of SMS to determine whether further rulemaking action is necessary to revise the safety fitness determination process.
Recommendation: To improve the CSA program, the Secretary of Transportation should direct the FMCSA Administrator to ensure that any determination of a carrier's fitness to operate properly accounts for limitations we have identified regarding safety performance information.
Agency Affected: Department of Transportation