Aviation Safety:

Improved Data Quality and Analysis Capabilities Are Needed as FAA Plans a Risk-Based Approach to Safety Oversight

GAO-10-414: Published: May 6, 2010. Publicly Released: Jun 10, 2010.

Additional Materials:

Contact:

Gerald L. Dillingham
(202) 512-4803
dillinghamg@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

To improve aviation safety, the Federal Aviation Administration (FAA) plans to have in place the initial capabilities of a risk-based approach to safety oversight, known as a safety management system (SMS), by the end of fiscal year 2010. FAA is also implementing new procedures and technologies to enhance the safety, capacity, and efficiency of the national airspace system. Data are central to SMS and FAA's ability to test the impact of these changes on safety. This congressionally requested report addresses FAA's (1) current and planned use of data to oversee aviation safety, (2) access to data for monitoring aviation safety and the safety performance of various industry sectors, and (3) efforts to help ensure data quality. To perform this work, GAO reviewed 13 databases that contain data on key aviation safety events, assessed data quality controls for the databases, and interviewed agency and industry officials, as well as 10 experts in aviation safety and data.

FAA analyzes data on past safety events, such as engine failures, to prevent their recurrence and plans to use data to support a more proactive approach to managing risk. For example, weather and air traffic control data helped identify factors associated with injuries from turbulence. As part of SMS, FAA plans to analyze data proactively to support a risk-based approach to safety oversight. For example, FAA plans to use data to model the impact of proposed changes in procedures and technologies on the safety of the national airspace system. Experts said that identifying risks is necessary to maintain the current level of safety and possibly achieve a higher level of safety in the future. Because SMS relies on data to identify emerging risks, FAA has an effort under way to enhance its access to industry data and improve its capability for automated analysis of multiple databases. According to FAA, this effort will allow for more efficient safety analyses. FAA is also developing a plan for managing data under SMS, but the plan does not fully address data, analysis, or staffing requirements. Without such requirements, the plan will not provide timely guidance for implementing SMS. FAA has access to some voluntarily reported data, which are important for SMS, but not all carriers and aviation personnel participate in FAA's voluntary reporting programs. Such data are gathered electronically by equipment on aircraft or reported by aviation personnel or carriers following noncriminal, unintentional violations or safety events. Industry personnel have some incentives to participate in voluntary programs, such as promised immunity from disciplinary action, but concerns about sanctions and the cost of equipment have deterred full participation, especially by smaller carriers. While FAA has some information on reasons for nonparticipation and has taken some steps to promote greater participation, it lacks carrier-specific information on why air carriers are not participating. FAA also lacks data to assess the safety performance of certain industry sectors, such as air cargo and air ambulance operators. GAO has previously made recommendations to address this lack of data. FAA concurred with GAO's prior recommendations and is taking actions to address them. To help ensure data quality--that is, data that are reliable (complete and accurate) and valid (measure what is intended)--FAA has implemented a number of data quality controls that are consistent with GAO's standards for data quality, but some weaknesses exit. For example, all the databases GAO reviewed had at least some controls in place to ensure that erroneous data are identified, reported, and corrected. However, about half the databases lack an important control--managers do not review the data prior to entry into the data system. FAA is taking steps to address its data weaknesses, but vulnerabilities remain, potentially limiting the usefulness of FAA's data for the safety analyses planned to support SMS.

Status Legend:

More Info
  • Review Pending-GAO has not yet assessed implementation status.
  • Open-Actions to satisfy the intent of the recommendation have not been taken or are being planned, or actions that partially satisfy the intent of the recommendation have been taken.
  • Closed-implemented-Actions that satisfy the intent of the recommendation have been taken.
  • Closed-not implemented-While the intent of the recommendation has not been satisfied, time or circumstances have rendered the recommendation invalid.
    • Review Pending
    • Open
    • Closed - implemented
    • Closed - not implemented

    Recommendations for Executive Action

    Recommendation: To help improve and expand FAA's capability to use data for aviation safety oversight, the Secretary of Transportation should direct the FAA Administrator to proceed with all deliberate speed to develop the capability to model the impact of NextGen changes on the national airspace system and manage any risks emerging from these changes.

    Agency Affected: Department of Transportation

    Status: Open

    Comments: When DOT provides information regarding the status of there efforts to address this open recommendation, we will update this information.

    Recommendation: To help improve and expand FAA's capability to use data for aviation safety oversight, the Secretary of Transportation should direct the FAA Administrator, given the importance of high-quality data, to extend standard quality controls, as appropriate, to the databases that support aviation safety oversight to ensure that the data are as reliable and valid as possible.

    Agency Affected: Department of Transportation

    Status: Closed - Implemented

    Comments: In 2010, GAO reported that the Federal Aviation Administration (FAA) was attempting to enhance aviation safety by shifting to a new data driven, risk-based safety oversight approach called a safety management system (SMS). Under this new approach, FAA planned to continue to use data on aviation accidents and incidents referred to as safety events to identify and address their causes. In addition, under SMS, FAA planned to use aviation safety data to identify conditions that could lead to safety events and to address them through changes in organizational processes, management, and culture. FAA, along with the international aviation community, recognized that high-quality data that are reliable and valid are essential to the effectiveness of a data-driven approach to safety, such as SMS. To help ensure data quality, FAA implemented a number of data quality controls that were consistent with GAO's standards for data quality, but some weaknesses exist. For example, all the databases GAO reviewed had at least some controls in place to ensure that erroneous data are identified, reported, and corrected. However, about half the databases lack an important control--managers do not review the data prior to entry into the data system. FAA was taking steps to address its data weaknesses, but vulnerabilities remain, potentially limiting the usefulness of FAA's data for the safety analyses planned to support SMS. Given the importance of high-quality data for this initiative, GAO recommended that FAA extend its standard quality controls, as appropriate, to the databases that support aviation safety oversight to ensure that the data are as reliable and valid as possible. In 2013, GAO confirmed that FAA has expanded its standard quality controls and procedures for its aviation safety databases. For example, in May 2012, FAA's Office of Aviation Safety implemented new standard operating procedures for making changes to data elements. As another example, FAA updated its data quality guidelines in October 2012 for aviation inspectors using the Air Transportation Oversight System database, which records inspection data and results. As a result of these types of efforts, FAA can continue to improve the quality of its safety data to support SMS and thus, better identify existing and emerging aviation safety risks.

    Recommendation: To help improve and expand FAA's capability to use data for aviation safety oversight, the Secretary of Transportation should direct the FAA Administrator to develop and implement a comprehensive plan that addresses how data fit into FAA's implementation of a proactive approach to safety oversight and ensure that this plan fully describes the relevant data challenges (such as ensuring data quality and continued access to voluntarily reported safety data), analytical approaches, and staffing requirements and integrates efforts to address them.

    Agency Affected: Department of Transportation

    Status: Open

    Comments: When DOT provides information regarding the status of there efforts to address this open recommendation, we will update this information.

    Recommendation: To help improve and expand FAA's capability to use data for aviation safety oversight, the Secretary of Transportation should direct the FAA Administrator to systematically identify the reasons that carriers are not participating in voluntary reporting programs, such as through a survey, and identify and implement further steps to encourage greater program participation, especially by smaller carriers.

    Agency Affected: Department of Transportation

    Status: Closed - Implemented

    Comments: In May 2010, GAO reported on the Federal Aviation Administration's (FAA) capacity to use available data to oversee aviation safety. For some data, GAO identified requirements for reporting to FAA, such as information on accidents and engine failures. However, other data is shared voluntarily, such as violations of federal regulations or safety events self reported by pilots, mechanics, and other aviation personnel. To encourage voluntary reporting, FAA agrees not to take enforcement actions against self-reported violations of regulations. Although voluntary reporting programs can generate safety information that is not available through other sources, GAO found that participation has been limited by concerns about confidentiality and cost considerations. Because of the criticality of voluntarily reported data, GAO recommended that FAA identify the reasons that carriers are not participating in such programs and implement further steps to encourage greater program participation, especially by smaller carriers. In 2013, GAO confirmed that FAA surveyed smaller carriers to obtain reasons why they were not participating in voluntary reporting programs and to identify some of the challenges to participating in these programs. To address the identified challenges, FAA improved the functionality of its web based tool for voluntary event reporting and analysis, which is cost free for U.S. carriers participating in an Aviation Safety Action Program. To encourage greater program participation, FAA has continued to promote participation by reaching out to individual carriers. Since 2010, overall participation of carriers in at least one voluntary reporting program has increased from 59 to 70 percent. As a result of these efforts, FAA has increased its access to data that provide insights into safety events that are not available from other sources and are critical to improving aviation safety.

    Apr 8, 2014

    Feb 28, 2014

    Feb 12, 2014

    Feb 5, 2014

    Feb 3, 2014

    Jan 31, 2014

    Jan 16, 2014

    Dec 9, 2013

    Looking for more? Browse all our products here