Aviation Safety:

Improved Data Quality and Analysis Capabilities Are Needed as FAA Plans a Risk-Based Approach to Safety Oversight

GAO-10-414: Published: May 6, 2010. Publicly Released: Jun 10, 2010.

Additional Materials:

Contact:

Gerald L. Dillingham
(202) 512-4803
dillinghamg@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

To improve aviation safety, the Federal Aviation Administration (FAA) plans to have in place the initial capabilities of a risk-based approach to safety oversight, known as a safety management system (SMS), by the end of fiscal year 2010. FAA is also implementing new procedures and technologies to enhance the safety, capacity, and efficiency of the national airspace system. Data are central to SMS and FAA's ability to test the impact of these changes on safety. This congressionally requested report addresses FAA's (1) current and planned use of data to oversee aviation safety, (2) access to data for monitoring aviation safety and the safety performance of various industry sectors, and (3) efforts to help ensure data quality. To perform this work, GAO reviewed 13 databases that contain data on key aviation safety events, assessed data quality controls for the databases, and interviewed agency and industry officials, as well as 10 experts in aviation safety and data.

FAA analyzes data on past safety events, such as engine failures, to prevent their recurrence and plans to use data to support a more proactive approach to managing risk. For example, weather and air traffic control data helped identify factors associated with injuries from turbulence. As part of SMS, FAA plans to analyze data proactively to support a risk-based approach to safety oversight. For example, FAA plans to use data to model the impact of proposed changes in procedures and technologies on the safety of the national airspace system. Experts said that identifying risks is necessary to maintain the current level of safety and possibly achieve a higher level of safety in the future. Because SMS relies on data to identify emerging risks, FAA has an effort under way to enhance its access to industry data and improve its capability for automated analysis of multiple databases. According to FAA, this effort will allow for more efficient safety analyses. FAA is also developing a plan for managing data under SMS, but the plan does not fully address data, analysis, or staffing requirements. Without such requirements, the plan will not provide timely guidance for implementing SMS. FAA has access to some voluntarily reported data, which are important for SMS, but not all carriers and aviation personnel participate in FAA's voluntary reporting programs. Such data are gathered electronically by equipment on aircraft or reported by aviation personnel or carriers following noncriminal, unintentional violations or safety events. Industry personnel have some incentives to participate in voluntary programs, such as promised immunity from disciplinary action, but concerns about sanctions and the cost of equipment have deterred full participation, especially by smaller carriers. While FAA has some information on reasons for nonparticipation and has taken some steps to promote greater participation, it lacks carrier-specific information on why air carriers are not participating. FAA also lacks data to assess the safety performance of certain industry sectors, such as air cargo and air ambulance operators. GAO has previously made recommendations to address this lack of data. FAA concurred with GAO's prior recommendations and is taking actions to address them. To help ensure data quality--that is, data that are reliable (complete and accurate) and valid (measure what is intended)--FAA has implemented a number of data quality controls that are consistent with GAO's standards for data quality, but some weaknesses exit. For example, all the databases GAO reviewed had at least some controls in place to ensure that erroneous data are identified, reported, and corrected. However, about half the databases lack an important control--managers do not review the data prior to entry into the data system. FAA is taking steps to address its data weaknesses, but vulnerabilities remain, potentially limiting the usefulness of FAA's data for the safety analyses planned to support SMS.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Comments: In May 2010, GAO reported on the Federal Aviation Administration's (FAA) capacity to use available data to oversee aviation safety. GAO found that FAA analyzes data on past safety events, such as engine failures, to prevent their recurrence and plans to use data to support a more proactive approach to managing risk, also known as the Safety Management System (SMS). FAA is also developing a plan for managing data under SMS, but the plan does not fully address data, analysis, or staffing requirements. Because of the importance of timely guidance for implementing SMS, GAO recommended that FAA develop and implement a comprehensive plan that addresses how data fit into FAA's implementation of SMS and ensure that this plan fully describes the relevant data challenges (such as ensuring data quality and continued access to voluntarily reported safety data) and analytical approaches. In 2014, GAO confirmed that FAA completed the plan for its Aviation Safety Information Analysis and Sharing (ASIAS) program, which includes strategies and goals for aviation safety analyses and capabilities under its planned proactive safety approach. One step FAA took was the development of a plan for its ASIAS program, which includes strategies and goals for aviation safety analysis capabilities under its planned proactive safety approach. The plan outlines how the ASIAS program will address data challenges, such as classifying text narratives, and analytical approaches, including how to identify data patterns that could uncover potential high risk safety events and vulnerabilities. The plan also outlines the use of firewalls and anti-virus software to protect voluntarily reported safety information from stakeholders, such as air carriers and pilots. As another example, FAA also created a program plan for SMS in September 2011, which includes steps to automate data quality monitoring in ASIAS data sources. As a result of these efforts, FAA can move forward with more specific strategies and priorities to fully implement its planned proactive safety approach.

    Recommendation: To help improve and expand FAA's capability to use data for aviation safety oversight, the Secretary of Transportation should direct the FAA Administrator to develop and implement a comprehensive plan that addresses how data fit into FAA's implementation of a proactive approach to safety oversight and ensure that this plan fully describes the relevant data challenges (such as ensuring data quality and continued access to voluntarily reported safety data), analytical approaches, and staffing requirements and integrates efforts to address them.

    Agency Affected: Department of Transportation

  2. Status: Closed - Implemented

    Comments: In 2010, GAO reported that the Federal Aviation Administration (FAA) was attempting to enhance aviation safety by shifting to a new data driven, risk-based safety oversight approach called a safety management system (SMS). Under this new approach, FAA planned to continue to use data on aviation accidents and incidents referred to as safety events to identify and address their causes. In addition, under SMS, FAA planned to use aviation safety data to identify conditions that could lead to safety events and to address them through changes in organizational processes, management, and culture. FAA, along with the international aviation community, recognized that high-quality data that are reliable and valid are essential to the effectiveness of a data-driven approach to safety, such as SMS. To help ensure data quality, FAA implemented a number of data quality controls that were consistent with GAO's standards for data quality, but some weaknesses exist. For example, all the databases GAO reviewed had at least some controls in place to ensure that erroneous data are identified, reported, and corrected. However, about half the databases lack an important control--managers do not review the data prior to entry into the data system. FAA was taking steps to address its data weaknesses, but vulnerabilities remain, potentially limiting the usefulness of FAA's data for the safety analyses planned to support SMS. Given the importance of high-quality data for this initiative, GAO recommended that FAA extend its standard quality controls, as appropriate, to the databases that support aviation safety oversight to ensure that the data are as reliable and valid as possible. In 2013, GAO confirmed that FAA has expanded its standard quality controls and procedures for its aviation safety databases. For example, in May 2012, FAA's Office of Aviation Safety implemented new standard operating procedures for making changes to data elements. As another example, FAA updated its data quality guidelines in October 2012 for aviation inspectors using the Air Transportation Oversight System database, which records inspection data and results. As a result of these types of efforts, FAA can continue to improve the quality of its safety data to support SMS and thus, better identify existing and emerging aviation safety risks.

    Recommendation: To help improve and expand FAA's capability to use data for aviation safety oversight, the Secretary of Transportation should direct the FAA Administrator, given the importance of high-quality data, to extend standard quality controls, as appropriate, to the databases that support aviation safety oversight to ensure that the data are as reliable and valid as possible.

    Agency Affected: Department of Transportation

  3. Status: Closed - Implemented

    Comments: In May 2010, GAO reported that FAA analyzes data on past safety events, such as engine failures, to prevent their recurrence and plans to use data to support a more proactive approach to managing risk. As a part of its safety efforts, FAA planned to implement the Next Generation Air Transportation System (NextGen) that includes new technologies and procedures, which were intended to increase the safety, efficiency, and capacity of the national airspace system. However, these NextGen changes could also lead to consequences that have unintended effects on system safety, such as increasing congestion and safety risks on airport taxiways. To avoid such unintended consequences, FAA planned to use safety data to model and analyze the impact of changes planned for NextGen. At the time of GAO's audit, FAA assessed risks for specific NextGen procedures and technologies, but could not model the risks across the national airspace system in a comprehensive manner. FAA had begun to obtain some operational data for the project to assess risks across the national airspace system and had contracted with the Volpe National Transportation Systems Center to integrate airport runway surface data. Although safety assessments had been conducted on individual NextGen technologies, until the agency had finalized this modeling project, it could not begin systemwide assessments of the safety of NextGen technologies and procedures that were already being deployed. Therefore, GAO recommended that FAA proceed with all deliberate speed to develop such modeling capability and manage any risks emerging from these changes. In 2014, GAO confirmed that FAA had moved out quickly to improve its capability to model and manage risk. FAA developed the Integrated Safety Assessment Model to analyze and assess baseline risks as well as emerging risks from NextGen changes. In November 2013, FAA used the results of this process to publish an annual impact analysis report, which provides causal explanations about why the undesirable events in the accident model sequence occurred. FAA also published its NextGen implementation plans, which provide updates on the agency's overall progress and the status of NextGen efforts. As a result, FAA has improved its ability to oversee aviation safety across the national airspace system.

    Recommendation: To help improve and expand FAA's capability to use data for aviation safety oversight, the Secretary of Transportation should direct the FAA Administrator to proceed with all deliberate speed to develop the capability to model the impact of NextGen changes on the national airspace system and manage any risks emerging from these changes.

    Agency Affected: Department of Transportation

  4. Status: Closed - Implemented

    Comments: In May 2010, GAO reported on the Federal Aviation Administration's (FAA) capacity to use available data to oversee aviation safety. For some data, GAO identified requirements for reporting to FAA, such as information on accidents and engine failures. However, other data is shared voluntarily, such as violations of federal regulations or safety events self reported by pilots, mechanics, and other aviation personnel. To encourage voluntary reporting, FAA agrees not to take enforcement actions against self-reported violations of regulations. Although voluntary reporting programs can generate safety information that is not available through other sources, GAO found that participation has been limited by concerns about confidentiality and cost considerations. Because of the criticality of voluntarily reported data, GAO recommended that FAA identify the reasons that carriers are not participating in such programs and implement further steps to encourage greater program participation, especially by smaller carriers. In 2013, GAO confirmed that FAA surveyed smaller carriers to obtain reasons why they were not participating in voluntary reporting programs and to identify some of the challenges to participating in these programs. To address the identified challenges, FAA improved the functionality of its web based tool for voluntary event reporting and analysis, which is cost free for U.S. carriers participating in an Aviation Safety Action Program. To encourage greater program participation, FAA has continued to promote participation by reaching out to individual carriers. Since 2010, overall participation of carriers in at least one voluntary reporting program has increased from 59 to 70 percent. As a result of these efforts, FAA has increased its access to data that provide insights into safety events that are not available from other sources and are critical to improving aviation safety.

    Recommendation: To help improve and expand FAA's capability to use data for aviation safety oversight, the Secretary of Transportation should direct the FAA Administrator to systematically identify the reasons that carriers are not participating in voluntary reporting programs, such as through a survey, and identify and implement further steps to encourage greater program participation, especially by smaller carriers.

    Agency Affected: Department of Transportation

 

Explore the full database of GAO's Open Recommendations »

Oct 9, 2014

Sep 26, 2014

Sep 25, 2014

Sep 23, 2014

Sep 12, 2014

Jul 31, 2014

Jul 23, 2014

Jun 25, 2014

Jun 24, 2014

Looking for more? Browse all our products here