This is the accessible text file for GAO report number GAO-10-414 
entitled 'Aviation Safety: Improved Data Quality and Analysis 
Capabilities Are Needed as FAA Plans a Risk-Based Approach to Safety 
Oversight' which was released on June 10, 2010. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to Congressional Requesters: 

United States Government Accountability Office: 
GAO: 

May 2010: 

Aviation Safety: 

Improved Data Quality and Analysis Capabilities Are Needed as FAA 
Plans a Risk-Based Approach to Safety Oversight: 

GAO-10-414: 

GAO Highlights: 

Highlights of GAO-10-414, a report to congressional requesters. 

Why GAO Did This Study: 

To improve aviation safety, the Federal Aviation Administration (FAA) 
plans to have in place the initial capabilities of a risk-based 
approach to safety oversight, known as a safety management system 
(SMS), by the end of fiscal year 2010. FAA is also implementing new 
procedures and technologies to enhance the safety, capacity, and 
efficiency of the national airspace system. Data are central to SMS 
and FAA’s ability to test the impact of these changes on safety. 

This congressionally requested report addresses FAA’s (1) current and 
planned use of data to oversee aviation safety, (2) access to data for 
monitoring aviation safety and the safety performance of various 
industry sectors, and (3) efforts to help ensure data quality. To 
perform this work, GAO reviewed 13 databases that contain data on key 
aviation safety events, assessed data quality controls for the 
databases, and interviewed agency and industry officials, as well as 
10 experts in aviation safety and data. 

What GAO Found: 

FAA analyzes data on past safety events, such as engine failures, to 
prevent their recurrence and plans to use data to support a more 
proactive approach to managing risk. For example, weather and air 
traffic control data helped identify factors associated with injuries 
from turbulence. As part of SMS, FAA plans to analyze data proactively 
to support a risk-based approach to safety oversight. For example, FAA 
plans to use data to model the impact of proposed changes in 
procedures and technologies on the safety of the national airspace 
system. Experts said that identifying risks is necessary to maintain 
the current level of safety and possibly achieve a higher level of 
safety in the future. Because SMS relies on data to identify emerging 
risks, FAA has an effort under way to enhance its access to industry 
data and improve its capability for automated analysis of multiple 
databases. According to FAA, this effort will allow for more efficient 
safety analyses. FAA is also developing a plan for managing data under 
SMS, but the plan does not fully address data, analysis, or staffing 
requirements. Without such requirements, the plan will not provide 
timely guidance for implementing SMS. 

FAA has access to some voluntarily reported data, which are important 
for SMS, but not all carriers and aviation personnel participate in FAA’
s voluntary reporting programs. Such data are gathered electronically 
by equipment on aircraft or reported by aviation personnel or carriers 
following noncriminal, unintentional violations or safety events. 
Industry personnel have some incentives to participate in voluntary 
programs, such as promised immunity from disciplinary action, but 
concerns about sanctions and the cost of equipment have deterred full 
participation, especially by smaller carriers. While FAA has some 
information on reasons for nonparticipation and has taken some steps 
to promote greater participation, it lacks carrier-specific 
information on why air carriers are not participating. FAA also lacks 
data to assess the safety performance of certain industry sectors, 
such as air cargo and air ambulance operators. GAO has previously made 
recommendations to address this lack of data. FAA concurred with GAO’s 
prior recommendations and is taking actions to address them. 

To help ensure data quality—that is, data that are reliable (complete 
and accurate) and valid (measure what is intended)—FAA has implemented 
a number of data quality controls that are consistent with GAO’s 
standards for data quality, but some weaknesses exit. For example, all 
the databases GAO reviewed had at least some controls in place to 
ensure that erroneous data are identified, reported, and corrected. 
However, about half the databases lack an important control—managers 
do not review the data prior to entry into the data system. FAA is 
taking steps to address its data weaknesses, but vulnerabilities 
remain, potentially limiting the usefulness of FAA’s data for the 
safety analyses planned to support SMS. 

What GAO Recommends: 

GAO recommends efforts to improve FAA’s capability to use data for 
oversight, including developing a comprehensive data management plan; 
identifying and, to the extent feasible, addressing reasons for 
nonparticipation in voluntary reporting programs; and applying data 
quality controls to more databases, as appropriate. The agency agreed 
to consider GAO’s recommendations. 

View [hyperlink, http://www.gao.gov/products/GAO-10-414] or key 
components. For more information, contact Gerald L. Dillingham, Ph.D., 
at (202) 512-2834 or dillinghamg@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

FAA Analyzes Data on Past Safety Events to Prevent Their Recurrence 
and Plans to Use Data to Support a More Proactive Approach to Managing 
Risks: 

FAA Has Limited Access to Some Voluntarily Reported Data and Lacks Key 
Data to Assess the Safety Performance of Certain Industry Sectors: 

FAA Has Various Processes in Place to Help Ensure Data Quality, but 
Weaknesses Still Exist: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments: 

Appendix I: Objectives, Scope, and Methodology: 

Appendix II: GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: Aviation Safety Databases We Reviewed: 

Table 2: Examples of How Data Have Been Used by FAA or Industry in 
Reactive Approaches to Managing Risk: 

Table 3: Examples of Additional Controls for Managing Data Quality: 

Table 4: Aviation Safety Events Included in Our Review: 

Table 5: Organizations That GAO Contacted: 

Figures: 

Figure 1: FAA's Emphasis Is Shifting from Reactive to Proactive 
Approaches to Data Analysis: 

Figure 2: Number of General Aviation Accidents and Fatalities, 2000 
through 2008: 

Figure 3: Extent to Which Aviation Safety Databases Reviewed Have 
Standard Data Quality Controls: 

Abbreviations: 

AIDS: Accident/Incident Data System: 

ASAP: Aviation Safety Action Program: 

ASIAS: Aviation Safety Information Analysis and Sharing: 

ASRS: Aviation Safety Reporting System: 

ATO: Air Traffic Organization: 

ATOS: Air Transportation Oversight System: 

ATSAP: Air Traffic Safety Action Program: 

CAST: Commercial Aviation Safety Team: 

DOT: Department of Transportation: 

FAA: Federal Aviation Administration: 

FOQA: Flight Operational Quality Assurance: 

ICAO: International Civil Aviation Organization: 

IG: inspector general: 

NAOMS: National Aviation Operations Monitoring Service: 

NASA: National Aeronautics and Space Administration: 

NextGen: Next Generation Air Transportation System: 

NMACS: Near Midair Collision System: 

NTSB: National Transportation Safety Board: 

OEDS: Operational Error/Deviation System: 

PDS: Pilot Deviation System: 

SDRS: Service Difficulty Report: 

SMS: safety management system: 

UK: United Kingdom: 

USDA: U.S. Department of Agriculture: 

VPDS: Vehicle Pedestrian Deviation System: 

VDRP: Voluntary Disclosure Reporting Program: 

[End of section] 

United States Government Accountability Office: 
Washington, DC 20548: 

May 6, 2010: 

Congressional Requesters: 

To increase the safety, capacity, and efficiency of the national 
airspace system, the Federal Aviation Administration (FAA), in 
cooperation with aviation industry stakeholders and other federal 
agencies, is developing a new, technologically advanced air traffic 
management system--the Next Generation Air Transportation System 
(NextGen). At the same time, FAA is attempting to further enhance 
aviation safety by shifting to a new data-driven, risk-based safety 
oversight approach, which industry and international air traffic 
management organizations are also adopting. Under this new approach, 
called a safety management system (SMS) approach, FAA will continue to 
use data on aviation accidents and incidents--referred to collectively 
as safety events--to identify and address their causes. In addition, 
under SMS, FAA plans to use aviation safety data to identify 
conditions that could lead to safety events and to address them 
through changes in organizational processes, management, and culture. 
Furthermore, SMS will allow FAA to test the impact of NextGen changes 
on aviation safety and to identify safety vulnerabilities and 
mitigating measures. 

You asked us to assess FAA's capacity to use available data to oversee 
aviation safety.[Footnote 1] To do so, we addressed the following 
questions: 

1. How does FAA use data to oversee aviation safety and what changes, 
if any, has it planned? 

2. To what extent does FAA have access to data for monitoring aviation 
safety and the safety of various aviation industry sectors? 

3. What does FAA do to help ensure the quality of the data it uses to 
oversee aviation safety? 

To answer these questions, we reviewed 13 aviation safety databases 
maintained by FAA, the National Aeronautics and Space Administration 
(NASA), the National Transportation Safety Board (NTSB), and the U.S. 
Department of Agriculture (USDA). We selected these databases because 
they covered key safety events identified by an FAA-industry effort, 
the Commercial Aviation Safety Team (CAST),[Footnote 2] and NASA's 
National Aviation Operations Monitoring Service (NAOMS) project--a 
research and development effort that examined the feasibility of using 
surveys of randomly selected pilots to identify accident precursors 
and potential safety issues.[Footnote 3] Those safety events are: 

* equipment problems, including engine shutdowns and cargo shifts; 

* turbulence; 

* weather events while airborne, including icing and wind shear; 

* passenger-related events, including medical emergencies; 

* flight crew problems, such as pilot fatigue; 

* airborne conflicts, including bird strikes; 

* ground events, including near collisions on the runway; 

* aircraft handling-related events, including near collisions with 
terrain, deviations from assigned routes, and hard landings; 

* altitude deviations; and: 

* adverse interactions with air traffic control, including the 
inability to communicate with air traffic controllers. 

We also selected these databases because they contain data from a 
range of internal FAA and external sources. To determine the quality 
of these data, we collected and compared information on the data 
quality controls used by FAA and others with GAO-identified practices 
for helping to ensure data reliability.[Footnote 4] These practices 
include ensuring that the data are complete and accurate, measure 
intended safety concerns, and are useful for their intended oversight 
purposes. FAA and industry gather many other types of data--such as 
FAA's air carriers' operations and maintenance data--that we did not 
assess as part of this review. 

We also interviewed 10 aviation safety and data experts to identify 
critical data-related issues facing FAA and, where applicable, to 
assess the agency's approaches for addressing data-related issues. We 
selected experts in aircraft operations, human factors, aircraft 
maintenance, and air traffic control. These experts have general 
knowledge about aviation safety and represent a cross section of the 
aviation industry. Throughout this report we have highlighted key 
comments of these stakeholders as sidebars. In addition, we analyzed 
documents and interviewed officials from FAA, NASA, NTSB, USDA, and 
air carriers; government contractors that analyze or help collect 
selected data; aviation industry organizations; data vendors; and air 
carrier employee groups. We also used the results of GAO studies that 
considered the availability, quality, and use of data in aviation 
safety oversight. We conducted this performance audit from August 2008 
through May 2010 in accordance with generally accepted government 
auditing standards. Those standards require that we plan and perform 
the audit to obtain sufficient, appropriate evidence to provide a 
reasonable basis for our findings and conclusions based on our audit 
objectives. We believe that the evidence obtained provides a 
reasonable basis for our findings and conclusions based on our audit 
objectives. (See appendix I for additional information on our methods.) 

Background: 

Federal agencies and aviation industry stakeholders gather and analyze 
aviation data for a variety of purposes. Federal agencies gather and 
analyze aviation data primarily to improve safety. To oversee aviation 
safety across the national airspace system, FAA maintains data on 
various aviation sectors, including passenger airlines, air cargo 
carriers, general aviation, and air ambulance operators. FAA also 
gathers and analyzes data on industry performance through its 
inspection and certification programs and uses these data to ensure 
that the industry complies with its safety regulations.[Footnote 5] In 
addition, FAA obtains information on safety events and incidents 
collected by other federal agencies, including NTSB, NASA, and USDA. 

The aviation industry gathers quantitative and narrative data on the 
performance of flights and analyzes these data to increase safety, 
efficiency, and profitability. Industry stakeholders also maintain 
historical data on equipment and maintenance issues. These 
stakeholders are required to report some data to FAA, such as data on 
accidents, engine failures, and near midair collisions, and they have 
agreements with FAA and other agencies to share other data 
voluntarily. The voluntarily shared data include both electronically 
recorded data from aircraft equipment under the Flight Operational 
Quality Assurance program (FOQA) and information on violations of 
federal regulations or on safety events self-reported by pilots, 
mechanics, and other airmen under three programs--Aviation Safety 
Action Program (ASAP), Aviation Safety Reporting System (ASRS), and 
Voluntary Disclosure Reporting Program (VDRP).[Footnote 6] FAA also 
recently established the Air Traffic Safety Action Program (ATSAP), 
modeled after the airlines' ASAP program. It is a confidential, 
voluntary reporting system available to FAA's approximately 17,000 air 
traffic control personnel, who can use the program to identify and 
report safety and operational concerns.[Footnote 7] 

Table 1 describes the 13 aviation safety databases that we reviewed. 

Table 1: Aviation Safety Databases We Reviewed: 

Voluntary databases: 

Database/date established: Aviation Safety Action Program/1997; 
Responsible entity: FAA, air carriers; 
Source of data: Members of participating aviation industry employee 
groups; 
Data collected: All types of safety events; 
Data format: Narrative. 

Database/date established: Aviation Safety Reporting System/1987; 
Responsible entity: NASA; 
Source of data: Industry personnel in the air and on the ground (e.g., 
air traffic controllers, mechanics, flight attendants, and ground 
crews); 
Data collected: All types of safety events; 
Data format: Narrative, quantitative. 

Database/date established: Flight Operational Quality Assurance/1995; 
Responsible entity: FAA, air carriers; 
Source of data: Devices on specially equipped aircraft that collect 
data from the aircraft's flight data recorders; 
Data collected: In-flight operations; 
Data format: Quantitative. 

Database/date established: Voluntary Disclosure Reporting Program/1990; 
Responsible entity: FAA; 
Source of data: Air carriers, repair stations, production approval 
holders, and fractional ownership programs operating under part 91[A]; 
Data collected: All types of safety events; 
Data format: Narrative. 

Other databases: 

Database/date established: Accident/Incident Data System (AIDS)/1978; 
Responsible entity: FAA; 
Source of data: FAA inspectors; 
Data collected: Some aviation accidents and those incidents not 
investigated by NTSB[B]; 
Data format: Primarily narrative, some quantitative. 

Database/date established: Air Transportation Oversight System 
(ATOS)/1998; 
Responsible entity: FAA; 
Source of data: FAA inspectors; 
Data collected: Inspection results; 
Data format: Narrative, quantitative. 

Database/date established: National Wildlife Strike Database 
(Wildlife)/1990; 
Responsible entity: FAA, USDA; 
Source of data: Pilots, air traffic control personnel, and others 
involved in civil aviation; 
Data collected: Bird and other wildlife strikes; 
Data format: Narrative, quantitative. 

Database/date established: Near Midair Collision System (NMACS)/1987; 
Responsible entity: FAA; 
Source of data: Pilots and other flight crew members; 
Data collected: Near midair collisions; 
Data format: Narrative, quantitative. 

Database/date established: National Transportation Safety Board 
Aviation Accident and Incident Database/1982; 
Responsible entity: NTSB; 
Source of data: NTSB investigators; 
Data collected: Aviation accidents and major incidents; 
Data format: Primarily quantitative, some narrative. 

Database/date established: Operational Error/Deviation System 
(OEDS)/1985; 
Responsible entity: FAA; 
Source of data: Air traffic controllers and air traffic equipment; 
Data collected: Air traffic control operational errors and pilot 
deviations[C]; 
Data format: Primarily quantitative, some narrative. 

Database/date established: Pilot Deviation System (PDS)/1987; 
Responsible entity: FAA; 
Source of data: Air traffic controllers and other FAA personnel; 
Data collected: Pilot deviations; 
Data format: Primarily quantitative, some narrative. 

Database/date established: Service Difficulty Reports (SDRS)/1986; 
Responsible entity: FAA; 
Source of data: Pilots, mechanics, inspectors, and others; 
Data collected: Aircraft equipment problems; 
Data format: Narrative, quantitative. 

Database/date established: Vehicle Pedestrian Deviation System 
(VPDS)/1988; 
Responsible entity: FAA; 
Source of data: Air traffic controllers, other FAA and industry 
personnel; 
Data collected: Unauthorized entry on a runway by a vehicle, 
pedestrian, or aircraft; 
Data format: Narrative, quantitative. 

Sources: FAA, NASA, NTSB, USDA, and GAO. 

[A] A production approval holder is an entity that holds a 
certificate, approval, or authorization from FAA to manufacture 
aircraft, aircraft engines, propellers, and related parts and 
articles. Fractional ownership refers to shared aircraft ownership. 

[B] NTSB investigates all aviation accidents and major aviation 
incidents. 

[C] An operational error/deviation is a violation of FAA separation 
standards that define minimum safe distances between aircraft, between 
aircraft and other physical structures, and between aircraft and 
otherwise restricted airspace. The event is classified as an 
operational error when an air traffic controller's actions cause the 
loss of minimum separation; the event is classified as a pilot 
deviation when a pilot's actions cause the loss, or both. 

[End of table] 

FAA Analyzes Data on Past Safety Events to Prevent Their Recurrence 
and Plans to Use Data to Support a More Proactive Approach to Managing 
Risks: 

FAA Is Shifting from a Reactive to a Proactive Approach to Using Data 
to Manage Risk: 

For decades, the aviation industry and federal regulators, including 
FAA, have used data reactively to identify the causes of aviation 
accidents and incidents and take actions to prevent their recurrence. 
Since 1998, for example, FAA has partnered with the airline industry 
through CAST with the goal of continuously improving aviation safety. 
Over the years, CAST has looked at the causes of past accidents--such 
as controlled flight into terrain[Footnote 8]--and various safety 
events--such as turbulence or runway incursions. CAST analyzes past 
instances of such accidents and events to identify precipitating 
conditions and causes. CAST then uses its analysis to formulate an 
intervention strategy designed to reduce the likelihood of a 
recurrence and validate the effectiveness of the intervention. 
According to CAST, its work has helped to decrease commercial airline 
fatalities--exceeding its goal to reduce fatal commercial accidents by 
80 percent by 2007--and is an important aspect of FAA's efforts to 
increase aviation safety by sharing and analyzing data. Table 2 
provides examples of how FAA and industry have used CAST's work. 
(Recent work by CAST to work with FAA's Aviation Safety Information 
Analysis and Sharing [ASIAS] initiative to develop safety enhancements 
and mitigate future threats is discussed later in this report.) 

Table 2: Examples of How Data Have Been Used by FAA or Industry in 
Reactive Approaches to Managing Risk: 

Safety event identified by CAST as important: Airborne conflict; 
Use of data to manage risk: Reporting of bird strikes has led to the 
identification of runways at over 290 airports that may require 
wildlife mitigation efforts. 

Safety event identified by CAST as important: Aircraft handling; 
Use of data to manage risk: A review of ASRS reports identified a risk 
of potential midair collisions at the Minden-Tahoe airport in Nevada 
near a parachuting center's drop zone, and the drop zone was moved to 
reduce this risk. 

Safety event identified by CAST as important: Controlled flight into 
terrain; 
Use of data to manage risk: A study of the Terrain Awareness and 
Warning System--which warns pilots that their aircraft is about to 
collide with terrain--revealed that all air carrier flight crew 
training should emphasize situational awareness and escape procedures 
in the event of a warning. 

Safety event identified by CAST as important: Flight deck automation; 
Use of data to manage risk: A review of automation-related events 
found inadequate pilot training and knowledge of automation systems 
and led airlines to change their automation policies and training to 
address these deficiencies. 

Safety event identified by CAST as important: Runway incursion; 
Use of data to manage risk: Analysis of data on the frequency of 
unauthorized entry or activity on an airport runway by a vehicle, 
pedestrian, or aircraft led to changes in signage at all airports with 
more than 370,000 annual enplaned passengers. 

Safety event identified by CAST as important: Turbulence; 
Use of data to manage risk: Integration of meteorological, crew 
activity, and air traffic control data led to the identification of 
factors associated with injuries from turbulence and recommendations 
for improvements in the use of available information and in 
communications between pilots and flight attendants and passengers. 

Safety event identified by CAST as important: Uncontained engine 
failure; 
Use of data to manage risk: An analysis of the causes of engine-
related accidents led FAA to issue an airworthiness directive 
requiring airlines to inspect engine components. 

Sources: CAST, FAA, and GAO. 

[End of table] 

Besides analyzing data on past safety events to develop intervention 
strategies, FAA staff perform such analyses to inform changes in 
agency policies. For example, to inform the rule-making requirement 
that the costs and benefits of a proposed regulation be determined 
under Executive Order 12866, FAA analysts identified the number of 
aircraft that could be certified as "light sport" aircraft and were 
involved in accidents.[Footnote 9] In some cases, FAA program managers 
request specific analyses to inform policy changes. For instance, in 
response to a 2007 recommendation by NTSB and a petition from Hawaiian 
air tour operators, FAA program managers requested an analysis of 
aircraft crashes associated with FAA's requirement for tour aircraft 
to maintain a 1,500-foot separation from the ground. After analyzing 
air crashes involving Hawaiian tour aircraft 13 years before and 13 
years after the requirement was implemented, FAA concluded that the 
requirement helped to reduce the number of crashes and significantly 
improved safety. 

While FAA will continue to use data to analyze past safety events, it 
is also working to use data proactively to search for risks and take 
actions to mitigate them before they result in accidents. FAA's 
emphasis is shifting to a proactive approach to data analysis because 
as accidents have become increasingly rare, less information is 
available for reactive analyses of their causes. As a result, 
according to a study of FAA's safety oversight by a 2008 independent 
review team, information that can be used to help identify accident 
and incident precursors, such as voluntarily reported data, has become 
more critical for accident prevention.[Footnote 10] In addition, 
several experts we spoke with said that proactively identifying risks 
is necessary to maintain the current level of safety and possibly 
achieve an even higher level of safety in the future. FAA is 
undertaking this transition in coordination with the international 
aviation community, working with the International Civil Aviation 
Organization (ICAO) to adopt applicable global standards for safety 
management. Senior FAA officials and ICAO agree that effective safety 
management is data driven and that data are essential to identifying 
emerging risks. Figure 1 illustrates the type of transition FAA plans 
as the agency shifts its emphasis to a proactive assessment of 
emerging safety risks, according to FAA officials. 

Figure 1: FAA's Emphasis Is Shifting from Reactive to Proactive 
Approaches to Data Analysis: 

[Refer to PDF for image: illustration] 

Reactive approach: 
* Accident occurs; 
* Accident investigation (data accumulation and integration); 
* Cause identification (decomposition of causes); 
* Precursor identification; 
* Recommendation (cost/benefit assessment). 

Proactive approach: 
* Emerging risk; 
* Precursor identification (data fusion and integration); 
* Risk characterization (model: exposure and consequence estimate); 
- Possible accident cause linkage (event sequence analysis; 
* Risk prioritization (model: systemwide impacts); 
* Recommendation (cost/benefit assessment). 

Source: FAA and GAO. 

[End of figure] 

FAA Plans to Use Data Proactively to Model the Impact of NextGen 
Changes on the Safety of the National Airspace System: 

The new technologies and procedures that FAA will implement for 
NextGen, which are intended to increase the safety, efficiency, and 
capacity of the national airspace system, could also lead to 
consequences that have unintended effects on system safety. For 
example, NextGen changes to landing procedures, which are designed to 
allow more frequent landings, could reduce congestion in the air and 
improve fuel efficiency, but might have the unintended effect of 
increasing congestion and safety risks on airport taxiways. To avoid 
such unintended consequences, FAA plans, as it improves its ability to 
integrate data and analyze trends, to model the impact of changes 
planned for NextGen. To do so, it has begun to develop a baseline of 
current conditions and then expects to analyze how NextGen changes 
will affect those conditions, according to a senior FAA official. FAA 
is in the process of designing tools that will allow it to model the 
changes. For example, a project called the National Level System 
Safety Assessment is designed to allow FAA to assess risks across the 
national airspace system. Currently, FAA assesses risks for specific 
NextGen procedures and technologies, but cannot model the risks across 
the national airspace system in a comprehensive manner. This project 
will integrate data on past safety events from a number of FAA offices 
and external sources to proactively identify risks that might emerge 
with the introduction of changes planned for NextGen. FAA has begun to 
obtain some operational data for the project and has contracted with 
the Volpe National Transportation Systems Center, which will be 
responsible for integrating airport runway surface data, including 
surface radar, weather, aircraft, and other data.[Footnote 11] A 
senior FAA official told us that although safety assessments had been 
conducted on individual NextGen technologies, until the agency has 
finalized this modeling project, it cannot begin systemwide 
assessments of the safety of NextGen technologies and procedures that 
are already being deployed, including 700 new navigational procedures 
that had been deployed as of October 2009. 

SMS and Its Use of Data Form the Linchpin of FAA's Risk-Based 
Approach, but Full Implementation of SMS Could Be Years Away: 

SMS is an integrated, data-driven approach to managing safety risk 
that FAA expects will help it continuously improve aviation safety. 
[Footnote 12] FAA stated that successfully implementing SMS is 
critical to meeting the challenges of a rapidly changing and expanding 
aviation system. As stated earlier in this report, FAA's traditional 
approach is to analyze data to determine the causes of an accident or 
incident after the fact. To achieve the next level of safety, FAA 
says, it now requires a more forward-thinking approach, which SMS 
provides, to identify systemwide trends and manage emerging risks 
before they result in incidents or accidents. To identify emerging 
risks, FAA plans to collect and analyze safety data, and it can then 
use the results of its analyses to make data-driven decisions about 
how to address safety risks. Issued in September 2008, FAA's guidance 
on implementing SMS explains the importance of data collection and 
analysis to the execution of SMS.[Footnote 13] This guidance defines 
four main components for SMS: safety policy and objectives, safety 
risk management, safety assurance, and safety promotion. First, safety 
policy and objectives describe an organization's requirements and 
oversight responsibilities for aviation activities. Second, safety 
risk management describes how an organization will identify hazards 
and safety risks in aviation operations, including how it will develop 
rules, regulations, and safety performance measures. Third, the safety 
assurance component of SMS uses data analyses to discover emerging 
risks and to model the impact of safety changes. Fourth, safety 
promotion is an organization's plan to ensure that training, 
communication, and dissemination of safety information take place. 
According to FAA, SMS with these components will enable the agency to 
provide the air transportation system and the public at large with 
enhanced safety. 

FAA's goal is for the Office of Aviation Safety to have initial 
operating capabilities in place for SMS by the end of fiscal year 
2010. According to FAA officials, these initial operating capabilities 
include training employees and defining how to apply SMS to the 
agency's overall oversight activities. To accomplish this, FAA's 
guidance for implementing SMS requires the Office of Aviation Safety, 
the Office of Airports, and the Air Traffic Organization (ATO) each to 
develop implementation plans that include schedules, procedures for 
acquiring and analyzing data, and measures to track implementation 
progress. ATO has issued its SMS implementation plan and has also 
created an SMS manual that provides specific operational information 
and guidance regarding the daily activities of ATO employees. In 
addition, it met its target to implement its initial SMS operating 
capabilities in March 2010. The Offices of Airports and Aviation 
Safety have yet to issue their implementation plans. However, the 
Office of Aviation Safety has issued guidance and safety documents 
that provide a general discussion of SMS, but they do not include a 
schedule of specific activities or time frames for completion as 
called for in the agencywide SMS guidance. Senior FAA officials told 
us Aviation Safety has formed working groups and expects those groups 
that are charged with defining the various SMS activities to meet the 
guidance requirements. In addition, according to FAA officials, 
various offices within Aviation Safety will be responsible for 
implementing processes to fulfill SMS requirements. 

Additionally, FAA's guidance for implementing SMS requires the 
formation of an agencywide SMS committee, which was chartered in July 
2009. The committee includes an executive council--whose members are 
the FAA associate administrators from the offices of Aviation Safety, 
Airports, and Commercial Space Transportation, and the Chief Operating 
Officer of ATO--and a committee composed of SMS professionals from 
each of those FAA offices. Chaired by the Office of Aviation Safety, 
the agencywide committee is tasked with recommending policy and 
guidance to the implementing FAA organizations and management. 

Because SMS relies on data to identify emerging risks, FAA has an 
effort under way to enhance its access to industry data and improve 
its analysis capability. The ASIAS initiative is a collaborative 
government-industry effort to share and analyze data. The Office of 
Aviation Safety's SMS implementation plan reiterates that data 
exchanges between ASIAS and other sources are crucial elements of 
emerging risk analysis. FAA's draft plan for ASIAS notes that this 
effort will require access to existing and previously unattainable 
data sources, enhanced analytical methodologies, and technical 
advancements to support safety risk analysis that are not achievable 
with current databases and analytical strategies. The draft plan would 
cover ASIAS activities through 2022. FAA did not confirm when a final 
plan would be completed. 

While FAA has issued agencywide guidance on implementing SMS and has 
some efforts such as ASIAS under way, it does not have a way to 
measure or specific times to indicate full implementation. FAA 
officials told us that the current efforts would provide a foundation 
for the full implementation of SMS. But without a clear description of 
the activities to be completed and time frames for their completion, 
it may be years before SMS is fully implemented and its benefits are 
realized. In commenting on a draft of this report, FAA officials noted 
that even with a clear description of the activities to be completed 
and time frames for their completion, it will be years before SMS is 
fully implemented and its benefits are realized. We agree with FAA and 
note that specific time frames establish expectations for FAA's 
implementation of SMS and provide a means of accountability for 
meeting those expectations. 

FAA Has Identified and Is Addressing Challenges to Using Data for 
Safety Oversight: 

FAA has efforts under way to address two key challenges to using data 
more effectively to manage risk. First, data are not coded to permit 
electronic integration, analysis, and sharing. Second, data from two 
voluntary reporting programs lack identifying details needed for some 
types of analysis, and the data do not remain available for long-term 
analysis. 

Data Are Not Coded to Permit Automated Analysis: 

FAA and the other federal agencies that gather the data FAA uses for 
analysis lack standard definitions, common identifiers, and common 
classification schemes for both quantitative and narrative data. For 
example, according to an NTSB official, databases used in ASIAS were 
not designed for searches of specific products, manufacturers, or 
airlines. As a result, FAA has had to develop common identifiers and 
standard classification schemes so that automation can be used to 
quickly integrate and analyze data from multiple external and internal 
sources. In addition, narrative information, in particular, poses 
challenges. Narrative reports from NTSB investigators and FAA 
inspectors, as well as narrative reports submitted voluntarily by 
pilots, air traffic controllers, mechanics, and others, constitute a 
rich source of information about safety events, but could not be coded 
to permit automated analysis until the ASIAS text analysis process was 
developed.[Footnote 14] Consequently, analysts are now able to 
automate coding, integration, and analysis of data from different 
databases--a time-consuming and costly process when manually 
performed. According to a senior FAA official, this automated analysis 
process is unique and pathbreaking and will allow for more efficient 
safety analyses. When FAA studied wrong runway departures, for 
example, data from six different databases--from NTSB, NASA, and its 
own offices--were extracted for analysis. FAA compared the cost and 
time needed to extract and integrate data from multiple databases 
using an automated versus a manual process for coding the data. FAA 
found that using the automated coding process would cost about $6,000 
and take 52 hours, while the manual process was estimated to cost over 
$750,000 and take more than 6,500 hours to complete. However, NASA 
noted that by using a "data mapping" approach, it can conduct an 
automated analysis of ASRS and other data sources in about 2 to 8 
hours, depending on the complexity of the analysis. 

Data from Voluntary Reporting Programs Have Characteristics That 
Impede Analysis: 

Details of reported incidents are redacted from ASAP and FOQA data 
before an FAA contractor analyzes the data. These details include the 
date, time and flight number, and the names of the carrier or 
individuals involved. We previously reported that these identifying 
details are redacted to safeguard the participants in ASAP from 
enforcement or disciplinary action and participants in FOQA from 
public release of the data.[Footnote 15] Additionally, ASAP and FOQA 
data are retained for only 3 years. Without identifying details and 
without maintaining data for longer periods, opportunities for some 
analyses are limited. To allow the contractor to perform more detailed 
analyses for FAA, the agency and the industry have agreed on a process 
through which the ASIAS Executive Board provides permission for the 
contractor to perform a specific, defined analysis and to use data 
with the identifying details needed for that particular analysis. At 
the conclusion of the analysis and with the approval of the ASIAS 
Executive Board, a final report containing detailed results is given 
to CAST to develop safety enhancements to mitigate the identified 
safety issues. As of March 2010, three such analyses had been 
completed. 

The Department of Transportation (DOT) Inspector General (IG) and the 
2008 independent review team found that FAA could improve its use of 
ASAP data by analyzing these data for national trends, noting that by 
not doing so, the agency is missing opportunities to help reduce 
recurrences of safety events or to identify national patterns 
indicative of risks.[Footnote 16] FAA concurred with this finding and 
said that it already has the ability to conduct national trend 
analyses. The IG reported, however, that FAA receives quarterly 
summaries of ASAP information from carriers of how many ASAP reports 
they received each month, and these summaries do not provide 
sufficient detail about ASAP events or corrective actions. According 
to the IG, these reports generally contain information about the 
number but not nature of ASAP submissions for that quarter and any 
resulting safety enhancement.[Footnote 17] 

While FAA's contractor loses access to ASAP reports after 3 years, 
about 62 percent of ASAP reports appear in ASRS, along with other 
reports voluntarily submitted by industry personnel, according to a 
NASA official.[Footnote 18] NASA, which maintains ASRS for FAA, 
[Footnote 19] has access to the identifying details in the submitted 
reports for no more than 90 days so it can follow up as necessary to 
clarify any questions that the reports raise. After that time, NASA 
removes the identifying details and incorporates ASAP and ASRS reports 
into ASRS. ASRS is accessible to the public, and NASA performs special 
analyses of ASRS data for FAA at its request. NASA also publishes an 
online newsletter every month with summary statistics and examples of 
a few reports selected to illustrate certain safety issues, such as 
general aviation pilots' use of new cockpit technology, and issues 
safety alerts regarding significant safety events identified by ASRS 
reports. However, NASA does not comment on these reports or respond to 
any questions or issues raised by the authors, since its role is to 
act as an "honest broker" as it collects and analyzes the data. By 
comparison, in the United Kingdom (UK), an advisory board reviews a 
selection of reports received, commenting on the appropriateness of 
the action taken and answering questions. These responses are useful 
to the aviation community because they communicate on commonly 
experienced safety issues. In commenting on a draft of this report, 
NASA indicated that the UK model would not be practicable for ASRS 
data because the UK receives only about 300 reports annually compared 
with about 50,000 reports for ASRS. However, NASA noted that in the 
past it had an ASRS advisory committee that had provided a forum for 
FAA and industry to discuss corrective action. The agency acknowledged 
the need to reestablish this committee. In addition, NASA noted that 
it receives feedback on its safety alerts, which indicated positive 
corrective actions for about 60 percent of the alerts. 

FAA Is Developing a Plan for Using Data under SMS, but the Plan Does 
Not Fully Address Data, Analysis, and Staffing Requirements: 

As part of its efforts to develop initial SMS capabilities, FAA 
expects to address how data and analysis will help it identify 
emerging aviation safety risks. Specifically, FAA has a draft plan--
which it calls a safety management plan[Footnote 20]--that defines the 
agency's analytical requirements and the role of safety analysis in 
improving safety, especially as NextGen is implemented. FAA's draft 
plan--which covers the risk management and safety assurance components 
of SMS discussed earlier in this report--recognizes the agency's 
future need for data and analysis, but does not specify requirements 
for them. For example, the draft safety management plan does not 
define the level of accuracy and completeness needed for the data, 
indicate what metrics and processes FAA will use to assess the data, 
or identify any specific data. According to senior FAA officials, as 
of February 2010, there was no specific date for finalizing the draft 
plan. 

[Sidebar: Experts Say: 
FAA needs analysts with operational experience as well as statistical 
skills. FAA needs to hire more analysts with operational and 
statistical knowledge to analyze data. End of sidebar] 

To meet its data challenges and develop needed analytical approaches, 
FAA will also have to identify staff with both aviation operational 
experience and statistical expertise to effectively analyze aviation 
safety data in the future. While aviation safety experts we 
interviewed were generally satisfied with the qualifications of 
current FAA analysts, they expressed concerns about FAA's capacity to 
meet future needs. Several experts we interviewed said FAA needs 
additional qualified analysts, but the draft safety management plan 
does not mention staffing requirements for implementing FAA's analysis 
strategy. In our 2009 report on FAA's human capital system, we made 
several recommendations to FAA on how it can help ensure the continued 
hiring, recruitment, and retention of staff needed to operate the 
national airspace system.[Footnote 21] According to FAA, it has not 
yet determined how many additional analysts it will need; however, the 
Office of Accident Investigation and Prevention has been approved to 
hire 11 additional analysts in fiscal year 2010. Without a plan that 
includes data and analysis requirements and staffing needs, the agency 
will not be able to link the resources it needs to the data 
capabilities it requires for its risk-based approach. 

FAA Has Limited Access to Some Voluntarily Reported Data and Lacks Key 
Data to Assess the Safety Performance of Certain Industry Sectors: 

Voluntary Reporting Programs Generate Safety Information That Is Not 
Available from Other Sources, although the Data Have Some Limitations: 

[Sidebar: Experts Say: 
Voluntary programs: 
* Enable “a forward look at risk.” 
* Are the way to expose and address problems; without them, the risk 
level would go up.
* Are the best source of information for hidden risk in the system.
* “FOQA and ASAP data are the best data we have.”
* Will be key to evaluating the effectiveness of NextGen. They provide 
insight into problems that would not be shared any other way. End of 
sidebar] 

FAA's voluntary reporting programs--ASAP, ASRS, FOQA, and VDRP-- 
generate safety information that FAA does not identify through other 
means. Whereas data from other sources are derived from inspections, 
audits, and other agency reports, these voluntary programs rely solely 
on cooperation between FAA and industry personnel. To obtain 
voluntarily reported information that can be used to improve safety, 
FAA agrees not to take enforcement action against carriers or industry 
personnel who self-report violations through ASAP, ASRS, and VDRP. 
Similarly, carriers that operate ASAPs agree not to take disciplinary 
action against personnel who self-report violations of FAA regulations 
or carriers' operating procedures. In both cases, this agreement holds 
only for actions that are reported in a timely manner, were not 
intentional or criminal, did not involve drugs or alcohol, did not 
result in accidents, and have not already been detected by FAA. 
Conversely, personnel who do not voluntarily report violations within 
the specified time face the threat of enforcement or disciplinary 
action if the violations are discovered later. This combination of 
promised immunity for self-reporting and threat of enforcement and 
disciplinary action for remaining silent creates an incentive for 
industry personnel to participate in the voluntary reporting programs. 

Through ASAP, ASRS, and VDRP, airspace users, including air carriers, 
air operators, and employees, self-report safety events and violations 
of their operating certificates or company procedures. For example, 
under ASAP, an employee reports an incident or event, which an event 
review committee, composed of representatives from FAA, the carrier, 
and the applicable employee group, then reviews. The ASAP event review 
committee assesses a report to determine (1) if it meets the criteria 
previously mentioned for inclusion in the program and (2), if 
included, what follow-up actions, enhancements, or mitigations should 
be implemented to address the safety concern. 

While ASAP, ASRS, and VDRP rely on individuals or the air carrier to 
file reports, FOQA data are generated by electronic equipment on 
aircraft, which continuously records more than a thousand parameters 
of data for individual flights. Vendors collect the data from carriers 
and can then, on their behalf, analyze the data and transfer files to 
the carriers' internal analysis teams or forward the data files to an 
FAA contractor for inclusion in ASIAS and subsequent analysis if a 
carrier is partnering with FAA. The FAA contractor also receives ASAP 
reports approved by event review committees. The contractor aggregates 
and analyzes the data from participating carriers and, starting in 
2010, will brief the ASIAS Executive Board on a quarterly basis. These 
briefings will consist of the status reports on directed studies, the 
number of FOQA and ASAP records, and industry benchmarks that will 
enable carriers to compare their individual safety performance 
relative to the national trends and prioritize their internal safety 
initiatives. According to FAA, 28 carriers participate in ASIAS. All 
ASIAS participants share ASAP data, and 13 carriers also share FOQA 
data. FAA estimates that airlines contributing ASAP data to ASIAS 
account for 80 percent of the flights of all commercial airlines with 
FAA-approved ASAP programs. 

FAA and industry officials, as well as experts we talked to, agreed 
that voluntarily reported data are critical to improving aviation 
safety. Moreover, according to ICAO, FAA officials, and safety experts 
we interviewed, voluntary reporting by operational personnel is a 
cornerstone of SMS, because, as ICAO has stated, operational personnel 
are in the best position to report the existence of safety hazards and 
to attest to what works and does not work during everyday operations. 
In 2007, NTSB reported that FOQA and ASAP programs are relevant to the 
safety assurance component of SMS because they provide a direct means 
for air carriers to evaluate the quality of their training and 
operations.[Footnote 22] Further, these programs can also be used in 
the safety risk management component of SMS because they can help 
identify emerging risks. In addition, FAA and industry officials 
agreed that voluntary programs help promote a healthy reporting 
culture and an increased awareness of safety by industry personnel. 
Furthermore, FAA officials told us they believe that the voluntary 
programs, such as ASAP, gather safety information that would not be 
discovered 95 percent of the time. In addition, in commenting on a 
draft of this report, NASA noted that voluntarily reported data are 
valuable in learning why an event occurred. 

Industry officials told us how their companies have used voluntarily 
reported data to implement changes that respond to safety concerns. 
For example, one airline analyzed ASAP data to decrease the number of 
unstable approaches.[Footnote 23] Safety officials from another 
airline told us that ASAP reports and FOQA data helped them to 
identify potential pilot issues, suggest additional training, and 
adjust processes and checklists based on human factors issues. They 
further commented that if pilots do not self-disclose potential safety 
issues, airlines may be limited in their ability to identify emerging 
safety trends. 

Because of the importance of voluntarily reported data to proactive 
safety analysis, NTSB and the DOT IG have also recommended that FAA 
further encourage participation in voluntary programs. For example, in 
2010, NTSB recommended that FAA require the establishment of FOQA 
programs by carriers regulated under part 121.[Footnote 24] As another 
example, the DOT IG reported in May 2009 that FAA was not realizing 
the full benefits of ASAP and recommended that the agency develop a 
central database for all air carriers' ASAP reports to be used for 
nationwide trend analysis.[Footnote 25] 

While voluntarily reported data have been used to enhance safety, they 
also have some limitations. First, the completeness of the data is 
unknown, since reporting is voluntary, and there is no way to know how 
many violations or safety situations are not reported. For example, 
according to FAA, factors such as an individual's awareness of ASRS, 
motivation to report a situation, and perception of an incident's 
severity may influence the decision to submit an ASRS report. Second, 
the completeness of the data is further limited because participation 
varies among airlines, with 73 airlines participating in ASAP and 36 
participating in FOQA. Third, as discussed later in this report, 
inadequate data quality controls can also limit the completeness of 
the data. For example, controls may not be adequate to ensure that the 
data entered into a database have been accurately compiled or 
processed. Fourth, the accuracy of voluntarily reported data cannot 
always be verified. Voluntarily reported data are subjective and are 
not always accompanied by supporting documentation, such as 
statistics, measurements, or other quantifiable information related to 
the reported events. According to an FAA analyst, a variety of factors 
can influence the accuracy of the data, including the reporter's 
experience, visibility conditions, the duration of the event, and any 
trauma experienced by the reporter. FAA notes, for example, that even 
senior pilots' estimates of how far aircraft descend during encounters 
with turbulence often differ considerably from the actual distances 
recorded on aircraft flight data recorders. Acknowledging this 
limitation, NASA notes that the information is nonetheless valuable, 
since a reporter is providing information on how it perceived the 
situation, which in large part determines its reactions. Fifth, 
electronically collected data from FOQA also have limitations. Vendors 
that process FOQA data explained that software bugs and inaccurate 
sensors can affect data results. To mitigate these problems, vendors 
review anomaly reports and validate data prior to analysis. 

In an effort to address the limitations of voluntarily reported data, 
NASA developed a survey methodology project--NAOMS--in 1997 to 
systematically collect information on safety events by conducting 
telephone interviews with randomly selected airspace users such as 
pilots. In our 2009 assessment of the survey, we concluded that NAOMS 
was a successful proof of concept and that a similar project, 
adequately funded and appropriately planned, could enhance FAA's 
safety knowledge.[Footnote 26] However, FAA maintained that FOQA 
provides better data by providing precise rates of occurrence on 
multiple parameters collected by flight data recorders that could 
obviate the benefits from NAOMS data. Nonetheless, we concluded that 
the NAOMS survey could be useful in complementing other databases, 
such as ASRS. The survey data, when properly analyzed, could be used 
to call attention to low-risk events that could serve as potential 
indicators for further investigation in conjunction with other data 
sources. Furthermore, in commenting on a draft of this report, both 
NTSB and NASA agreed on the usefulness of a survey similar to NAOMS in 
complementing other data. NASA pointed out that NAOMS-type data could 
provide the data for trends and explain "what" is happening in the 
system while ASRS provides "why" it is happening. NTSB further noted 
that its investigations of numerous serious incidents and accidents 
found that FOQA data gave no indication of underlying problems. 

Participants' Concerns Limit FAA's Access to Voluntarily Reported Data: 

Although FAA, carriers, and experts we interviewed agreed that 
voluntarily reported data are an important source of information for 
understanding and addressing safety issues, some carriers and industry 
personnel are not participating in FAA's ASAP and FOQA voluntary 
reporting programs. According to airline and FAA officials, two 
factors have primarily affected participation: (1) the fears of 
employees that their employers will take disciplinary action to 
address self-reported violations and (2) the costs to the airlines of 
purchasing and installing FOQA equipment and analyzing the data. 
[Footnote 27] 

[Sidebar: Air Carrier and Union Experts Say: 
Confidentiality is key to all the voluntary programs. Protection for 
participants is key to ensuring continued and increased participation in
voluntary programs. End of sidebar] 

Specifically, we found that, partly because of employees' fears of 
disciplinary action, from 2006 through 2008, four large air carriers 
[Footnote 28] and their pilot unions suspended their ASAP.[Footnote 
29] According to safety officials at one airline, pilots had raised 
concerns about letters of reprimand or unpaid time off that directly 
resulted from ASAP reports. Pilot unions and air carriers that we 
spoke with agreed on the importance of confidentiality. However, pilot 
union officials we interviewed also expressed concern about whether 
airlines were ensuring the confidentiality of ASAP reports and 
suggested that, for an airline's safety culture to improve, pilots, in 
particular, must be able report certain events without fear of 
reprisal. Despite these concerns, as of June 2009, the four carriers 
and pilot unions that had suspended their ASAP had restarted or were 
restarting their participation in the program. 

Additionally, according to industry officials and experts we 
interviewed, the costs of purchasing and installing equipment and 
analyzing data have deterred participation in FOQA, especially for 
smaller carriers.[Footnote 30] Several large carriers we interviewed 
said more than 50 percent of their fleets already have FOQA equipment 
and they plan to expand their fleets' capability by retrofitting 
aircraft or ensuring that new aircraft include the equipment. By 
contrast, officials from smaller carriers were concerned about costs 
and estimated that installing FOQA equipment would cost an average of 
$12,000 for each new aircraft and up to $35,000 for retrofits of older 
aircraft models. As a result of cost concerns, according to airline 
officials, only 11 of 65 smaller carriers have approved FOQA programs, 
and according to FAA, an additional small carrier FOQA program is 
pending. Officials from these smaller carriers said that incentives to 
cover equipment costs, which are not currently available, would help 
increase participation. FAA officials noted that as carrier fleets 
age, newer replacement aircraft will already be fitted for FOQA 
equipment and, therefore, costs for participating will continue to 
decrease. However, the life span of an aircraft is usually at least 30 
years for large carriers, so the transition to a fully equipped fleet 
could take decades. 

Currently, large carriers are the principal participants in FOQA and 
ASAP, and they provide service for the majority of passengers on 
domestic and international flights. Nonetheless, we found that 4 of 
the 25 large carriers do not have active FOQA programs and 1 large 
carrier did not have an ASAP. According to FAA officials, an 
additional 4 carriers have FOQA programs pending. To the extent that 
operators do not participate in the programs, they do not obtain 
information that they could use to monitor and improve the safety 
performance of their aircraft, related equipment, and personnel, and 
to the extent that they do not partner with FAA, opportunities to 
identify nationwide safety trends and improvements are lost. To 
encourage greater participation in FOQA and ASAP, FAA provides 
training to smaller carriers on how to develop versions of both 
programs that do not require as much capital investment but do allow 
the carriers to collect unique safety data. However, FAA lacks carrier-
specific information on why air carriers are not participating in 
voluntary reporting programs. Having such information would allow FAA 
to identify further actions to encourage participation. 

FAA Lacks Data to Assess the Safety of Certain Industry Sectors: 

FAA's ability to monitor and manage risk for certain industry sectors, 
such as general aviation, air ambulance operators, and air cargo 
carriers, is limited by incomplete data. While FAA collects data on 
actual flight hours and numbers of departures for large air carriers 
that operate under part 121 and scheduled flights with fewer than 10 
seats that operate under part 135, it does not collect actual flight 
activity data for smaller air carriers that provide on-demand service, 
such as air taxis and air ambulances, and general aviation operators-- 
sectors that have had a higher number of fatal accidents in recent 
years than large air carriers. For instance, in 2008, large air 
carriers providing scheduled service had 20 accidents, none of which 
were fatal. By comparison, during that same year, there were 1,559 
general aviation accidents, including 275 fatal accidents involving 
495 fatalities. (Figure 2 shows the trends in general aviation 
accidents.) Without information on the number of general aviation 
flights, FAA cannot compare the safety performance among industry 
sectors or assess trends. 

Figure 2: Number of General Aviation Accidents and Fatalities, 2000 
through 2008: 

[Refer to PDF for image: combination stacked vertical bar and line 
graph] 

Year: 2000; 
Nonfatal accidents: 1,492; 
Fatal accidents: 345; 
Fatalities: 596. 

Year: 2001; 
Nonfatal accidents: 1,402; 
Fatal accidents: 325; 
Fatalities: 562. 

Year: 2002; 
Nonfatal accidents: 1,370; 
Fatal accidents: 345; 
Fatalities: 581. 

Year: 2003; 
Nonfatal accidents: 1,388; 
Fatal accidents: 352; 
Fatalities: 633. 

Year: 2004; 
Nonfatal accidents: 1,303; 
Fatal accidents: 314; 
Fatalities: 559. 

Year: 2005; 
Nonfatal accidents: 1,349; 
Fatal accidents: 321; 
Fatalities: 563. 

Year: 2006; 
Nonfatal accidents: 1,213; 
Fatal accidents: 307; 
Fatalities: 705. 

Year: 2007; 
Nonfatal accidents: 1,362; 
Fatal accidents: 288; 
Fatalities: 496. 

Year: 2008; 
Nonfatal accidents: 1,284; 
Fatal accidents: 275; 
Fatalities: 495. 

Source: NTSB. 

[End of figure] 

Additionally, the number of accidents for air ambulance and air cargo 
operators points to safety vulnerabilities in these areas. From 1998 
through 2008, the air ambulance sector averaged 13 accidents per year. 
While the total number of air ambulance accidents peaked at 19 in 
2003, the number of fatal accidents peaked in 2008, when 9 fatal 
accidents occurred. Without data on the number of flights or flight 
hours, FAA and the air ambulance industry are unable to determine 
whether the increased number of accidents has resulted in an increased 
accident rate, or whether it is a reflection of growth in the 
industry. According to FAA, it annually surveys a sample of 
potentially active general aviation aircraft, and in the latest survey 
in 2008, it surveyed all air ambulance operators. However, we noted in 
our April 2009 testimony that less than 40 percent of the air 
ambulance operators responded, raising questions about the reliability 
of the information collected.[Footnote 31] Similarly, our review of 
air cargo safety found that small cargo carriers had more accidents 
and fatal accidents than large cargo carriers, but the available 
information was not sufficient to assess the significance of this 
difference. We found that smaller air cargo carriers averaged 29 
accidents per year from 1997 through 2008, while large cargo carriers 
averaged 8 accidents each year during this period. However, a lack of 
operations data for small cargo carriers makes it difficult for FAA to 
prioritize risks and better target safety improvements and oversight 
to the areas of highest risk. 

To address the lack of data, we previously recommended that FAA 
identify the data necessary to better understand the air ambulance 
industry and develop a systematic approach for gathering and using 
these data.[Footnote 32] Similarly, we recommended that FAA gather 
comprehensive and accurate data on smaller air cargo operations (those 
covered under part 135) to gain a better understanding of air cargo 
accident rates and better target safety initiatives.[Footnote 33] FAA 
agreed with both recommendations, but has not fully addressed either. 
In response to our recommendation on air ambulance data, FAA has 
surveyed all helicopter air ambulance operators to collect flight 
activity data. However, as mentioned earlier in this report, FAA's 
survey response rate was low, raising questions about whether this 
information can serve as an accurate measure or indicator of flight 
activity. FAA plans to evaluate ways to collect the air cargo data 
over the long term. 

FAA Has Various Processes in Place to Help Ensure Data Quality, but 
Weaknesses Still Exist: 

FAA Has Taken a Number of Steps to Help Ensure Data Quality: 

FAA, along with the international aviation community, recognizes that 
high-quality data--that is, reliable, valid data--are essential to the 
effectiveness of a data-driven approach to safety, such as SMS. 
[Footnote 34] To help ensure data quality, FAA has issued guidance, 
established procedures, and implemented controls. For example, FAA has 
issued an order that establishes an agencywide policy on data 
management.[Footnote 35] This policy applies to all information from 
FAA and other sources used to perform the agency's mission. In 
accordance with the data management order, FAA's Office of Aviation 
Safety has established a data management framework that includes a 
four-step process for importing data from other FAA offices and 
external sources. This process includes: 

* data acquisition--obtaining information from various data owners, 

* data standardization--validating data by comparing a new data set 
with previous data sets to identify inconsistencies, 

* data integration--translating data values into plain English and 
correcting data errors, and: 

* data loading--importing data into the agency's own systems. 

This four-step process applies to 10 of the 13 databases we reviewed-- 
8 maintained by FAA offices and 2 maintained by NTSB and USDA. (The 
process applies to 2 of the 4 voluntary reporting databases.) NTSB and 
USDA also have data quality assurance processes that apply to their 
databases. For example, NTSB conducts annual reviews of aircraft 
accident data, and according to USDA, airport managers and wildlife 
biologists are asked to check data from their respective airports and 
report errors. 

In addition, to help ensure the quality of its data, FAA applies 
various quality controls, such as validation and verification 
processes, to better ensure accuracy and completeness. We have 
identified some standard quality controls that an agency should employ 
to achieve these high-quality results.[Footnote 36] For example, 
agencies should have managers review data, have procedures in place to 
verify that data are complete and accurate, and correct erroneous 
data. We assessed 12 databases against these standard quality controls 
and found that the extent to which such controls were applied varied. 
[Footnote 37] (See figure 3.) While NTSB's aviation accident and 
incident database (NTSB), VDRP, and USDA's wildlife strike database 
applied all five of the quality control activities we considered for 
this analysis to some extent, the remaining 9 databases lacked one or 
more quality control activity. In addition, we found that all of the 
databases we reviewed fully or to some extent had procedures in place 
to validate and edit data to help ensure that accurate data are 
entered into electronic systems and to help ensure that erroneous data 
are identified, reported, and corrected. To the extent that the 
databases lack various controls, FAA lacks assurance that the 
information it uses for oversight is accurate and complete. 

Figure 3: Extent to Which Aviation Safety Databases Reviewed Have 
Standard Data Quality Controls: 

[Refer to PDF for image: illustrated table] 

Quality characteristic: Managers review data before they are entered 
into the data system; 
Databases: 
AIDS: Not present; 
ASRS: Fully present; 
FOQA: Not present; 
ATOS: Not present; 
NMACS: Not present; 
NTSB: Present to some extent; 
OEDS: Not present; 
PDS: Present to some extent; 
SDRS: Present to some extent; 
VDRP: Present to some extent; 
VPDS: Present to some extent; 
Wildlife: Fully present. 

Quality characteristic: Reconciliations are performed to verify the 
data’s completeness.[A] 
Databases: 
AIDS: Fully present; 
ASRS: Not present; 
FOQA: Fully present; 
ATOS: Present to some extent; 
NMACS: Present to some extent; 
NTSB: Fully present; 
OEDS: Present to some extent; 
PDS: Not present; 
SDRS: Present to some extent; 
VDRP: Present to some extent; 
VPDS: Present to some extent; 
Wildlife: Fully present. 

Quality characteristic: Data entry processes are designed to enhance 
accuracy; 
Databases: 
AIDS: Not present; 
ASRS: Present to some extent; 
FOQA: Fully present; 
ATOS: Not present; 
NMACS: Not present; 
NTSB: Fully present; 
OEDS: Not present; 
PDS: Not present; 
SDRS: Not present; 
VDRP: Present to some extent; 
VPDS: Not present; 
Wildlife: Fully present. 

Quality characteristic: Procedures are in place to validate and edit 
data to help ensure that accurate data are entered into electronic 
systems; 
Databases: 
AIDS: Present to some extent; 
ASRS: Present to some extent; 
FOQA: Fully present; 
ATOS: Present to some extent; 
NMACS: Present to some extent; 
NTSB: Fully present; 
OEDS: Present to some extent; 
PDS: Present to some extent; 
SDRS: Fully present; 
VDRP: Present to some extent; 
VPDS: Present to some extent; 
Wildlife: Present to some extent. 

Quality characteristic: Procedures are in place to help ensure that 
erroneous data are identified, reported, and corrected; 
Databases: 
AIDS: Fully present; 
ASRS: Fully present; 
FOQA: Present to some extent; 
ATOS: Present to some extent; 
NMACS: Present to some extent; 
NTSB: Fully present; 
OEDS: Present to some extent; 
PDS: Present to some extent; 
SDRS: Present to some extent; 
VDRP: Present to some extent; 
VPDS: Fully present; 
Wildlife: Present to some extent. 

Source: GAO. 

Note: We did not assess ASAP controls because ASAP contains 
proprietary airline data that were not available for review. 

[A] "Data completeness" refers to the accuracy with which data entered 
into a database have been compiled or processed. It does not refer to 
the scope of the data that have been reported or collected. 

[End of figure] 

While the databases we reviewed varied in the extent to which they had 
standard quality controls, FAA has other data quality controls in 
place for some databases that we consider good practices for handling 
data, as shown in table 3.[Footnote 38] 

Table 3: Examples of Additional Controls for Managing Data Quality: 

Data quality controls[A]: Event represented in a database is 
objectively defined so it can be replicated by a third party; 
Examples of databases with data quality controls implemented: AIDS, 
FOQA, NTSB. 

Data quality controls[A]: Data are validated to ensure that they 
represent what was intended; 
Examples of databases with data quality controls implemented: AIDS, 
FOQA, NTSB. 

Data quality controls[A]: Data systems training is available for users 
entering data into computer systems; 
Examples of databases with data quality controls implemented: AIDS, 
ATOS, NTSB. 

Data quality controls[A]: Data users receive periodic updates about 
the data in electronic systems; 
Examples of databases with data quality controls implemented: AIDS, 
ASRS, NMACS, NTSB. 

Data quality controls[A]: Access to the database is restricted to 
personnel who are authorized to enter or edit the data; 
Examples of databases with data quality controls implemented: ATOS, 
OEDS, VDRP. 

Data quality controls[A]: Electronic data system tracks and maintains 
changes to the database; 
Examples of databases with data quality controls implemented: AIDS, 
ASRS, ATOS. 

Data quality controls[A]: Data are verified after all electronic 
operations; 
Examples of databases with data quality controls implemented: All 
databases. 

Source: GAO. 

[A] These data quality controls are found in GAO-09-680G. 

[End of table] 

Furthermore, other data quality controls apply to the voluntary 
reporting systems we reviewed. For example, as previously discussed, 
an event review committee at each participating carrier is tasked with 
reviewing and analyzing reports submitted under ASAP. This committee 
determines whether such reports qualify for inclusion in the program, 
identifies and proposes solutions for actual or potential problems 
with information contained in the reports, and annually reviews the 
ASAP database to determine whether corrective actions have been 
effective in preventing or reducing the recurrence of targeted safety-
related events. 

For ASRS, NASA officials told us that each individual ASRS report is 
reviewed by two expert analysts within 3 days of receipt. Each report 
captures data for seven criteria and data fields, which are screened 
to ensure accuracy. The analysts also evaluate the database to ensure 
that publicly released ASRS data do not include information that might 
identify the reporter. In addition, for FOQA, vendors have quality 
assurance procedures in place. For instance, one vendor's procedures 
include automated checks and tests of each flight data file to detect 
parameter problems (for example, anomalies in the flight data 
attributable to faulty sensors), reports of anomalies created for the 
affected airline, and manual reviews of any data indicating warning- 
level risk events. These controls are designed to monitor the 
reliability and validity of FOQA data and to identify technical 
problems that affect data quality and need to be corrected. However, 
according to the vendor, some data will always be missing because of 
data-recording equipment failures or lost flight data cards, but such 
missing data do not affect the statistical validity of the large FOQA 
data set. 

FAA Is Taking Steps to Address Data Quality Weaknesses Identified by 
Its Analysts and Others: 

FAA is taking steps to address data weaknesses identified by its 
analysts, the DOT IG, and us. For example, FAA officials told us that 
to mitigate problems from external data sources, they combine data 
from various sources to validate analysis results. According to FAA 
analysts, they typically combine AIDS data with wildlife strike, ASRS, 
or PDS data using a manual process to verify study findings. As 
another example, we previously reported the importance of aggregating 
data from multiple sources to understand icing-related incidents. We 
reported that the AIDS database included 200 icing-related incidents 
involving large commercial airplanes that occurred from 1998 through 
2007.[Footnote 39] During this same time period, ASRS received over 
600 icing-and winter weather-related incident reports involving large 
commercial airplanes. These incidents revealed a variety of safety 
issues such as runways contaminated by snow or ice, ground deicing 
problems, and in-flight icing encounters. This suggested that risks 
from icing and other winter weather operating conditions may be 
greater than indicated by the AIDS database. 

FAA analysts also communicate with other data providers and experts 
about quality concerns or sometimes make independent corrections to 
data. For example, one FAA official told us that analysts communicate 
with NTSB to report incorrect data in a field and then rely on NTSB to 
correct the data in its database. FAA analysts said that they retrieve 
data from public Web sites and then collaborate with subject matter 
experts to identify and correct any errors in those data. They make 
such corrections based on their knowledge of the data's reliability 
and their own expertise in working with such data. Analysts also said 
they use information from the narrative sections of a report to 
correct data fields. 

In addition, the IG has identified weaknesses in the quality of 
specific data, which FAA is working to address. For instance, 
according to the IG, ATOS data are inconsistent and incomplete because 
the database has undergone multiple revisions since it was introduced 
in 1998 and some data fields have changed from one year to another. 
During these revisions, some data have been lost. Though designed to 
improve ATOS's value and usability as an inspection tool, the 
revisions limit opportunities for analysis of long-term trends to the 
extent that data fields have changed over time. The revisions do not, 
however, affect FAA's ability to analyze the data at a particular 
point in time.[Footnote 40] In addition, the process for reporting 
inspection findings is time-consuming and creates an incentive for 
inspectors to underreport their inspection results. To report, 
inspectors must complete a Yes/No checklist and, for every No check, 
provide a narrative explanation. According to the 2008 independent 
review team, inspectors have an incentive to check Yes so they can 
complete their reports in a timely manner. Consequently, the system 
may underreport problems that inspectors have identified but not taken 
the time to report. 

The IG has also found, and FAA agreed, that OEDS has some missing and 
incorrect data on operational errors and pilot deviations because 
personnel have intentionally or unintentionally misclassified these 
events. Such misclassification is problematic because it can lead to 
errors in FAA's assessment and reporting of how well the agency is 
meeting its annual performance targets for operational errors and 
pilot deviations. In 2007, the IG investigated operational errors at 
the Dallas-Fort Worth terminal radar approach control facility and 
found that FAA air traffic managers had intentionally misclassified 
operational errors as either pilot deviations or nonoccurrences. 
[Footnote 41] On the basis of this finding, FAA agreed with the IG's 
recommendation that the agency establish a follow-up mechanism to 
ensure compliance with guidance for investigating pilot deviations. 

Finally, over the years, we have identified weaknesses in the quality 
of aviation safety data that hinder FAA's ability to oversee the 
industry. In response, the agency has taken steps to address many of 
the problems that we have identified. For example, in our 2007 review 
of runway safety, we found that FAA's categorization of the severity 
of runway incursions involves a level of subjectivity, raising 
questions about the accuracy of the data.[Footnote 42] We reported 
that an internal FAA audit of 2006 runway incursion data found that 
the subjectivity of the severity classifications has the potential to 
affect the accuracy of the classifications. We also found that FAA did 
not systematically collect data on the number of runway overruns that 
do not result in damage or injury that could be used for analytical 
purposes to study trends in and causes of these incidents. In July 
2009, FAA indicated that it was working to establish procedures that 
will ensure that all runway overruns and other excursions are reported. 

Conclusions: 

Aviation safety data are critical to FAA's safety oversight and its 
planned implementation of SMS. To its credit, FAA has taken steps to 
help ensure the quality of the data it uses, such as implementing 
quality controls to help ensure that errors are identified, reported, 
and corrected, but these procedures are not applied consistently 
across all databases. Although FAA is developing a plan that will 
address how data fit into its new oversight approach, that plan lacks 
a description of the data that will be required to conduct proactive 
data analyses, an inventory of the skills personnel will need to 
perform such analyses and help ensure data quality, and a description 
of the steps needed to address continuing data quality problems. 
Unless the plan links FAA's data requirements and staffing needs to 
the analyses that will drive its proactive safety management system 
and addresses the agency's data quality problems, available data may 
not be as reliable and useful as they could be to support SMS. 

While NextGen technologies and procedures are intended to increase the 
safety, efficiency, and capacity of the national airspace system, 
their introduction could have unintended effects on system safety if 
not done in a comprehensive manner. As FAA improves its ability to 
integrate and analyze data from multiple sources, it plans to increase 
its capacity to model the impact of NextGen changes and to identify 
and manage risks. Because some NextGen changes are already taking 
place, it is urgent that FAA move with all deliberate speed to advance 
its analytical capability. 

The data that FAA obtains through voluntary reporting programs afford 
insights into safety events that are not available from other sources 
and are critical to improving aviation safety, but participation in 
these programs has been limited by concerns about the impact of 
disclosure and, especially in the case of smaller carriers, by cost 
considerations. Efforts such as the training FAA provides to smaller 
carriers on how to develop programs that require less capital 
investment have the potential to increase participation and improve 
safety. However, without carrier-specific information on why air 
carriers are not participating in these programs, FAA cannot determine 
if its efforts to increase participation are sufficient. 

Recommendations for Executive Action: 

To help improve and expand FAA's capability to use data for aviation 
safety oversight, we recommend that the Secretary of Transportation 
direct the FAA Administrator to take the following four actions: 

* develop and implement a comprehensive plan that addresses how data 
fit into FAA's implementation of a proactive approach to safety 
oversight and ensure that this plan fully describes the relevant data 
challenges (such as ensuring data quality and continued access to 
voluntarily reported safety data), analytical approaches, and staffing 
requirements and integrates efforts to address them; 

* given the importance of high-quality data, extend standard quality 
controls, as appropriate, to the databases that support aviation 
safety oversight to ensure that the data are as reliable and valid as 
possible; 

* proceed with all deliberate speed to develop the capability to model 
the impact of NextGen changes on the national airspace system and 
manage any risks emerging from these changes; and: 

* systematically identify the reasons that carriers are not 
participating in voluntary reporting programs, such as through a 
survey, and identify and implement further steps to encourage greater 
program participation, especially by smaller carriers. 

Agency Comments: 

We provided copies of a draft of this report to DOT, NASA, USDA, and 
NTSB for their review and comment. DOT agreed to consider our 
recommendations. DOT and NASA provided technical corrections and 
clarifications, which we incorporated as appropriate. USDA had no 
comments. NTSB generally agreed with our findings and recommendations 
to FAA and provided several comments. First, NTSB noted that our use 
of the terms "reactive" and "proactive" implied a new approach to 
aviation safety data analysis that is different from past analyses of 
accidents and incidents to improve safety. The agency noted that a 
more efficient, effective approach to safety analysis should continue 
to include FAA's previous reactive approach as well as new, more 
predictive capabilities. We agree with NTSB's comment and note that 
our report indicates that FAA plans to continue to use data to analyze 
past safety events as it also works to use data more proactively. NTSB 
further noted that the success of SMS will depend on the maturation of 
FAA's data analysis capabilities. 

Second, NTSB agreed with our finding that the lack of a final plan for 
ASIAS and for SMS implementation, which are key elements of FAA's 
planned proactive safety analysis capability, was a cause for concern. 
The agency noted that it had made several recommendations to FAA to 
require SMS programs for part 121, part 135, and part 91 carriers and 
that FAA had not yet taken action to require these programs. 

Third, regarding FAA's access to voluntarily reported data, NTSB 
agreed with our finding that the redaction of flight details from ASAP 
and FOQA analyses is a serious constraint on the thoroughness and 
potential utility of ASIAS and other assessments of safety. If FAA 
does not address these data limitations, NTSB observed, such 
constraints are likely to pose serious and continuing threats to the 
broader use of voluntary reporting programs to support safety 
analysis. In NTSB's view, our recommendations to FAA do not go far 
enough to recommend mechanisms besides redaction, such as statutory 
exemptions from disclosure, to protect these data from enforcement and 
disciplinary uses or public release. We did not revise our 
recommendations to FAA to include these issues because, while we found 
that participation was temporarily affected, in part, by employees' 
fears of disciplinary action by their employers, we did not find 
evidence that participation was inhibited by the fear of enforcement 
action by FAA or public disclosure. In addition, our work indicated 
that the current mechanisms to protect the data appeared to be working. 

Fourth, regarding FAA's access to data on various safety events, NTSB 
noted the importance of FAA collecting the necessary data to support 
its new approaches to data analysis rather than simply combining 
existing data sources into an analysis program. NTSB also agreed with 
our finding that independent survey efforts like NAOMS could provide a 
useful complement to other data sources, including FOQA, in providing 
improved data quality and analysis capabilities. 

Finally, NTSB agreed with our finding that the availability of 
operations data for sectors other than large commercial carriers 
(i.e., part 121 operators) is severely limited. The agency noted that 
accurate flight activity data are not available for most of these 
operations and must be estimated from FAA's annual survey of a sample 
of active general aviation aircraft. NTSB also pointed out that FAA 
does not require reporting for the majority of equipment reliability 
or maintenance related events. To address these shortcomings, NTSB 
noted its recent recommendation to FAA to take steps to increase 
general aviation reporting to FAA's service difficulty reporting 
system. To correct these and other data deficiencies, NTSB believes 
that FAA should explore the development of new aviation safety data 
collection techniques or methods to supplement current areas of data 
deficiency. 

As arranged with your offices, unless you publicly announce its 
contents earlier, we plan no further distribution of this report until 
30 days after the date of this letter. At that time, we will send 
copies of this report to relevant congressional committees, the 
Secretaries of Transportation and Agriculture, the Administrator of 
the Federal Aviation Administration, the Chairman of the National 
Transportation Safety Board, and the Administrator of the National 
Aeronautics and Space Administration. We will also make copies 
available to others on request. In addition, this report will be made 
available at no charge on the GAO Web site at [hyperlink, 
http://www.gao.gov]. 

If you or your staff have any questions about this report, please 
contact me at (202) 512-2834 or dillinghamg@gao.gov. Contact points 
for our Offices of Congressional Relations and Public Affairs may be 
found on the last page of this report. GAO staff who made major 
contributions to this report are listed in appendix II. 

Signed by: 

Gerald L. Dillingham, Ph.D. 
Director, Physical Infrastructure Issues: 

List of Requesters: 

The Honorable Bart Gordon:
Chairman:
Committee on Science and Technology:
House of Representatives: 

The Honorable Gabrielle Giffords:
Chair:
Subcommittee on Space and Aeronautics:
Committee on Science and Technology:
House of Representatives: 

The Honorable Brad Miller:
Chairman:
Subcommittee on Investigations and Oversight:
Committee on Science and Technology:
House of Representatives: 

The Honorable Mark Udall:
United States Senate: 

The Honorable Jerry Costello:
House of Representatives: 

The Honorable Daniel Lipinski:
House of Representatives: 

[End of section] 

Appendix I: Objectives, Scope, and Methodology: 

In this report, we assessed the Federal Aviation Administration's 
(FAA) capacity to use available data to oversee aviation safety. To do 
so, we addressed the following questions: (1) How does FAA use data to 
oversee aviation safety, and what changes, if any, has it planned? (2) 
To what extent does FAA have access to data for monitoring aviation 
safety and the safety of various aviation industry sectors? (3) What 
does FAA do to help ensure the quality of the data it uses to oversee 
aviation safety? 

To perform our review, we selected 10 safety events that were among 
those previously identified as key by the National Aeronautics and 
Space Administration's (NASA) National Aviation Operations Monitoring 
Service (NAOMS) or by the FAA-industry Commercial Aviation Safety Team 
(CAST). This selection allowed us to focus our review on a manageable 
subset of FAA oversight activities and data sources. (See table 4.) We 
then identified 13 databases available to FAA that contained data on 
these safety events and reviewed these databases. The databases are 
described in the background section of this report. 

Table 4: Aviation Safety Events Included in Our Review: 

Type of safety event: Equipment problems; 
Definition: Any problem related to aircraft equipment; 
Examples of safety events: Problems with computer navigation system, 
low fuel levels, engine failures, dirty or shattered windshields. 

Type of safety event: Turbulence; 
Definition: Encounters with turbulence due to wake or weather; 
Examples of safety events: Injuries to passengers and crew due to 
turbulence. 

Type of safety event: Weather events while airborne; 
Definition: Problems due to weather-related factors other than 
turbulence; 
Examples of safety events: Strong winds, thunderstorms, icing, wind 
shear. 

Type of safety event: Passenger-related events; 
Definition: Any passenger-related problems; 
Examples of safety events: Passengers walking around while seat belt 
light is on, engaging in drunken or disruptive conduct, smoking on 
board, and disabling smoke detectors. 

Type of safety event: Flight crew problems; 
Definition: Flight crew performance issues other than those resulting 
in airborne conflicts, ground events, or altitude deviations; 
Examples of safety events: Pilot distraction, shortage of crew 
members, poor pilot judgment, insufficient pilot proficiency, pilot 
fatigue. 

Type of safety event: Airborne conflicts; 
Definition: Any conflicts with other aircraft in the air; 
Examples of safety events: Congestion, insufficient spacing behind 
aircraft, pilots unaware of warnings. 

Type of safety event: Ground events; 
Definition: Runway and taxiway transgressions, ground conflicts, and 
other ground-based events; 
Examples of safety events: Congestion on taxiways or ramps, potential 
for planes to strike vehicles or equipment in areas where aircraft 
have the right of way. 

Type of safety event: Aircraft handling-related events; 
Definition: Deviations in aircraft direction, speed, and configuration 
due to pilot error; 
Examples of safety events: Near collisions with terrain, deviations 
from assigned routes, hard landings. 

Type of safety event: Altitude deviations; 
Definition: Any deviation from assigned altitude; 
Examples of safety events: Altitude overshoots, descents below minimum 
safe altitude. 

Type of safety event: Adverse interactions with air traffic control; 
Definition: Events rooted in difficulties between pilot and air 
traffic control; 
Examples of safety events: Improper use of phraseology resulting in 
miscommunication, blocked transmissions due to problems with 
frequency, language barriers when English is not the primary language. 

Source: NAOMS and CAST. 

[End of table] 

To determine how FAA uses data to oversee aviation safety, we reviewed 
reports by FAA, the International Civil Aviation Organization, and 
industry and other published documents. In addition, we interviewed 
FAA, industry associations, and other selected industry groups. 

To determine the extent to which FAA has access to data for monitoring 
aviation safety and the safety performance of various aviation 
industry sectors, we interviewed FAA data analysts, contractors, and 
other officials responsible for data management. We also reviewed 
previous GAO reports on FAA's access to data on certain aviation 
sectors, such as air ambulances and air cargo operations. 

To determine how FAA ensures the quality of its data, we interviewed 
FAA and industry officials (see table 5). In addition, we reviewed 
assessments of selected FAA data by the Department of Transportation 
Inspector General and an independent review team appointed by the 
Secretary of Transportation in 2008. We also derived a number of data 
quality principles from our previous work on internal controls, 
[Footnote 43] and we assessed the quality of 12 of the 13 selected 
aviation safety databases by comparing our data quality principles 
with FAA's practices for ensuring data quality.[Footnote 44] These 
principles include ensuring that the data are complete and accurate, 
measure intended safety concerns, and are useful for their intended 
oversight purposes. To measure the extent to which FAA's practices 
were consistent with these principles, we evaluated information and 
all other materials regarding the databases using a three-point scale. 
To validate the results, multiple reviewers independently scored each 
principle. When the initial scores differed, the reviewers 
collectively agreed on a final score for each principle. Further, we 
used the results of GAO studies that considered the availability, 
quality, and use of data in aviation safety oversight. 

Table 5: Organizations That GAO Contacted: 

Type of organization: FAA office; 
Organization interviewed: 
* Air Traffic Organization; 
* Airports; 
* Aviation Safety, Office of Accident Investigation and Prevention 
(formerly the Office of Analytical Safety Services); 
* Aviation Safety, Air Traffic Oversight Service; 
* Aviation Safety, Flight Standards. 

Type of organization: Other government agency; 
Organization interviewed: 
U.S.: 
* U.S. Department of Agriculture, Animal and Plant Health Inspection 
Service, Wildlife Services; 
* U.S. Department of Transportation, Office of Inspector General; 
* NASA; 
* National Transportation Safety Board; 
International: 
* International Civil Aviation Organization. 

Type of organization: Industry organization; 
Organization interviewed: 
Associations and research institutes: 
* Flight Safety Foundation; 
* MITRE; 
Regional Airline Association; 
U.S. airlines: 
* American Airlines; 
* Comair; 
* Continental Airlines; 
* Delta/Northwest Airlines; 
* Southwest Airlines; 
* US Airways; 
Unions: 
* Air Line Pilots Association, International; 
* Professional Aviation Safety Specialists, AFL-CIO; 
* U.S. Airline Pilots Association; 
Data vendors: 
* Austin Digital; 
* Sagem Avionics, Inc. 

Source: GAO. 

[End of table] 

In addition, to address all three research questions, we individually 
interviewed 10 aviation safety experts and asked them to identify 
challenges to using data for overseeing aviation safety, the 
reasonableness of FAA's current and planned efforts to use aviation 
safety data, and ways that FAA could enhance its data collection and 
analysis processes to improve its oversight capabilities. We selected 
experts who represent a cross section of aviation stakeholders, 
including persons with general knowledge of aviation safety, aircraft 
operations, human factors, aircraft maintenance, and air traffic 
control. The experts have operational, academic, or other professional 
expertise in these areas. Those experts are: 

* Mr. Basil Barimo, Vice President, Safety and Operations, Air 
Transport Association; 

* Mr. James Burin, Director of Technical Plans and Programs, Flight 
Safety Foundation; 

* Kim Cardosi, Ph.D., U.S. Department of Transportation, Volpe Center; 

* Todd Curtis, Ph.D., Director, The Airsafe.com Foundation; 

* Mr. John Goglia, Senior Vice President for Aviation Operations and 
Safety Programs, JDA Aviation Technology Solutions, former board 
member of the National Transportation Safety Board; 

* Mr. Keith Hagy, Director, Engineering and Air Safety, Air Line 
Pilots Association, International; 

* Brigadier General Leon Johnson (Air Force, retired), former Flight 
Operations Manager, United Parcel Service; 

* Mr. Bruce Landsberg, Executive Director, Aircraft Owners and Pilots 
Association; 

* Thomas Weitzel, Ed.D., Associate Professor, Embry Riddle 
Aeronautical University; and: 

* Mr. Dale Wright, Director, Safety and Technology, National Air 
Traffic Controllers Association. 

[End of section] 

Appendix II: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Gerald L. Dillingham, Ph.D., (202) 512-2834, or dillinghamg@gao.gov: 

Staff Acknowledgments: 

In addition to the person named above, Teresa Spisak, Assistant 
Director; Elizabeth Eisenstadt; N'Kenge Gibson; H. Brandon Haller; 
Erica Miles; and Richard Scott made key contributions to this report. 

[End of section] 

Footnotes: 

[1] In this report, we define "oversight" broadly as FAA's activities 
to monitor aviation safety through the use of data. By comparison, FAA 
uses the term "oversight" only to refer to its determination of 
regulatory compliance and "monitoring" to refer to FAA's voluntary 
safety programs (defined later in this report), which monitor aviation 
safety through the use of voluntarily supplied data that provides 
insight into the total flight operations environment. 

[2] CAST selects and analyzes past aviation safety events to identify 
their precipitating conditions and causes and formulates an 
intervention strategy to reduce the likelihood of a recurrence. 

[3] In assessing NASA's project, we determined that the safety events 
used in the survey were developed based on a solid foundation of 
available data, literature, and information from aviation 
stakeholders. See GAO, Aviation Safety: NASA's National Aviation 
Operations Monitoring Service Project Was Designed Appropriately, but 
Sampling and Other Issues Complicate Data Analysis, [hyperlink, 
http://www.gao.gov/products/GAO-09-112] (Washington, D.C.: Mar. 13, 
2009). 

[4] GAO, Internal Control Management and Evaluation Tool, [hyperlink, 
http://www.gao.gov/products/GAO-01-1008G] (Washington, D.C.: Aug. 1, 
2001). 

[5] For example, FAA regulates both U.S. commercial passenger and air 
cargo carriers under parts 121 and 135 of Title 14 of the Code of 
Federal Regulations. Briefly stated, part 121 applies to air carrier 
operations involving airplanes with a seating capacity of 10 or more 
passengers or a maximum payload capacity of more than 7,500 pounds. 
Part 135 governs small aircraft that have a seating capacity of fewer 
than 10 passengers or a payload of less than 7,500 pounds. The agency 
regulates general aviation under part 91 and air ambulances under part 
135 (when patients are on board) and part 91 (when patients are not 
present). 

[6] FAA agrees not to take enforcement actions against self-reported 
violations of regulations to encourage reporting. The types of 
violations not covered under the programs are described later in this 
report. 

[7] We did not include ATSAP in our analysis of voluntarily reported 
data systems because of the recentness of this program. However, 
according to an FAA analysis, in 3,860 (or 38 percent) of more than 
12,000 reports, air traffic control personnel have provided 
information that they believe could introduce risk into the system, 
thereby facilitating proactive containment of the hazard in question. 

[8] Controlled flight into terrain occurs when an airworthy aircraft 
under the control of the pilot is inadvertently flown into terrain, 
water, or an obstacle. The pilot is generally unaware of the danger 
until it is too late. 

[9] Light sport aircraft have, among other criteria, a maximum gross 
takeoff weight of not more than 1,320 pounds, a maximum airspeed of 
140 miles per hour, and seating for only one passenger besides the 
pilot. 

[10] Independent Review Team, Managing Risks in Civil Aviation: A 
Review of the FAA's Approach to Safety (Washington, D.C.: Sept. 2, 
2008). The team was chartered by the Secretary of Transportation to 
assess FAA's safety culture and approach to safety management. 

[11] The Volpe Center, which is part of the Department of 
Transportation's Research and Innovative Technology Administration, is 
a federal fee-for-service organization that focuses on crosscutting 
transportation, research, education, innovation, and other multimodal 
issues. 

[12] In this report, SMS refers only to FAA's internal implementation 
of SMS as it applies to the Office of Aviation Safety, the Air Traffic 
Organization, and the Office of Airports. FAA intends that the 
aviation industry will also implement SMS. 

[13] This guidance is modeled on ICAO's international framework and 
standards for safety management. 

[14] According to an NTSB official, in addition to narrative text, 
NTSB's Aviation Accident and Incident Database includes approximately 
200 coded fields that record aircraft, engine, and flight crew 
details. The database also includes coded fields identifying other 
accident information such as phase of flight, events, and findings of 
probable cause. In addition, according to NASA, ASRS data have 1,200 
potentially coded fields. 

[15] GAO, Aviation Safety: Better Management Controls Are Needed to 
Improve FAA's Safety Enforcement and Compliance Efforts, [hyperlink, 
http://www.gao.gov/products/GAO-04-646] (Washington, D.C.: July 6, 
2004). 

[16] Independent Review Team, Managing Risks in Civil Aviation: A 
Review of the FAA's Approach to Safety. 

[17] DOT Office of Inspector General, FAA is not Realizing the Full 
Benefits of the Aviation Safety Action Program, Report No. AV-2009-057 
(Washington, D.C.: May 14, 2009). 

[18] In addition to the ASAP reports, ASRS receives reports from eight 
other airlines that do not participate in ASAP, including corporate, 
emergency medical services, and other air operators, according to NASA. 

[19] NASA maintains ASRS on behalf of FAA because it is knowledgeable 
about aviation safety issues, is not a regulator, and has a reputation 
for protecting reporters' anonymity. 

[20] FAA, Aviation Safety Analytical Services, System Safety 
Management Transformation Program Plan, version 1.0 (Washington, D.C.: 
Dec. 24, 2009). 

[21] GAO, Federal Aviation Administration: Human Capital System 
Incorporates Many Leading Practices, but Improving Employees' 
Satisfaction with Their Workplace Remains a Challenge, [hyperlink, 
http://www.gao.gov/products/GAO-10-89] (Washington, D.C.: Oct. 28, 
2009). 

[22] NTSB, Safety Recommendation Letter to FAA, and A-07-1 through 11 
(Washington, D.C.: Jan. 23, 2007). 

[23] A stable approach typically occurs when an aircraft approaches a 
runway with a glide path that allows it to descend at a constant rate 
of 300 feet per nautical mile traveled forward. An unstable approach 
occurs when an aircraft is approaching a runway too fast, too high, or 
with insufficient engine power, causing it to deviate from the glide 
path. 

[24] NTSB, Aircraft Accident Report: Loss of Control on Approach, 
Colgan Air, Inc. Operating as Continental Connection Flight 3407 
Bombardier DHC-8-400, N200WQ Clarence Center, New York, February 12, 
2009, NTSB/AAR-10/01 (Washington, D.C.: Feb. 2, 2010). 

[25] DOT Office of Inspector General, FAA is not Realizing the Full 
Benefits of the Aviation Safety Action Program. 

[26] [hyperlink, http://www.gao.gov/products/GAO-09-112]. 

[27] Proposed legislation, H.R. 1586, §554, 111th Cong. (2009) (Senate 
version of FAA Reauthorization), offers additional protections for 
FOQA and ASAP data. Additionally, H.R. 1586, §353, 111th Cong. (2009) 
(House version of FAA Reauthorization), would require FAA to report on 
the status of voluntary programs, including which carriers are 
participating and why others do not have voluntary programs. (In 
consideration of FAA Reauthorization, the Senate adopted the House 
bill, H.R. 1586, and amended it inserting its own language. 
Differences between the House version of H.R. 1586 and the Senate 
version of H.R. 1586 will have to be reconciled in a conference 
committee.) 

[28] We defined "large carriers" as the top 25 U.S. carriers with 
5,000,000 or more enplaned passengers from July 2008 through June 
2009. We defined "small carriers" as those with fewer than 5,000,000 
passengers for the same period. 

[29] Concerns that litigants could require them under the Freedom of 
Information Act to disclose the information to the public may also 
have contributed to the carriers' and unions' decisions to withdraw 
from ASAP. These concerns were heightened after court rulings called 
for the release of Comair ASAP reports in the aftermath of the fatal 
crash of a Comair jet in Lexington, Kentucky, in 2006. 

[30] H.R. 1586, §354, 111TH Cong. (2009) (House version of FAA 
Reauthorization), if enacted, would require FAA to develop and 
implement a plan that would facilitate the establishment of ASAP and 
FOQA programs by all part 121 air carriers. Furthermore, smaller 
fleets would be assisted, if necessary, in establishing FOQA programs. 

[31] GAO, Aviation Safety: Potential Strategies to Address Air 
Ambulance Safety Concerns, [hyperlink, 
http://www.gao.gov/products/GAO-09-627T] (Washington, D.C.: Apr. 22, 
2009). 

[32] GAO, Aviation Safety: Improved Data Collection Needed for 
Effective Oversight of Air Ambulance Industry, [hyperlink, 
http://www.gao.gov/products/GAO-07-353] (Washington, D.C.: Feb. 21, 
2007). 

[33] GAO, Aviation Safety: Better Data and Targeted FAA Efforts Needed 
to Identify and Address Safety Issues of Small Air Cargo Carriers, 
[hyperlink, http://www.gao.gov/products/GAO-09-614] (Washington, D.C.: 
June 24, 2009). 

[34] In GAO, Assessing the Reliability of Computer-Processed Data, 
[hyperlink, http://www.gao.gov/products/GAO-09-680G], (Washington, 
D.C.: Feb.2, 2009), we defined reliable data as data that are 
reasonably complete and accurate, meet intended purposes, and are not 
subject to inappropriate alteration. "Completeness" refers to the 
extent to which relevant records are present and the fields in each 
record are populated appropriately. "Accuracy" refers to the extent to 
which recorded data reflect the actual underlying information. These 
terms define data reliability. "Validity" refers to whether the data 
actually represent what they are intended to measure. 

[35] FAA Order 1375.1D, Information/Data Management, July 25, 2006. 

[36] [hyperlink, http://www.gao.gov/products/GAO-01-1008G]. 

[37] We did not assess controls for ASAP because it contains 
proprietary airline data that were not available for our review. 

[38] These and other data quality controls, which we use in assessing 
the reliability of data for our analyses, are found in GAO-09-680G. 

[39] GAO, Aviation Safety: Preliminary Information on Aircraft Icing 
and Winter Operations, [hyperlink, 
http://www.gao.gov/products/GAO-10-441T] (Washington, D.C.: Feb. 24, 
2010). 

[40] H.R. 1586, §520, 111TH Cong. (2009) (Senate version of FAA 
Reauthorization), if enacted, would require FAA to conduct monthly 
reviews of ATOS data and submit quarterly reports to Congress on the 
results of these reviews. Those reports would allow for analyses at a 
particular point in time and, if the same data were reported 
consistently over time, would eventually permit trend analyses. 

[41] DOT Office of Inspector General, FAA's Process for Reporting and 
Investigating Operational Errors, Report No. AV-2009-045 (Washington, 
D.C.: Mar. 24, 2009). 

[42] GAO, Aviation Runway and Ramp Safety: Sustained Efforts to 
Address Leadership, Technology, and Other Challenges Needed to Reduce 
Accidents and Incidents, [hyperlink, 
http://www.gao.gov/products/GAO-08-29] (Washington, D.C.: Nov. 20, 
2007). 

[43] [hyperlink, http://www.gao.gov/products/GAO-01-1008G]. 

[44] We did not assess controls for the Aviation Safety Action Program 
because it contains proprietary airline data that were not available 
for our review. 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO’s Web site, 
[hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional 
information. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, dawnr@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: