Skip to main content

Aviation Forecasting: FAA Should Implement Additional Risk-Management Practices in Forecasting Aviation Activity

GAO-16-210 Published: Mar 08, 2016. Publicly Released: Mar 16, 2016.
Jump To:
Skip to Highlights

Highlights

What GAO Found

Both of the Federal Aviation Administration's (FAA) annual activity forecasts—the National Aerospace Forecast (Aerospace) and airport-level Terminal Area Forecast (TAF)—have consistently overestimated aviation activity since 2004 and 2010, respectively, and have been less accurate the further out they forecast. For example, for Aerospace passenger enplanement forecasts made between fiscal year 2004 and 2014, the mean percentage error was less than 1 percent for 1-year-ahead forecasts, 15 percent for 5-year-ahead forecasts, and 31 percent for 10-year-ahead forecasts (see table). An important factor affecting forecast accuracy was the inaccuracy of the inputs used in the TAF and Aerospace models—such as gross domestic product and fuel prices—resulting from events such as the 2007–2009 recession and fuel price spikes.

Accuracy of the Federal Aviation Administration's (FAA) 1-Year, 5-Year, and 10-Year National Aerospace Forecasts Made between Fiscal Years 2004 and 2014

 

Mean percentage error

Aviation activity metric

1-Year forecasta

5-Year forecastb

10-Year forecastc

Passenger enplanements (number of passengers boarding planes)

-0.2%

14.7%

31.3%

Revenue passenger miles

-0.6%

14.0%

26.9%

Total operationsd

1.9%

25.5%

54.7%

Available seat miles

0.1%

18.6%

40.5%

Source: GAO analysis of FAA data. | GAO-16-210

aincludes 11 1-year Aerospace forecasts made from fiscal year 2004 through 2014.
bincludes 7 5-year forecasts made from fiscal year 2004 through 2010.
cincludes 2 10-year forecasts made from fiscal year 2004 through 2005.
dincludes air carrier, air taxi, general aviation, and military flights arriving at and departing from airports

In developing forecasts, FAA has implemented most of the practices associated with five key risk-management principles GAO selected as applicable to forecasting: (1) setting measurable goals; (2) using best available data; (3) identifying, analyzing, and documenting risk; (4) adopting strategies to respond to risks; and (5) monitoring and reviewing performance. However, FAA lacks some risk-management practices that could enhance FAA's ability to manage for risk and uncertainty. For example, FAA monitors forecast performance by reevaluating the Aerospace and TAF forecast models, but has not fully identified, analyzed, and documented risk. While FAA has identified risks in the Aerospace forecast, it has not reported on the likelihood of these risks. For the TAF, FAA has not systematically assessed the TAF's uncertainty, partly due to recent changes in forecasting methodology. Nor has FAA established error thresholds that would trigger more thorough review of the forecast methodology. Given FAA's reliance on forecasts for decision-making, managing and understanding the nature of uncertainty is important to good decision-making. While FAA has provided limited information about the Aerospace and TAF models, it has not documented the models and assumptions in a manner that would allow stakeholders outside FAA to understand how forecasts are developed or enable FAA to retain organizational knowledge.

Why GAO Did This Study

The FAA annually prepares forecasts of future aviation activity and uses these forecasts to help manage most of its $15 billion in annual spending. While forecasting is inherently uncertain, managing that uncertainty is essential to informed decisions.

GAO was asked to examine the accuracy of and FAA's use of two annual forecasts of aviation activity. This report discusses the accuracy of FAA's forecasts from 2004 through 2014 and strengths and limitations of FAA's consideration of risks in developing its forecasts. This report focuses on the use of the Aerospace and TAF forecasts to inform key operational and investment decisions. GAO compared these two forecasts to actual activity from 2004 through 2014 for the Aerospace forecasts and from 2010 through 2014 for the TAF forecasts and identified factors affecting that accuracy. GAO compared FAA's treatment of risk in developing forecasts to selected risk- management practices recommended by the Office of Management and Budget, GAO, and others.




Recommendations

GAO recommends that the Secretary of the Department of Transportation (DOT) require the FAA to: (1) report on uncertainty and set error response thresholds for both forecasts and (2) document FAA's methods and assumptions for the forecasting models. The DOT partially concurs with the first recommendation and fully concurs with the second. DOT agrees to report on uncertainty but not to set thresholds. GAO believes that thresholds ensure systematic review of forecast accuracy.

Recommendations for Executive Action

Agency Affected Recommendation Status
Department of Transportation To help FAA better manage and understand the uncertainties of its forecasts, the Secretary of Transportation should direct the FAA to apply risk-management practices to analyze and report on uncertainty. Specifically, the FAA should, for both the Aerospace and TAF forecasts, analyze and report the forecast's uncertainty, establish forecast error thresholds, and develop an approach that will prompt forecast review when error thresholds are exceeded, and, for TAF forecasts, monitor and publish multi-year historical error performance, as FAA does for the Aerospace Forecast.
Closed – Implemented
The Federal Aviation Administration (FAA) annually prepares forecasts of future aviation activity, including projections of the number of flights and passengers, 20 of more years into the future. These forecasts are used by FAA to develop major areas of its operational plans and manage its budget and by external stakeholders, such as airlines and airports, for future planning and investment. The two main forecasts that FAA develops are the National Aerospace Forecast (Aerospace), which forecasts aviation activity at the national level, and the Terminal Area Forecast (TAF), which forecasts aviation activity at the individual airport level. While forecasting is inherently uncertain, managing that uncertainty is essential to making informed decisions. In 2016, GAO reported that it compared FAA's treatment of risk in creating aviation forecasts to select risk-management practices recommended by the Office of Management and Budget, GAO, and others, and found that FAA lacks some risk management practices that could enhance FAA's ability to manage for risk and uncertainty. FAA followed or partially followed most of the selected risk-management principles and practices in developing its aviation activity forecasts. However, FAA did not analyze the forecast uncertainty or establish forecast error thresholds for either the Aerospace or the TAF forecasts. Further, FAA did not monitor and publish multi-year historical error performance for the TAF forecast. By not taking these steps, FAA was limited in its ability to manage forecast uncertainty and users were limited in their understanding of the source and nature of forecast uncertainty. Therefore, GAO recommended that FAA should report on uncertainty and set error response thresholds for both forecasts. In 2020, FAA officials told us that they established error forecast thresholds for the Aerospace and TAF forecasts. In addition, FAA published information about forecast uncertainty in the Aerospace and TAF forecasts and published multi-year historical error performance for the TAF. These actions should improve FAA's ability to manage forecast uncertainty and increase users' understanding of the sources and nature of forecast uncertainty, which essentially meets the intent of GAO's recommendation.
Department of Transportation To help FAA better manage and understand the uncertainties of its forecasts, the Secretary of Transportation should direct the FAA to fully document its methods and assumptions in developing the Aerospace and TAF forecasting models to provide greater transparency to internal users and external stakeholders.
Closed – Implemented
The Federal Aviation Administration (FAA) annually prepares forecasts of future aviation activity, including projections of the number of flights and passengers, 20 of more years into the future. These forecasts are used by FAA to develop major areas of its operational plans and manage its budget and by external stakeholders, such as airlines and airports, for future planning and investment. The two main forecasts that FAA develops are the National Aerospace Forecast (Aerospace), which forecasts aviation activity at the national level, and the Terminal Area Forecast (TAF), which forecasts aviation activity at the individual airport level. While forecasting is inherently uncertain, managing that uncertainty is essential to making informed decisions. In 2016, GAO reported that it compared FAA's treatment of risk in creating aviation forecasts to select risk-management practices recommended by the Office of Management and Budget, GAO, and others, and found that FAA lacks some risk management practices that could enhance FAA's ability to manage for risk and uncertainty. While FAA had provided limited information about the Aerospace and the TAF models, it had not documented the models and the assumptions used when developing the models in a manner that would allow stakeholders outside of FAA to understand how the forecasts were made or replicate FAA's results. DOT's guidance favors sufficient transparency about methods, such as assumptions and supporting documentation, to allow independent reanalysis by qualified members of the public. Furthermore, without documentation, FAA would not be able retain organizational knowledge of the models and assumptions. Therefore, GAO recommended that FAA should fully document its methods and assumptions in developing the Aerospace and TAF models to provide greater transparency to internal users and external stakeholders. In 2020, FAA documented the methods and assumptions used in developing both the Aerospace and TAF forecasts on its website. Further, FAA officials told us that they are updating these documents and will issue these updates when the agency publishes its next annual forecasts in spring of 2021. These actions should (1) provide internal and external stakeholders' greater transparency and understanding of the methods and assumptions FAA uses when developing the forecasts, (2) ensure the continuity of forecasting within FAA, and (3) allow external stakeholders the ability to examine how forecasts are developed.

Full Report

Office of Public Affairs

Topics

AviationData integrityFuture budget projectionsInternal controlsPlanning programming budgetingProjectionsRisk adjustmentsRisk managementAviation forecastingAirports