Aviation Forecasting:

FAA Should Implement Additional Risk-Management Practices in Forecasting Aviation Activity

GAO-16-210: Published: Mar 8, 2016. Publicly Released: Mar 16, 2016.

Additional Materials:

Contact:

Gerald Dillingham, Ph.D.
(202) 512-2834
dillinghamg@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

What GAO Found

Both of the Federal Aviation Administration's (FAA) annual activity forecasts—the National Aerospace Forecast (Aerospace) and airport-level Terminal Area Forecast (TAF)—have consistently overestimated aviation activity since 2004 and 2010, respectively, and have been less accurate the further out they forecast. For example, for Aerospace passenger enplanement forecasts made between fiscal year 2004 and 2014, the mean percentage error was less than 1 percent for 1-year-ahead forecasts, 15 percent for 5-year-ahead forecasts, and 31 percent for 10-year-ahead forecasts (see table). An important factor affecting forecast accuracy was the inaccuracy of the inputs used in the TAF and Aerospace models—such as gross domestic product and fuel prices—resulting from events such as the 2007–2009 recession and fuel price spikes.

Accuracy of the Federal Aviation Administration's (FAA) 1-Year, 5-Year, and 10-Year National Aerospace Forecasts Made between Fiscal Years 2004 and 2014

 

Mean percentage error

Aviation activity metric

1-Year forecasta

5-Year forecastb

10-Year forecastc

Passenger enplanements (number of passengers boarding planes)

-0.2%

14.7%

31.3%

Revenue passenger miles

-0.6%

14.0%

26.9%

Total operationsd

1.9%

25.5%

54.7%

Available seat miles

0.1%

18.6%

40.5%

Source: GAO analysis of FAA data. | GAO-16-210

aincludes 11 1-year Aerospace forecasts made from fiscal year 2004 through 2014.
bincludes 7 5-year forecasts made from fiscal year 2004 through 2010.
cincludes 2 10-year forecasts made from fiscal year 2004 through 2005.
dincludes air carrier, air taxi, general aviation, and military flights arriving at and departing from airports

In developing forecasts, FAA has implemented most of the practices associated with five key risk-management principles GAO selected as applicable to forecasting: (1) setting measurable goals; (2) using best available data; (3) identifying, analyzing, and documenting risk; (4) adopting strategies to respond to risks; and (5) monitoring and reviewing performance. However, FAA lacks some risk-management practices that could enhance FAA's ability to manage for risk and uncertainty. For example, FAA monitors forecast performance by reevaluating the Aerospace and TAF forecast models, but has not fully identified, analyzed, and documented risk. While FAA has identified risks in the Aerospace forecast, it has not reported on the likelihood of these risks. For the TAF, FAA has not systematically assessed the TAF's uncertainty, partly due to recent changes in forecasting methodology. Nor has FAA established error thresholds that would trigger more thorough review of the forecast methodology. Given FAA's reliance on forecasts for decision-making, managing and understanding the nature of uncertainty is important to good decision-making. While FAA has provided limited information about the Aerospace and TAF models, it has not documented the models and assumptions in a manner that would allow stakeholders outside FAA to understand how forecasts are developed or enable FAA to retain organizational knowledge.

Why GAO Did This Study

The FAA annually prepares forecasts of future aviation activity and uses these forecasts to help manage most of its $15 billion in annual spending. While forecasting is inherently uncertain, managing that uncertainty is essential to informed decisions.

GAO was asked to examine the accuracy of and FAA's use of two annual forecasts of aviation activity. This report discusses the accuracy of FAA's forecasts from 2004 through 2014 and strengths and limitations of FAA's consideration of risks in developing its forecasts. This report focuses on the use of the Aerospace and TAF forecasts to inform key operational and investment decisions. GAO compared these two forecasts to actual activity from 2004 through 2014 for the Aerospace forecasts and from 2010 through 2014 for the TAF forecasts and identified factors affecting that accuracy. GAO compared FAA's treatment of risk in developing forecasts to selected risk- management practices recommended by the Office of Management and Budget, GAO, and others.

What GAO Recommends

GAO recommends that the Secretary of the Department of Transportation (DOT) require the FAA to: (1) report on uncertainty and set error response thresholds for both forecasts and (2) document FAA's methods and assumptions for the forecasting models. The DOT partially concurs with the first recommendation and fully concurs with the second. DOT agrees to report on uncertainty but not to set thresholds. GAO believes that thresholds ensure systematic review of forecast accuracy.

For more information, contact Gerald L. Dillingham, Ph.D. at (202) 512-2834 or dillinghamg@gao.gov.

Recommendations for Executive Action

  1. Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.

    Recommendation: To help FAA better manage and understand the uncertainties of its forecasts, the Secretary of Transportation should direct the FAA to apply risk-management practices to analyze and report on uncertainty. Specifically, the FAA should, for both the Aerospace and TAF forecasts, analyze and report the forecast's uncertainty, establish forecast error thresholds, and develop an approach that will prompt forecast review when error thresholds are exceeded, and, for TAF forecasts, monitor and publish multi-year historical error performance, as FAA does for the Aerospace Forecast.

    Agency Affected: Department of Transportation

  2. Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.

    Recommendation: To help FAA better manage and understand the uncertainties of its forecasts, the Secretary of Transportation should direct the FAA to fully document its methods and assumptions in developing the Aerospace and TAF forecasting models to provide greater transparency to internal users and external stakeholders.

    Agency Affected: Department of Transportation

 

Explore the full database of GAO's Open Recommendations »

Oct 5, 2017

Sep 20, 2017

Sep 7, 2017

Aug 28, 2017

Aug 3, 2017

Jul 27, 2017

Jul 26, 2017

Jul 20, 2017

May 31, 2017

Looking for more? Browse all our products here