This is the accessible text file for GAO report number GAO-02-1071 entitled 'National Airspace System: Status of FAA's Standard Terminal Automation Replacement System' which was released on September 17, 2002. This text file was formatted by the U.S. General Accounting Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products’ accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. Report to Congressional Requesters: September 2002: National Airspace System: Status of FAA’s Standard Terminal Automation Replacement System: GAO-02-1071: Contents: Letter: Results in Brief: Background: Major STARS Project Changes Make Comparisons of Costs and Schedules Difficult: FAA Has Changed Its Official Estimates Twice: FAA Officials Said They Plan to Follow Agency Policy in Testing STARS and Addressing Software Problems but Acknowledge that Challenges Remain: Changes in the Schedule for Deploying STARS Led FAA to Implement Interim Projects: Agency Comments: Scope and Methodology: Appendix: Appendix I: GAO Contacts and Staff Acknowledgements: GAO Contacts: Staff Acknowledgments: Table: Table 1: Baseline and Approved Changes to STARS Figures: Figure 1: Standard Terminal Automation Replacement System (STARS) Figure 2: STARS Software Development Phases Figure 3: STARS Operational Schedule Abbreviations: ARTS: Automated Radar Terminal System: DOD: Department of Defense: DOT: Department of Transportation: FAA: Federal Aviation Administration: IG: Inspector General: PTR: program trouble report: STARS: Standard Terminal Automation Replacement System: Letter: September 17, 2002: The Honorable Sonny Callahan The Honorable Ellen O. Tauscher House of Representatives: Since September 1996, the Federal Aviation Administration (FAA) has been developing the Standard Terminal Automation Replacement System (STARS) project to replace the outdated computer equipment that air traffic controllers currently use in some facilities to control air traffic within 5 to 50 nautical miles of an airport. Under the STARS project, controllers at both FAA and the Department of Defense (DOD) locations will receive new hardware and software that produce color displays of aircraft position and flight information.[Footnote 1] In the future, FAA will be able to upgrade the software to provide air traffic control tools, such as those that will allow better spacing of aircraft as they descend into airports. The STARS project is complex, costly, and software intensive. Since 1997, GAO and the Department of Transportation’s (DOT) Office of the Inspector General (IG) have reported on the status of STARS. Recently, the DOT IG raised concerns about FAA’s plans for testing STARS and resolving problems identified during testing before deploying STARS later this year at the Philadelphia air traffic control facility.[Footnote 2] As agreed with your offices, this report addresses the following questions: * How do the currently projected cost and deployment schedule for STARS compare with the original cost and schedule? * How often has FAA changed its approved estimates? * How has FAA responded to the DOT IG’s concerns about the agency’s plans for deploying STARS in Philadelphia? * What has been the impact of changes in the schedule for deploying STARS? You also asked how our analysis of the cost and schedule for STARS compares with FAA’s and the DOT IG’s analyses. We did not perform an independent analysis of the cost and schedule of STARS, in part because of time constraints and in part because FAA is reassessing its plans for modernizing the equipment in terminal facilities. Specifically, FAA is exploring how it can reduce its future STARS costs by using portions of equipment developed for STARS with other automated equipment that it recently deployed to around 140 facilities. FAA officials indicated that they hope to have information in early spring 2003 for making a decision about the future direction of the STARS project. Our work covers FAA’s efforts to develop and deploy STARS;[Footnote 3] we did not review similar efforts by DOD. To conduct this work, we reviewed FAA biweekly status reports, deployment schedules, and test results; and we interviewed officials from FAA’s program and independent testing offices, the MITRE Corporation,[Footnote 4] the DOT IG, and union officials representing the air traffic controller and maintenance technician workforces. We did not independently verify cost and performance data that we received from FAA and did not adjust the costs that FAA and others have reported to reflect the effects of inflation. We performed our work in August 2002 in accordance with generally accepted government auditing standards. Results in Brief: Comparing the currently projected cost and deployment schedule for STARS with the original cost and schedule is difficult because the program presently bears little resemblance to the program envisioned in 1996. At that time, FAA contracted with the Raytheon Corporation for a system using commercially available technology that could be deployed throughout the National Airspace System with very little software development. The plan was to install STARS in 172 facilities at a cost of $940 million with implementation to begin in 1998 and be completed in 2005. However, in 1997, when FAA controllers first tested an early version of this commercially available system, they raised some concerns about the way aircraft position and other data were displayed and updated on the controllers’ radar screens. For example, the controllers said the system did not provide enough detail about an aircraft’s position and movement, which could hamper their ability to monitor traffic movement. To address these concerns, as well as others identified by the technicians who maintain the system, FAA decided to develop a more customized system that is software dependent and to deploy an incremental approach, thereby enabling controllers and technicians to adjust to some changes before introducing others. Because this incremental approach requires more custom software than initially planned, it also costs more and is taking longer to implement than the original STARS project. FAA has officially changed the cost, schedule, and requirements for STARS twice. In October 1999, FAA estimated the cost for its new approach at $1.4 billion with a schedule to begin deploying STARS in 2002 at 188 facilities and complete installation at all facilities by 2008. The second change occurred in March 2002, when FAA lowered its estimate from $1.4 billion to $1.33 billion, reduced the number of facilities receiving STARS from 188 to 74, and changed the date to complete installation at all facilities from 2008 to 2005. The date for deploying STARS to the first location is still 2002. This estimate reflected FAA’s decision to deploy STARS first to those facilities with frequent equipment failures and then to determine how to deploy STARS at remaining facilities in a cost effective manner. Since some facilities had already received new automated terminal equipment when STARS was delayed, these facilities may not get the full STARS system. FAA responded to the DOT IG’s concerns about the agency’s plans for deploying STARS at Philadelphia by stating that it plans to follow its policy for testing STARS and addressing critical software problems. In June 2002, the IG questioned whether FAA’s commitment to deploy STARS in Philadelphia, before testing it first in Memphis as planned, would allow the agency to test the system adequately and address critical software problems identified before the deployment. According to FAA, its plans for deploying STARS in Philadelphia are consistent with its testing policy, which calls for independent operational testing of a system after it has been deployed in one location. While FAA planned to deploy and test STARS in Memphis, which has fewer and less complex air traffic control operations than Philadelphia, FAA believes that going directly to Philadelphia will serve as a more demanding and instructive test site. The independent operational test is scheduled to start in Philadelphia, on November 18, 2002, the day after the system begins operating, and to continue through December 2002. FAA also responded that it is working with Raytheon to address the most critical software problems before using STARS to control air traffic on November 17, 2002. These problems, if not corrected, might prevent FAA from using STARS to control air traffic and might jeopardize safety. In addition, FAA is working with Raytheon and with the system’s internal users to address other issues associated with its deployment schedule and to ensure that problems that might jeopardize safety are being adequately addressed. In spite of these efforts, FAA acknowledges that challenges remain. Specifically, both FAA and the unions representing controllers and maintenance technicians are concerned that FAA may not have enough time to train all employees on the new system before deployment. The agency and the unions are working to address training issues. Because FAA changed the date for deploying STARS at the first facility from 1998 to 2002, it had to implement interim systems to allow it to continue to meet demands for air traffic services. Specifically, it had to replace radar displays and other hardware that were difficult to maintain and had limited capacity to accommodate software that would allow FAA to add new features. FAA documents show that the cost to implement these interim solutions when STARS was delayed was around $90.5 million. We provided DOT with a draft of this report for review and comment. Transportation officials generally agreed with the facts and provided technical and clarifying comments, which we have incorporated into this report as appropriate. Background: STARS will replace controller workstations with new color displays, processors, and computer software at FAA and DOD terminal air traffic control facilities. (See fig. 1.) The total number of facilities scheduled to receive STARS has fluctuated between 70 and 190 because some of the facilities have received interim systems and may not get full STARS. FAA is designing STARS to provide a platform that allows easy and rapid incorporation of new hardware-and software-based tools to help improve controllers’ productivity and make the nation’s airspace safer and managed more efficiently. Figure 1: Standard Terminal Automation Replacement System (STARS): [See PDF for image] Source: FAA. [End of figure] For each acquisition project that the agency undertakes, FAA officially estimates, or develops baselines for, the project’s life-cycle costs, schedule, benefits, and performance in a formal document called the acquisition program baseline. This baseline, which is approved by the Joint Resources Council, FAA’s acquisition decision-making body, is used to monitor a project’s progress in these four areas. The initial acquisition plan for STARS was approved in March 1996; and in September 1996, FAA signed a contract with Raytheon Corporation to acquire STARS. The initial strategy for STARS included two phases: (1) initial system capability, which was to provide the same functions as the equipment in use at the time and (2) final system capability, which was to implement new functions to help controllers move traffic more safely and efficiently. FAA’s acquisition policy requires that projects follow a structured and disciplined test and evaluation process appropriate to the product or facility being tested. Typically, this process includes system testing and field familiarization testing. System testing usually includes development and operational, production, and site acceptance testing. Field familiarization testing includes system and software testing in an operational environment to verify operational readiness. Raytheon and FAA have already conducted a series of tests of the STARS software and plan to continue such testing. As problems arise during these tests, they are documented using program trouble reports (PTR) and are classified from type 1, the most severe, to type 4, the least severe. FAA’s policy defines each type. The policy states that type-1 PTRs prevent the accomplishment of an operational or mission-essential capability and could jeopardize safety and security. Type-2 PTRs adversely affect but does not preclude the performance of an operational or mission-essential capability and a workaround solution is not available. Type-3 PTRs adversely affect but does not preclude the performance of an operational or mission-essential capability and a workaround solution is available. Type-4 PTRs prevent or adversely affect the accomplishment of a nonessential capability and can be handled procedurally. FAA’s contract with Raytheon calls for the contractor to correct all type-1 and type-2 PTRs and, as directed by the government, to correct type-3 and type-4 PTRs. The timing of the corrective action depends, in part, on the severity of the PTR and on its relevance to upcoming activities. From the project’s inception until 2001, a multidisciplinary team oversaw STARS and was responsible for carrying out the acquisition strategy for implementing the project.[Footnote 5] In November 2000, FAA began formulating a new organization that would be responsible for all terminal modernization activities. This new organization, the Terminal Business Service, was intended to move the agency from a project-driven to a point-of-service approach, which would address performance issues at each facility in an integrated fashion. This new organization is now responsible for the STARS program along with other projects for terminal facilities. Major STARS Project Changes Make Comparisons of Costs and Schedules Difficult: The current STARS program is not the program that FAA contracted for in 1996. When FAA awarded the contract in September 1996, it estimated that STARS would cost $940 million and be implemented at 172 facilities by 2005. This estimate was based on acquiring STARS through a commercial off-the-shelf technology with limited development, since a version of this technology was already in use in other countries. In 1997, when FAA first introduced STARS, FAA controllers, who were accustomed to using the older equipment, began to voice concerns about computer-human interface issues that could hamper their ability to monitor air traffic. For example, the controllers noted that many features of the old equipment could be operated with knobs, allowing controllers to focus on the screen. By contrast, STARS was menu-driven and required the controllers to make several keystrokes and use a trackball, diverting their attention from the screen. The maintenance technicians also identified differences between STARS and its backup system that made it difficult to monitor the system. For example, the visual warning alarms and the color codes identifying problems were not consistent between the two systems. Addressing these and other issues required extensive software development because the commercial version of STARS that Raytheon delivered to FAA very tightly coupled the software for the information that would be displayed on the screen and the software that would calculate aircraft position. Because of this coupling, it was difficult for Raytheon to implement the new or modified display requirements that FAA had identified. Accordingly, FAA directed Raytheon to separate the display software from the air traffic control software so that Raytheon could more efficiently implement future display-and air traffic control-related changes to each type of software. To help ensure that STARS meets all of these and other requirements, FAA is developing multiple versions of STARS software, each with specific features, and plans to integrate them into a single version, which will be deployed nationwide. (See fig. 2.) This incremental approach, according to FAA, gives air traffic controllers early experience with the software as it is being developed rather than introducing an entirely new system at the end, as was the case with the commercially available version. For example, FAA has developed a version known as early display configuration, which would replace the controllers’ current displays and monitoring equipment but would use the existing computer and processing software. Figure 2: STARS Software Development Phases: [See PDF for image] Source: GAO’s analysis of FAA data. [End of figure] Figure 2 shows FAA’s new strategy for developing STARS software incrementally. In the early display configurations, FAA separated the display software from the original commercial version and installed and tested the display software, together with some of the original software, at El Paso and Syracuse. In the initial system configuration, FAA took the original software and added some air traffic control software and tested this software at Eglin Air Force Base. After each type of software was tested, FAA began combining the two types to run together in a version called full STARS 2. Subsequent versions of full STARS incorporate additional functions. Figure 3 provides the schedule for when each version of STARS became or is scheduled to become operational at the first facility.[Footnote 6] Figure 3: STARS Operational Schedule: [See PDF for image] Source: GAO’s analysis of FAA data. [End of figure] FAA Has Changed Its Official Estimates Twice: Since 1996, FAA acquisition executives have approved two changes to the cost and schedule estimates for STARS. These changes are presented in table 1. The October 1999 change was approved to give Raytheon enough time to add and modify the display software in order to resolve computer-human interface issues. The March 2002 change was approved after FAA decided to deploy STARS to facilities where frequent equipment failures caused delays; to new facilities; and to facilities where a digital radar, needed to operate STARS, is available.[Footnote 7] Under this strategy, FAA is also assessing how to deploy STARS to remaining facilities in a cost-effective manner. Facilities that previously received new hardware and software so that they could continue to operate while waiting for STARS would get new technology but may not get the full STARS system. Table 1: Baseline and Approved Changes to STARS: Dollars in billions. Date: February 1996 (baseline); Number of FAA facilities receiving STARS: 172; Projected date for first deployment: 1998; Projected date for last deployment: 2005; Estimated cost: $0.94. Date: October 1999; Number of FAA facilities receiving STARS: 188; Projected date for first deployment: 2002; Projected date for last deployment: 2008; Estimated cost: $1.40. Date: March 2002; Number of FAA facilities receiving STARS: 74; Projected date for first deployment: 2002; Projected date for last deployment: 2005; Estimated cost: $1.33. Source: GAO’s presentation of FAA data. [End of table] FAA Officials Said They Plan to Follow Agency Policy in Testing STARS and Addressing Software Problems but Acknowledge that Challenges Remain: FAA responded to the DOT IG’s concerns about the agency’s plans for deploying STARS at Philadelphia by stating that FAA plans to follow its policy for testing STARS and addressing critical software problems. However, FAA officials, controllers and maintenance technicians all have concerns about whether required training can be completed by the November 17, 2002, deployment date. In June 2002, the DOT IG questioned whether FAA’s commitment to deploy STARS in Philadelphia before testing it first in Memphis, as planned, would allow the agency to test the system adequately and address critical software problems that might be identified before deployment. While the Memphis terminal facility has fewer and less complex air traffic control operations than more congested facilities, such as the one in Philadelphia, FAA changed its plans because meeting the commitment to deploy STARS in Philadelphia would not allow enough time to test STARS first in Memphis. FAA testified in September 2001 that it would deploy STARS to Philadelphia to coincide with the opening of a new terminal, scheduled for November 17, 2002. FAA officials said they view the achievement of the November 17, 2002, deployment as important to the agency’s credibility and that they believe they will learn more from testing STARS in Philadelphia, which is more representative of terminal facilities, than they would have learned in Memphis. According to FAA, its plans for deploying STARS in Philadelphia are consistent with its testing policy, which calls for independent operational testing of a system after it has been deployed in one location. Under the current plan, FAA will use STARS to control live traffic at Philadelphia beginning on November 17, 2002--a step signifying initial operating capability--but the current air traffic control system will remain available as a backup. In accordance with its policy, the agency will then conduct independent testing after a “period of use,” scheduled from the day after initial operations through December 2002. At that point, as the policy directs, the agency will declare the system ready for operational use and will complete the switch to the new system. At that time, now scheduled for February 2003, the new system will be formally commissioned and the current system decommissioned. To address critical STARS software problems identified prior to deploying STARS, FAA is attempting to resolve the most critical problems (type-1 and type-2 PTRs) before November 17, 2002. According to FAA’s definition, type-1 problems are those that, if not corrected, might prevent the accomplishment of an operational or mission-essential capability or might jeopardize safety, while type-2 problems adversely affect but does not prevent the accomplishment of an operational or mission-critical capability. FAA’s data showed that as of August 30, 2002, there were 5 type-1 PTRs and 68 type-2 PTRs, against the system being deployed in Philadelphia, that still need to be resolved. FAA officials stated that they have assigned these problems to the contractor and plan to validate the contractor’s fixes. Validation is important because,in some instances, the fixes have not performed as intended. In addition, FAA has identified at least 12 type-3 PTRs and other issues, such as completing required training, that need to be resolved prior to deployment in Philadelphia. FAA is also meeting biweekly with Raytheon to monitor the contractor’s progress in implementing and testing fixes for PTRs. In addition, FAA has installed STARS hardware and an earlier version of STARS software at Philadelphia so that users can become familiar with the system. On September 19, 2002, FAA plans to begin testing the most recent STARS software in Philadelphia. While FAA maintains that its plans for testing STARS and addressing critical software problems are adequate to address the DOT IG’s concerns, the agency is less certain that it will be able to complete the certification training required for maintenance technicians at the Philadelphia terminal before the new version of STARS begins operation in November. The union representing maintenance technicians expressed concern because FAA has not yet finalized the content and schedule of the training for controllers and maintenance technicians on the software that will be deployed in Philadelphia. Under a new training agreement between the union and FAA, on-site certification training-- rather than training at FAA’s central facility in Oklahoma City--is required for all employees before a new system begins operation. Union officials expressed concern that without a finalized training schedule, its members will not have enough time to receive training for certification before the November deployment. FAA officials acknowledged that having enough time for training is an issue. Union and FAA officials are working to solve these concerns prior to deployment. Moreover, according to FAA officials, FAA is meeting with maintenance technicians and controllers to discuss issues related to training, as well as maintenance and testing. Changes in the Schedule for Deploying STARS Led FAA to Implement Interim Projects: Because FAA was not able to deploy STARS according to its original schedule, under which some terminals would have received the new equipment by 1998, FAA implemented several interim projects. Under these projects, FAA replaced failing equipment with new software, radar displays, and other hardware so that the terminals could continue operating while STARS was delayed. Under one project, Common Automated Radar Terminal System (Common ARTS), FAA procured common software for the automated equipment at some of its largest terminal facilities and about 130 smaller facilities.[Footnote 8] Common ARTS provides functions similar to those being designed for STARS, such as the ability to support simultaneous multiple radar displays and adapt to site changes. FAA also purchased 294 ARTS color displays, which replaced aging radar displays at six terminals with those that are high-resolution. The cost for Common ARTS and the ARTS color displays attributable to STARS delays was around $90.5 million. Agency Comments: We provided a draft of this report to DOT. We met with DOT officials, including the Director, Terminal Business Service, FAA. These officials generally agreed with the facts and made technical and clarifying comments, which we have incorporated into this report as appropriate. Scope and Methodology: To determine the projected cost and deployment schedule for STARS, the system’s cost and schedule when the contract was awarded in 1996, and how often FAA changed the approved estimates, we reviewed documentation provided by FAA, including FAA acquisition baseline reports, STARS briefing documents, and IG and GAO reports and testimonies. To determine how FAA responded to the DOT IG’s concerns about FAA’s plans for deploying STARS in Philadelphia, we interviewed officials from FAA’s Terminal Business Service and the Office of Independent Operational Test and Evaluation, the DOT IG, the MITRE Corporation, and union officials representing the air traffic controller and maintenance technician workforces. We reviewed STARS program office documentation on the prioritization of trouble reports and agency policy and guidance on critical trouble reports and test and evaluation requirements. To determine the impact of changes in the schedule for deploying STARS, we reviewed FAA documentation on the interim projects and the associated costs and also reviewed IG and GAO products on the impact of delays on implementing STARS. We did not independently verify the data we received from FAA. We performed our work in August 2002 in accordance with generally accepted government auditing standards.: We are sending copies of this report to interested Members of Congress, the Secretary of Transportation, and the Administrator, FAA. We will also make copies available to others upon request. In addition, this report will be available at no charge on the GAO Web site at http:// www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3650. I can also be reached by E-mail at dillinghamg@gao.gov. Key contributors to this report are listed in appendix I. Gerald L. Dillingham, Ph.D. Director, Physical Infrastructure: Signed By Gerald L. Dillingham [End of section] Appendixes: Appendix I: GAO Contacts and Staff Acknowledgments: GAO Contacts: Gerald L. Dillingham (202) 512-3650 Belva Martin (202) 512-4285: Staff Acknowledgments: In addition to those individuals listed above, Nabajyoti Barkakati, Geraldine Beard, Elizabeth Eisenstadt, Tammi Nguyen, Madhav Panwar, and Glenda Wright made key contributions to this report. : FOOTNOTES: [1] STARS is a joint procurement with DOD, which is purchasing up to 199 units for its facilities. [2] Department of Transportation Inspector General, Follow-up on Federal Aviation Administration ‘s Acquisition of Standard Terminal Automation Replacement System, JA-10:x60500 (Washington, D.C.: June 3, 2002). [3] Throughout this report, we use the term “deploy” to denote efforts by FAA to put STARS software, hardware, and other supporting equipment into a facility to test it and eventually use it to control traffic. [4] MITRE provides air traffic control, air traffic management, and airport systems engineering support to FAA and others. [5] The team included, among others, members from FAA’s acquisition, systems engineering, air traffic, and airway facilities organizations. [6] “Operational” denotes that STARS has been deployed, implemented, and is meeting requirements. [7] The Joint Resources Council verbally approved the March 2002 estimate but written approval is pending receipt of additional information from another program that was also reviewed. [8] FAA initially began the Common ARTS project because of delays in a program that preceded STARS. Under the initial phase of this project, developed by Lockheed Martin Corporation, equipment was delivered to 131 small-to medium-sized facilities beginning in 1997 and to 5 large facilities in 1998 and 1999. However, FAA later purchased equipment for five additional facilities, which was installed in 2001 and 2002. GAO’s Mission: The General Accounting Office, the investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO’s Web site ( www.gao.gov ) contains abstracts and full-text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics. Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select “Subscribe to daily E-mail alert for newly released products” under the GAO Reports heading. Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Public Affairs: Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S. General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C. 20548: