This is the accessible text file for GAO report number GAO-10-535R 
entitled 'GAO Review of the Department of Homeland Security's 
Certification of the Secure Flight Program--Cost and Schedule 
Estimates' which was released on April 6, 2010. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

GAO-10-535R: 

United States Government Accountability Office: 
Washington, DC 20548: 

April 5, 2010: 

Congressional Committees: 

Subject: GAO Review of the Department of Homeland Security's 
Certification of the Secure Flight Program--Cost and Schedule 
Estimates: 

The matching of airline passenger information against terrorist 
watchlist records (watchlist matching) is a frontline defense against 
acts of terrorism that target the nation's civil aviation system. In 
general, passengers identified as matches to the No-Fly list are 
prohibited from boarding commercial flights, while those matched to 
the Selectee list are required to undergo additional 
screening.[Footnote 1] Historically, airline passenger prescreening 
against watchlist records has been performed by commercial air 
carriers. 

As required by the Intelligence Reform and Terrorism Prevention Act of 
2004, the Department of Homeland Security's (DHS) Transportation 
Security Administration (TSA) has developed an advanced passenger 
prescreening program--known as Secure Flight--to assume from air 
carriers the function of matching passenger information against 
terrorist watchlist records.[Footnote 2] Since fiscal year 2004, TSA 
has received $358 million in appropriated funds for the development 
and implementation of Secure Flight, according to program officials. 

Also, since fiscal year 2004, GAO has been mandated to assess the 
development and implementation of the Secure Flight program.[Footnote 
3] We have reported on numerous challenges the program has faced, 
including those related to protecting passenger privacy, completing 
performance testing, fully defining and testing security requirements, 
and establishing reliable cost and schedule estimates, among other 
things.[Footnote 4] We have made recommendations to address these 
challenges, and TSA has generally agreed with them and has taken 
corrective actions. 

Section 522(a) of the Department of Homeland Security Appropriations 
Act, 2005, set forth 10 conditions related to the development and 
implementation of the Secure Flight program that the Secretary of 
Homeland Security must certify have been successfully met before the 
program may be implemented or deployed on other than a test basis. 
[Footnote 5] Although DHS certified that it had satisfied all 10 
conditions in September 2008, our initial assessment found that TSA 
generally had not achieved 5 of the 10 statutory conditions and that 
the agency had not demonstrated Secure Flight's operational readiness. 
In response, TSA took additional actions and, in late January 2009, we 
found that the agency had generally achieved 6 of the 10 conditions, 
conditionally achieved 3 conditions--subject to the timely completion 
of planned activities--and generally had not achieved 1 condition. We 
also concluded that the actions TSA had taken were sufficient to 
support beginning Secure Flight initial operations. We continued to 
review the program and reported in May 2009 that TSA had generally 
achieved 9 of the 10 statutory conditions and had conditionally 
achieved 1 condition, subject to the timely completion of planned 
activities for developing appropriate cost and schedule estimates. 
[Footnote 6] Table 1 shows the status of the 10 conditions as of April 
2009. 

Table 1: GAO Assessment of Whether DHS Had Achieved the 10 Statutory 
Conditions, as of April 2009: 

Legislative condition topic: System of due process (redress); 
Generally achieved[A]: [Check]; 
Conditionally achieved[B]: [Empty]; 
Generally not achieved[C]: [Empty]. 

Legislative condition topic: Extent of false-positive errors; 
(misidentifications); 
Generally achieved[A]: [Check]; 
Conditionally achieved[B]: [Empty]; 
Generally not achieved[C]: [Empty]. 

Legislative condition topic: Performance of stress testing and; 
efficacy and accuracy of search tools; 
Generally achieved[A]: [Check]; 
Conditionally achieved[B]: [Empty]; 
Generally not achieved[C]: [Empty]. 

Legislative condition topic: Establishment of an internal oversight; 
board; 
Generally achieved[A]: [Check]; 
Conditionally achieved[B]: [Empty]; 
Generally not achieved[C]: [Empty]. 

Legislative condition topic: Operational safeguards to reduce abuse; 
opportunities; 
Generally achieved[A]: [Check]; 
Conditionally achieved[B]: [Empty]; 
Generally not achieved[C]: [Empty]. 

Legislative condition topic: Substantial security measures to prevent; 
unauthorized access by hackers; 
Generally achieved[A]: [Check]; 
Conditionally achieved[B]: [Empty]; 
Generally not achieved[C]: [Empty]. 

Legislative condition topic: Effective oversight of system use and; 
operation; 
Generally achieved[A]: [Check]; 
Conditionally achieved[B]: [Empty]; 
Generally not achieved[C]: [Empty]. 

Legislative condition topic: No specific privacy concerns with the; 
system's technological architecture; 
Generally achieved[A]: [Check]; 
Conditionally achieved[B]: [Empty]; 
Generally not achieved[C]: [Empty]. 

Legislative condition topic: Accommodation of states with unique; 
transportation needs; 
Generally achieved[A]: [Check]; 
Conditionally achieved[B]: [Empty]; 
Generally not achieved[C]: [Empty]. 

Legislative condition topic: Appropriateness of life-cycle cost; 
estimates and program plans; 
Generally achieved[A]: [Empty]; 
Conditionally achieved[B]: [Check]; 
Generally not achieved[C]: [Empty]. 

Source: GAO analysis. 

[A] For generally achieved, TSA had demonstrated that it completed all 
key activities related to the condition in accordance with applicable 
federal guidelines and related best practices, which should reduce the 
risk of the program experiencing cost, schedule, or performance 
shortfalls. 

[A] For conditionally achieved, TSA had completed some key activities 
and had defined plans for completing remaining activities that if 
effectively implemented as planned, should result in a reduced risk of 
the program experiencing cost, schedule, or performance shortfalls. 

[C] For generally not achieved, TSA had not demonstrated that it 
completed all key activities related to the condition in accordance 
with applicable federal guidelines and related best practices and did 
not have defined plans for completing the remaining activities, and 
the uncompleted activities result in an increased risk of the program 
experiencing cost, schedule, or performance shortfalls. 

[End of table] 

In our May 2009 report, we concluded that the actions TSA had 
completed and those planned had reduced the risks associated with 
implementing the program. We noted, however, that while TSA's ability 
to fully achieve the statutory condition related to developing 
appropriate cost and schedule estimates did not affect the Secure 
Flight system's operational readiness, having reliable cost and 
schedule estimates would allow for better insight into the management 
of program resources and time frames as the program is deployed. DHS 
concurred with our assessment and noted that TSA would continue to 
work on the Secure Flight program's cost and schedule estimates until 
the statutory condition is generally achieved. 

The Conference Report accompanying the Department of Homeland Security 
Appropriations Act, 2010, directed GAO to continue its review of the 
Secure Flight program until all 10 statutory conditions are generally 
achieved.[Footnote 7] In accordance with that mandate, this report 
addresses the extent to which TSA met the Secure Flight condition 
related to the appropriateness of cost and schedule estimates and any 
shortfalls or limitations in meeting the requirements in GAO's Cost 
Estimating and Assessment Guide.[Footnote 8] 

Our overall methodology included (1) identifying key activities 
related to the cost and schedule condition; (2) identifying federal 
guidance and related best practices that are relevant to successfully 
meeting the condition; (3) analyzing whether TSA has demonstrated 
through verifiable analysis and documentation, as well as oral 
explanation, that the guidance has been followed and best practices 
have been met; and (4) assessing any risks associated with not fully 
following applicable guidance and meeting best practices. 
Specifically, we reviewed the program's life-cycle cost estimate, 
integrated master schedule,[Footnote 9] and other relevant 
documentation against best practices, including those contained in our 
Cost Estimating and Assessment Guide. We also interviewed key program 
officials overseeing these activities and consulted with a scheduling 
expert to identify risks to the integrated master schedule. We 
assessed the status of TSA's efforts to develop appropriate life-cycle 
cost estimates and program plans based on the agency's plan of action 
developed in early 2009 and related documentation that TSA provided to 
us through February 2010. The plan detailed the Secure Flight program 
management office's proposed activities and time frames for addressing 
weaknesses that we identified in the program's cost and schedule 
estimates. 

We conducted this performance audit from October 2009 to April 2010 in 
accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe 
that the evidence obtained provides a reasonable basis for our 
findings and conclusions based on our audit objectives. 

Results in Brief: 

TSA has generally achieved the statutory condition related to the 
appropriateness of Secure Flight's life-cycle cost and schedule 
estimates, and thus has generally achieved all 10 statutory conditions 
related to the development and implementation of the program. Although 
the program's cost and schedule estimates do not fully meet all 
related best practices, TSA has demonstrated that it completed all key 
activities and our overall assessment found that the agency had 
substantially satisfied best practices for developing the cost and 
schedule estimates, as shown in table 2. In general, GAO's methodology 
for assessing a program's cost and schedule estimates is based on the 
extent to which an agency has "satisfied" best practices for 
developing the estimates. For purposes of this review, our assessment 
that TSA had substantially satisfied best practices equates to the 
agency generally achieving the statutory condition. 

Table 2: GAO Final Assessment of Secure Flight Cost and Schedule 
Estimates versus Best Practices, as of February 2010: 

Reliable cost estimate: 

Best practice[A]: Comprehensive - The estimate should include all 
costs over the life of the program; 
Extent satisfied: Partially satisfied[B]; 
Overall assessment: Substantially satisfied[C]. 

Best practice[A]: Well documented - The estimate should clearly define 
all key program or system characteristics; 
Extent satisfied: Satisfied[D]; 
Overall assessment: Substantially satisfied[C]. 

Best practice[A]: Accurate - The estimate should provide for results 
that are unbiased and should not be overly conservative or optimistic; 
Extent satisfied: Partially satisfied; 
Overall assessment: Substantially satisfied[C]. 

Best practice[A]: Credible - The estimate should discuss any 
limitations because of uncertainty surrounding data or assumptions; 
Extent satisfied: Satisfied; 
Overall assessment: Substantially satisfied[C]. 

Reliable schedule: 

Best practice[A]: Capturing key activities - The schedule should 
reflect all key government and contractor activities; 
Extent satisfied: Partially satisfied; 
Overall assessment: Substantially satisfied[C]. 

Best practice[A]: Sequencing key activities - The schedule should be 
planned so that critical program dates can be met; 
Extent satisfied: Substantially satisfied; 
Overall assessment: Substantially satisfied[C]. 

Best practice[A]: Establishing the duration of key activities - The 
schedule should realistically reflect how long each activity will take 
to execute; 
Extent satisfied: Substantially satisfied; 
Overall assessment: Substantially satisfied[C]. 

Best practice[A]: Assigning resources to key activities - The schedule 
should reflect what resources (e.g., labor and materials) are needed 
to do the work; 
Extent satisfied: Satisfied; 
Overall assessment: Substantially satisfied[C]. 

Best practice[A]: Integrating key activities horizontally and 
vertically - The schedule should sequentially link activities 
horizontally and show relationships between tasks and subtasks 
vertically; 
Extent satisfied: Substantially satisfied; 
Overall assessment: Substantially satisfied[C]. 

Best practice[A]: Establishing the critical path for key activities - 
The scheduling software should identify the longest duration path 
through the sequenced list of key activities; 
Extent satisfied: Partially satisfied; 
Overall assessment: Substantially satisfied[C]. 

Best practice[A]: Identifying the float time between key activities - 
The schedule should identify the time that a predecessor activity can 
slip before the delay affects successor activities; 
Extent satisfied: Partially satisfied; 
Overall assessment: Substantially satisfied[C]. 

Best practice[A]: Schedule risk analysis should be performed - The 
schedule risk analysis should predict the level of confidence in 
meeting a program's completion date; 
Extent satisfied: Satisfied; 

Best practice[A]: Distributing reserves to high-risk activities - The 
schedule should include a buffer or reserve of extra time for 
contingencies; 
Extent satisfied: Substantially satisfied; 
Overall assessment: Substantially satisfied[C]. 

Source: GAO analysis. 

[A] Enclosure I contains additional information on each cost and 
schedule best practice. 

[B] For partially satisfied, TSA officials provided evidence that 
satisfies about half of the criterion. 

[C] For substantially satisfied, TSA officials provided evidence that 
satisfies the majority of the criterion. 

[D] For satisfied, TSA officials provided complete evidence that 
satisfies the entire criterion. 

[End of table] 

According to TSA, the Secure Flight program's estimated life-cycle 
cost is $1.36 billion through fiscal year 2020. TSA plans to complete 
assumption of the watchlist-matching function from air carriers for 
all domestic flights in May 2010 and to assume this function for all 
international flights departing to and from the United States by 
December 2010. If effectively maintained and updated, TSA's cost and 
schedule estimates should help ensure oversight and accountability of 
the Secure Flight program and provide assurance that it will be 
delivered within estimated costs and time frames. 

TSA Has Generally Achieved the Statutory Condition Related to 
Appropriate Life-Cycle Cost and Schedule Estimates: 

In May 2009, we reported that TSA had conditionally achieved the 
statutory requirement related to Secure Flight's life-cycle cost and 
schedule estimates, based on the agency's plan of action for 
addressing weaknesses we identified. Since then, TSA has taken several 
steps to improve these estimates and implement our prior 
recommendations; thus, we now consider the legislative requirement to 
be generally achieved. 

Secure Flight's Life-Cycle Cost Estimate Substantially Satisfies GAO 
Best Practices: 

A reliable cost estimate is critical to the success of any program 
since it provides the basis for informed investment decision making, 
realistic budget formulation and program resourcing, meaningful 
progress measurement, proactive course correction when warranted, and 
accountability for results. 

In our May 2009 report, we noted that Secure Flight's $1.36 billion 
life-cycle cost estimate was well documented in that it clearly stated 
the purpose, source, assumptions used, and calculations made. However, 
it was not comprehensive (it did not include all costs), fully 
accurate, or credible. As a result, the life-cycle cost estimate did 
not provide a meaningful baseline from which to track progress, hold 
TSA accountable, and provide a basis for sound investment decision 
making. To address recommendations that were in the draft of our May 
2009 report,[Footnote 10] TSA established a plan of action to, among 
other things, (1) provide more detail in the work necessary to 
accomplish the program's objectives, (2) properly align the cost 
estimate with the schedule of work to be performed, (3) develop an 
independent cost estimate performed by a contractor, (4) have the DHS 
Cost Analysis Division assess the life-cycle cost estimate, and (5) 
perform cost uncertainty and sensitivity analyses on the estimate. 
[Footnote 11] 

Over the past year, TSA has significantly improved the Secure Flight 
program's life-cycle cost estimate and our overall assessment found 
that the agency has substantially satisfied GAO best practices related 
to the four characteristics of a reliable cost estimate--
comprehensive, well documented, accurate, and credible. For example, 
in October 2009, a TSA contractor developed an independent cost 
estimate for the program, which the agency used to validate the 
credibility of the existing life-cycle cost estimate. TSA has also 
conducted a cost uncertainty analysis and a sensitivity analysis on 
the independent cost estimate. 

The DHS Cost Analysis Division also reviewed the Secure Flight life- 
cycle cost estimate in late 2009 and found that the estimate generally 
met GAO best practices for cost estimating. One of the reasons the 
estimate still did not fully meet GAO best practices was because the 
estimate was based on a functional breakdown of the work to be 
performed (the work breakdown structure, or WBS), instead of a product-
oriented structure. While functional activities--for example, 
engineering or quality control--are necessary for supporting a 
product's development, a WBS generally should not be organized around 
them. Rather, best practices recommend that a product-oriented WBS be 
used that reflects the cost, schedule, and technical performance on 
specific portions of a program so that project performance can be 
directly related to developed products. However, we recognize that it 
is not feasible to reorganize the WBS for Secure Flight at this point 
in the program, since the time to reorganize it would exceed the time 
phase of the program and rebaselining the program--which would be 
required for a new WBS--would be cost prohibitive.[Footnote 12] DHS 
Cost Analysis Division officials stated that they expect the life-
cycle cost estimate to adhere to a product-oriented WBS for the next 
major acquisition milestone, which is scheduled to be completed by 
February 2011. While it is important for TSA to develop a product-
oriented WBS for Secure Flight, the other actions the agency has taken 
are sufficient for us to conclude that TSA has substantially satisfied 
GAO's best practices for developing a reliable cost estimate for the 
program. 

Enclosure I contains additional information on our final assessment of 
Secure Flight's life-cycle cost estimate relative to GAO's best 
practices. 

Secure Flight's Program Schedule Substantially Satisfies GAO Best 
Practices: 

The success of any program depends in part on having a reliable 
schedule specifying when the program's set of work activities will 
occur, how long they will take, and how they relate to one another. As 
such, the schedule not only provides a road map for the systematic 
execution of a program, but it also provides the means by which to 
gauge progress, identify and address potential problems, and promote 
accountability. 

In our May 2009 report, we noted that Secure Flight's schedule was 
developed using some of GAO's best practices for developing schedules, 
but several key practices were not fully employed that are fundamental 
to having a schedule that provides a sufficiently reliable basis for 
estimating costs, measuring progress, and forecasting slippages. In 
addition, best practices require that a schedule identify the longest 
duration path through the sequenced list of key activities--known as 
the schedule's critical path--where if any activity slips along this 
path, the entire program will be delayed. TSA's schedule did not 
include a critical path, which could help program managers better 
understand the effect of any delays. We also noted that updating the 
Secure Flight program's schedule is important because of the 
significant cost and time that remained to be incurred for TSA to 
assume the watchlist-matching function for all domestic flights and to 
develop, test, and deploy the functionality to assume the watchlist 
matching function for international flights. 

To address recommendations that were in the draft of our May 2009 
report,[Footnote 13] TSA established a plan of action to develop, 
among other things, (1) a sequenced and logical schedule to accurately 
calculate float time and a critical path;[Footnote 14] (2) a schedule 
that fully identifies the resources needed to complete key 
activities;[Footnote 15] (3) a schedule that includes realistic 
estimates of activity duration; and (4) a schedule risk analysis that 
will be used by TSA leadership to distribute resources to high-risk 
activities on the critical path, which if delayed would delay the 
entire program. According to TSA, this revised schedule would better 
forecast the completion date for the project. 

Since we issued our May 2009 report, TSA has taken several steps to 
improve the Secure Flight program's schedule, and our overall 
assessment found that the agency has substantially satisfied GAO's 
best practices related to the nine characteristics of a reliable 
schedule estimate--capturing key activities, sequencing key 
activities, establishing duration of key activities, assigning 
resources to key activities, integrating key activities horizontally 
and vertically,[Footnote 16] establishing a critical path, identifying 
float time, performing a schedule risk analysis, and distributing 
reserves to high-risk activities.[Footnote 17] For example, TSA has 
assigned resources for each activity, conducted a schedule risk 
analysis, and undertaken strategies to mitigate risks to the program 
that could affect the schedule. To minimize the risk of schedule 
delay, TSA prioritized the schedule for assuming the watchlist-
matching function from air carriers and reallocated staff resources. 

In addition, TSA hired a contractor to complete an independent 
schedule risk analysis and implemented some of the recommendations 
made in that analysis. However, best practices for scheduling are not 
fully satisfied, because scheduling constraints prevent the scheduling 
software from automatically calculating the start and finish dates of 
remaining activities. These constraints also prevent the software from 
automatically creating a valid critical path, which hampers the 
ability of decision makers to optimally allocate resources (e.g., move 
resources from low-risk activities that are not on the critical path 
to high-risk activities). However, the Secure Flight program office 
has justified the use of these constraints, as the start dates for 
activities related to TSA's assumption of the watchlist-matching 
function from air carriers are legally binding between the Secure 
Flight program office and the airlines. 

On January 27, 2009, the Secure Flight program began initial 
operations--assuming the watchlist-matching function for a limited 
number of domestic flights for one airline--and has since phased in 
additional flights and airlines. According to TSA, Secure Flight plans 
to assume this function for a total of over 70 U.S. air carriers and 
about 150 foreign carriers.[Footnote 18] As of March 31, 2010, TSA was 
working with 74 U.S. air carriers and 19 foreign carriers. 
Specifically, 

* Secure Flight had fully assumed the watch-list matching function for 
39 U.S. air carriers (domestic flights only), had partially assumed 
this function for another 11 U.S. carriers,[Footnote 19] and was in 
testing with another 24 U.S. carriers, and: 

* Secure Flight had fully assumed the matching function for 5 foreign 
air carriers (international flights departing to and from the United 
States) and was in testing with another 14 foreign carriers. 

TSA plans to complete assumption of the watchlist-matching function 
from air carriers for all domestic flights in May 2010 and to assume 
this function for all international flights departing to and from the 
United States by December 2010, assuming the air carriers make the 
necessary system changes as required to be compliant with the Secure 
Flight Final Rule.[Footnote 20] Specifically, the rule contains 
requirements for air carriers to follow as TSA implements and operates 
Secure Flight, including the collection of full name and date-of-birth 
information from airline passengers to facilitate watch-list matching. 
To date, TSA has not experienced any unexpected challenges with 
aircraft operators currently testing with Secure Flight that would 
necessitate an extension to the schedule, according to program 
officials. 

Enclosure I contains additional information on our final assessment of 
Secure Flight's schedule estimate relative to GAO's best practices. 

Agency Comments: 

On March 29, 2010, we provided a draft of this report to DHS for 
comment. DHS did not provide written agency comments. However, TSA 
provided technical comments, which we incorporated in this report 
where appropriate. TSA also noted that it appreciated the assistance 
we have provided in helping TSA meet Congress' 10 statutory conditions 
related to the Secure Flight program, and our public recognition that 
TSA has generally achieved the single outstanding statutory condition 
related to the program's appropriate life-cycle cost and schedule 
estimates. 

We are sending copies of this report to the appropriate congressional 
committees, the Secretary of Homeland Security, and other interested 
parties. This report also is available at no charge on the GAO Web 
site at [hyperlink, http://www.gao.gov]. 

Should you or your staff have any questions about this report, please 
contact me at (202) 512-4379 or lords@gao.gov. Contact points for our 
Offices of Congressional Relations and Public Affairs may be found on 
the last page of this report. Key contributors to this report were 
Karen Richey, Assistant Director; Eric Erdman, Assistant Director; 
Jason Lee; and Victoria Miller. 

Signed by: 

Stephen M. Lord:
Director, Homeland Security and Justice Issues: 

Enclosures - 1: 

List of Committees: 

The Honorable John D. Rockefeller, IV:
Chairman:
The Honorable Kay Bailey Hutchison:
Ranking Member:
Committee on Commerce, Science, and Transportation:
United States Senate: 

The Honorable Joseph I. Lieberman:
Chairman:
The Honorable Susan M. Collins:
Ranking Member:
Committee on Homeland Security and Governmental Affairs:
United States Senate: 

The Honorable Patrick J. Leahy:
Chairman:
The Honorable Jeff Sessions:
Ranking Member:
Committee on the Judiciary:
United States Senate: 

The Honorable Robert C. Byrd:
Chairman:
The Honorable George Voinovich:
Ranking Member:
Subcommittee on Homeland Security:
Committee on Appropriations:
United States Senate: 

The Honorable Bennie G. Thompson:
Chairman:
The Honorable Peter T. King:
Ranking Member:
Committee on Homeland Security:
House of Representatives: 

The Honorable Edolphus Towns:
Chairman:
The Honorable Darrell Issa:
Ranking Member:
Committee on Oversight and Government Reform:
House of Representatives: 

The Honorable James L. Oberstar:
Chairman:
The Honorable John L. Mica:
Ranking Member:
Committee on Transportation and Infrastructure:
House of Representatives: 

The Honorable David E. Price:
Chairman:
The Honorable Harold Rogers:
Ranking Member:
Subcommittee on Homeland Security:
Committee on Appropriations:
House of Representatives: 

[End of section] 

Enclosure I: GAO Analyses of Secure Flight's Life-Cycle Cost and 
Schedule Estimates against Best Practices: 

Our research has identified several best practices that serve as the 
basis for effective program cost and schedule estimating.[Footnote 21] 
Our assessments of the extent to which Secure Flight's cost and 
schedule estimates satisfied best practices were based on the 
following criteria: 

* Not satisfied: Project officials provided no evidence that satisfies 
any part of the criterion. 

* Minimally satisfied: Project officials provided evidence that 
satisfies less than half of the criterion. 

* Partially satisfied: Project officials provided evidence that 
satisfies about half of the criterion. 

* Substantially satisfied: Project officials provided evidence that 
satisfies the majority of the criterion. 

* Satisfied: Project officials provided complete evidence that 
satisfies the entire criterion. 

Specifically, we have identified four characteristics of a reliable 
cost estimate: comprehensive, well documented, accurate, and credible. 
Table 3 summarizes the results of our final assessment of the Secure 
Flight program's cost estimate relative to the four characteristics of 
a reliable cost estimate, as of February 2010. 

Table 3: GAO Final Assessment of Secure Flight Cost Estimate Compared 
to Best Practices for a Reliable Cost Estimate, as of February 2010: 

Best practice: Comprehensive; 
Explanation: The cost estimates should include both government and 
contractor costs over the program's full life cycle, from the 
inception of the program through design, development, deployment, and 
operation and maintenance to retirement. They should also provide an 
appropriate level of detail to ensure that cost elements are neither 
omitted nor double counted and include documentation of all cost-
influencing ground rules and assumptions; 
Satisfied? Partially satisfied; 
GAO analysis: The Department of Homeland Security's (DHS) Cost 
Analysis Division (CAD) reviewed the Secure Flight life-cycle cost 
estimate in 2009. CAD concluded that one of the reasons the estimate 
did not entirely meet GAO best practices for cost estimating was 
because the cost estimate adhered to a functional work breakdown 
structure (WBS) instead of the best practice standard product-oriented 
WBS. However, CAD accepted the estimate conditionally, determining 
that the time to correct the WBS would exceed the time phase of the 
program, and that rebaselining the program--required for a new WBS--
would be cost prohibitive. CAD officials stated that they expected the 
life cycle cost estimate to adhere to a product-oriented WBS for the 
next major acquisition milestone. GAO agreed with CAD's conclusions. 

Best practice: Well; 
documented; 
Explanation: The cost estimates should have clearly defined 
descriptions of key program or system characteristics. Additionally, 
they should capture in writing such things as the source data used and 
their significance, the calculations performed and their results, and 
the rationale for choosing a particular estimating method. Moreover, 
this information should be captured in such a way that the data used 
to derive the estimate can be traced back to, and verified against, 
their sources. The final cost estimate should be reviewed and accepted 
by management; 
Satisfied? Satisfied; 
GAO analysis: TSA has fully met this criterion. 

Best practice: Accurate; 
Explanation: The cost estimates should provide for results that are 
unbiased and not overly conservative or optimistic. In addition, the 
estimates should be updated regularly to reflect material changes in 
the program, and steps should be taken to minimize mathematical 
mistakes and their significance. Among other things, an estimate 
should be grounded in a historical record of cost estimating and 
actual experiences on comparable programs; 
Satisfied? Partially satisfied; 
GAO analysis: As in the case of the "Comprehensive" best practice, CAD 
concluded that one of the reasons the estimate did not entirely meet 
GAO best practices for cost estimating was because the cost estimate 
adhered to a functional WBS instead of the best practice standard 
product-oriented WBS. However, CAD accepted the estimate 
conditionally, determining that the time to correct the WBS would 
exceed the time phase of the program, and that rebaselining the 
program--required for a new WBS--would be cost prohibitive. CAD 
officials stated that they expected the life cycle cost estimate to 
adhere to a product-oriented WBS for the next major acquisition 
milestone. GAO agreed with CAD's conclusions. 

Best practice: Credible; 
Explanation: The cost estimates should discuss any limitations in the 
analysis performed due to uncertainty surrounding data or assumptions. 
Further, the estimates' derivation should provide for varying any 
major assumptions and recalculating outcomes based on sensitivity 
analyses, and their associated risks/uncertainty should be disclosed. 
Also, the estimates should be verified based on cross-checks using 
other estimating methods and by comparing the results with those of 
independent cost estimates; 
Satisfied? Satisfied; 
GAO analysis: CAD had performed assessments of the Secure Flight life-
cycle cost estimate prior to 2009 and identified several areas of 
deficiency in the estimate. These deficiencies included the estimate 
not being well documented, an inability to trace data back to their 
original sources, a functional WBS instead of the best practice 
standard product-oriented WBS, and the lack of a risk analysis. 
However, CAD accepted the estimates conditionally, determining that 
the time to correct these deficiencies would exceed the time phase of 
the program, and rebaselining the program--required for a new WBS--
would be cost prohibitive. CAD officials reviewed an updated Secure 
Flight life-cycle cost estimate from April 2009 and determined that 
the estimate generally adhered to best practices in GAO's Cost 
Estimating and Assessment Guide. CAD officials told us that they will 
require the Secure Flight program to adhere to a product-oriented WBS 
for the next major acquisition milestone point. The Secure Flight 
program office hired a contractor to develop an independent cost 
estimate, which was completed in October 2009. This independent 
estimate was reviewed by the program office and used to assess the 
reasonableness of the Secure Flight life-cycle cost estimate. The 
independent cost estimate included a sensitivity analysis on key 
variables that might affect the long-term cost of the program, as well 
as a cost risk uncertainty analysis that determined the level of 
confidence associated with the point estimate. 

Source: GAO analysis. 

[End of table] 

GAO has also identified nine best practices that characterize a 
reliable schedule estimate--capturing key activities, sequencing key 
activities, establishing duration of key activities, assigning 
resources to key activities, integrating key activities horizontally 
and vertically, establishing critical path, identifying float time, 
performing a schedule risk analysis, and distributing reserves to high-
risk activities. Table 4 summarizes the results of our final 
assessment of the Secure Flight program's schedule relative to the 
nine schedule estimating best practices, as of February 2010. 

Table 4: GAO Final Assessment of Secure Flight Schedule Compared to 
Best Practices for Schedule Estimating, as of February 2010: 

Best practice; 
Explanation; 
Satisfied?; 
GAO analysis. 

Best practice: Capturing key activities; 
Explanation: The schedule should reflect all key activities as defined 
in the program's WBS, to include activities to be performed by both 
the government and its contractors; 
Satisfied? Partially satisfied; 
GAO analysis: The updated integrated master schedule (IMS) includes 
baselined international deployment dates. DHS's CAD reviewed the 
Secure Flight WBS in 2009 and concluded that it was functionally based 
rather than being product oriented, which is the best practice 
standard. However, CAD accepted the WBS conditionally, determining 
that the time to correct the WBS would exceed the time phase of the 
program, and that rebaselining the program--required for a new WBS--
would be cost prohibitive. CAD officials stated that they expected the 
program to adhere to a product-oriented WBS for the next major 
acquisition milestone. GAO agreed with CAD's conclusions. 

Best practice: Sequencing key activities; 
Explanation: The schedule should be planned so that it can meet 
critical program dates. To meet this objective, key activities need to 
be logically sequenced in the order that they are to be carried out. 
In particular, activities that must finish prior to the start of other 
activities (i.e., predecessor activities), as well as activities that 
cannot begin until other activities are completed (i.e., successor 
activities), should be identified. By doing so, interdependencies 
among activities that collectively lead to the accomplishment of 
events or milestones can be established and used as a basis for 
guiding work and measuring progress; 
Satisfied? Substantially satisfied; 
GAO analysis: The logic in the updated IMS is straightforward, and 
none of the remaining activities have missing dependencies. However, 
the use of "Start No Earlier Than" constraints for each airline 
testing phase 1 activity prevents the schedule from being able to 
dynamically calculate the completion date. The program office has 
justified the use of these constraints, as airline testing phase start 
dates are legally binding dates negotiated between the program office 
and each of the approximately 200 airlines. 

Best practice: Establishing the duration of key activities; 
Explanation: The schedule should realistically reflect how long each 
activity will take to execute. In determining the duration of each 
activity, the same rationale, historical data, and assumptions used 
for cost estimating should be used. Durations should be as short as 
possible and have specific start and end dates. Excessively long 
periods needed to execute an activity should prompt further 
decomposition so that shorter execution durations will result. The 
schedule should be continually monitored to determine when forecasted 
completion dates differ from the planned dates, which can determine 
whether schedule variances will affect downstream work; 
Satisfied? Substantially satisfied; 
GAO analysis: The remaining activities in the updated IMS generally 
met best practices for activity duration. For example, 75 percent of 
the remaining activities have duration of 4 days or less, and only 1 
percent of remaining activities have duration of greater than 13 days. 
However, because 48 percent of the remaining activities have duration 
of 1 day, concern remains that the schedule is assuming too much 
productivity for an 8-hour day. 

Best practice: Assigning resources to key activities; 
Explanation: The schedule should reflect what resources (e.g., labor, 
material, and overhead) are needed to do the work, whether all 
required resources will be available when needed, and whether any 
funding or time constraints exist; 
Satisfied? Satisfied; 
GAO analysis: Based on our analysis of the December 2009 IMS and 
interviews with program office officials, we found that the IMS is 
resource loaded and the program office is actively assessing the 
loaded resources. 

Best practice: Integrating key activities horizontally and vertically; 
Explanation: The schedule is horizontally integrated, meaning that it 
linked the products and outcomes associated with already sequenced 
activities. These links are commonly referred to as "handoffs" and 
serve to verify that activities are arranged in the right order to 
achieve aggregated products or outcomes. The schedule should also be 
vertically integrated, meaning that traceability exists among varying 
levels of activities and supporting tasks and subtasks. Such mapping 
or alignment among levels enables different groups to work to the same 
master schedule; 
Satisfied? Substantially satisfied; 
GAO analysis: The updated IMS is fully integrated vertically and 
somewhat integrated horizontally. The use of Start No Earlier Than 
constraints for each airline testing phase 1 activity prevents the 
schedule from being able to dynamically calculate the completion date. 
However, as airline testing phase start dates are legally binding 
dates negotiated between the program office and each of the 
approximately 200 airlines, the program office has justified the use 
of these constraints. 

Best practice: Establishing the critical path for key activities; 
Explanation: Using scheduling software, the critical path--the longest 
duration path through the sequenced list of key activities--should be 
identified. The establishment of a program's critical path is 
necessary for examining the effects of any activity slipping along 
this path. Potential problems that might occur along or near the 
critical path should also be identified and reflected in the 
scheduling of the time for high-risk activities; 
Satisfied? Partially satisfied; 
GAO analysis: The updated IMS does not contain a valid critical path 
as calculated by the software because of the Start No Earlier Than 
constraints on each airline testing phase beginning milestone. 
However, as airline testing phase start dates are legally binding 
dates negotiated between the program office and each of the 
approximately 200 airlines, the program office has justified the use 
of these constraints. 

Best practice: Identifying the float time between key activities; 
Explanation: The schedule should identify float time--the time that a 
predecessor activity can slip before the delay affects successor 
activities--so that schedule flexibility can be determined. Generally, 
activities along the critical path typically have the least amount of 
float time. Total float describes the amount of time flexibility an 
activity has without delaying the project completion (if everything 
else goes according to plan). Total float is used to find out which 
activities or paths are crucial to project completion; 
Satisfied? Partially satisfied; 
GAO analysis: The updated IMS schedule does not reflect realistic 
float values because of the high number of Start No Earlier Than 
constraints on the on each airline testing phase beginning milestone. 
However, as airline testing phase start dates are legally binding 
dates negotiated between the program office and each of the 
approximately 200 airlines, the program office has justified the use 
of these constraints. The IMS shows that 159 activities have from 250 
to 300 days of float, suggesting that these activities can slip up to 
300 days and not affect the completion date of the project. The 
updated IMS contains baselined international deployment dates. 

Best practice: Schedule risk analysis should be performed; 
Explanation: A schedule risk analysis should be performed using 
statistical techniques to predict the level of confidence in meeting a 
program's completion date. This analysis focuses not only on critical 
path activities but also on activities near the critical path, since 
they can potentially affect program status; 
Satisfied? Satisfied; 
GAO analysis: A schedule risk analysis was performed by an independent 
contractor and delivered in August 2009. The schedule risk analysis 
made several recommendations regarding the technical aspects of the 
schedule as well as programmatic risk reduction recommendations. Among 
the risk reduction recommendations made were (1) that the amount of 
work required for each task be estimated before resources are assigned 
and that an attempt be made to capture the actual work performed 
throughout the project; (2) that aircraft operator start dates be 
moved earlier when resources are available; (3) that because the 
schedule may be delayed if less than 17 aircraft operators are 
deployed concurrently, the program office should determine the maximum 
number of aircraft operators that can be deployed concurrently; 
and (4) that additional resources be assigned to the testing 
environment. 

Best practice: Distributing reserves to high-risk activities; 
Explanation: The baseline schedule should include a buffer or a 
reserve of extra time. Schedule reserve for contingencies should be 
calculated by performing a schedule risk analysis. Generally, the 
reserve should be applied to high-risk activities, which are typically 
found along the critical path; 
Satisfied? Substantially satisfied; 
GAO analysis: According to TSA's plan of action, the schedule risk 
analysis was completed in August 2009, and program leadership took 
related actions to distribute reserves. 

Source: GAO analysis. 

[End of table] 

[End of section] 

Footnotes: 

[1] The No-Fly and Selectee lists contain the names of individuals 
with known or suspected links to terrorism. These lists are subsets of 
the consolidated terrorist watchlist that is maintained by the Federal 
Bureau of Investigation's Terrorist Screening Center. 

[2] See Pub. L. No. 108-458, § 4012(a), 118 Stat. 3638, 3714-18 (2004) 
(codified at 49 U.S.C. § 44903(j)(2)(C)). 

[3] GAO has performed this work in accordance with statutory mandates, 
beginning in fiscal year 2004 with the Department of Homeland Security 
Appropriations Act, 2004, Pub. L. No. 108-90, § 519, 117 Stat. 1137, 
1155-56 (2003) (establishing the initial mandate that GAO assess the 
Computer-Assisted Passenger Prescreening System II, the precursor to 
Secure Flight, and setting forth the original eight statutory 
conditions related to the development and implementation of the 
prescreening system), and pursuant to the requests of various 
congressional committees. 

[4] See, for example, GAO, Aviation Security: Transportation Security 
Administration Has Strengthened Planning to Guide Investments in Key 
Aviation Security Programs, but More Work Remains, [hyperlink, 
http://www.gao.gov/products/GAO-08-456T] (Washington, D.C.: Feb. 28, 
2008); Aviation Security: Significant Management Challenges May 
Adversely Affect Implementation of the Transportation Security 
Administration's Secure Flight Program, [hyperlink, 
http://www.gao.gov/products/GAO-06-374T] (Washington, D.C.: Feb. 9, 
2006); and Aviation Security: Secure Flight Development and Testing 
Under Way, but Risks Should Be Managed as System Is Further Developed, 
[hyperlink, http://www.gao.gov/products/GAO-05-356] (Washington, D.C.: 
Mar. 28, 2005). 

[5] See Pub. L. No. 108-334, § 522, 118 Stat. 1298, 1319-20 (2004). 
The appropriations acts for each subsequent fiscal year through fiscal 
year 2009 included the same requirement, referring back to the 10 
conditions from fiscal year 2005. See Department of Homeland Security 
Appropriations Act, 2006, Pub. L. No. 109-90, § 518(a), 119 Stat. 
2064, 2085 (2005); Department of Homeland Security Appropriations Act, 
2007, Pub. L. No. 109-295, § 514(a), 120 Stat. 1355, 1379 (2006); 
Consolidated Appropriations Act, 2008, Pub. L. No. 110-161, div. E, § 
513(a), 121 Stat. 1844, 2072 (2007); and Department of Homeland 
Security Appropriations Act, 2009, Pub. L. No. 110-329, div. D, § 
512(a), 122 Stat. 3652, 3682 (2008). 

[6] GAO, Aviation Security: TSA Has Completed Key Activities 
Associated with Implementing Secure Flight, but Additional Actions Are 
Needed to Mitigate Risks, [hyperlink, 
http://www.gao.gov/products/GAO-09-292] (Washington, D.C.: May 13, 
2009). 

[7] See H.R. Rep. No. 111-298, at 80 (2009) (Conf. Rep.). 

[8] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for 
Developing and Managing Capital Program Costs, [hyperlink, 
http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March 2009). 

[9] In general, an integrated master schedule contains all of the 
detailed work and planning activities necessary to support key program 
milestones. 

[10] Specifically, we recommended that TSA update the Secure Flight 
program's life-cycle cost estimate to include all costs, compare the 
updated estimate to an independent cost estimate, align cost estimates 
with the program's schedule, quantify the risks facing the program, 
and determine a level of confidence in meeting the cost estimate. 
Because TSA provided us with its plan of action to address these 
recommendations in April 2009, we did not include the recommendations 
in our May 2009 report. 

[11] In general, an uncertainty analysis provides a level of 
confidence about the program's cost estimate, while a sensitivity 
analysis allows decision makers to understand the impact of changing 
one assumption or cost driver at a time (e.g., the effect of a program 
milestone delay on the total cost of the program). 

[12] At times, an organization may conclude that the remaining budget 
and schedule targets for completing a program are significantly 
insufficient and that the current baseline is no longer valid for 
realistic performance measurement. If so, the program may be 
rebaselined. 

[13] Specifically, we recommended that TSA establish a reliable 
benchmark schedule for the Secure Flight program's remaining 
activities using scheduling best practices, maintain the schedule 
throughout the program's life cycle using proper scheduling methods, 
rely on calculated dates in the schedule rather than imposing target 
dates, add projected resources to the program's schedule to better 
track progress, and periodically assess the risks to the schedule. 
Because TSA provided us with its plan of action to address these 
recommendations in April 2009, we did not include the recommendations 
in our May 2009 report. 

[14] In general, float is the amount of time an activity can slip 
before affecting successor activities. 

[15] The schedule should reflect what resources (e.g., labor, 
material, and overhead) are needed to do the work, whether all 
required resources will be available when needed, and whether any 
funding or time constraints exist. 

[16] The schedule should be horizontally integrated, meaning that it 
links the products and outcomes associated with already sequenced 
activities, and should be vertically integrated, meaning that 
traceability exists among varying levels of activities and supporting 
tasks and subtasks. 

[17] The schedule should include a buffer or a reserve of extra time 
to complete work. Generally, the reserve should be applied to high-
risk activities, which are typically found along the critical path. 

[18] Because of fluctuations in passenger service, the total number of 
covered carriers in the Secure Flight program will vary. For example, 
a new carrier could either start new service or cancel existing 
service. 

[19] Partial assumption involves air carriers that are phasing in 
their flights or provide service to multiple airlines. 

[20] See 73 Fed. Reg. 64,018 (Oct. 28, 2008) (codified at 49 C.F.R. 
pt. 1560). 

[21] [hyperlink, http://www.gao.gov/products/GAO-09-3SP]. 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO’s Web site, 
[hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional 
information. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, dawnr@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: