TSA Modernization:

Use of Sound Program Management and Oversight Practices Is Needed to Avoid Repeating Past Problems

GAO-18-46: Published: Oct 17, 2017. Publicly Released: Oct 17, 2017.

Additional Materials:

Contact:

Carol Harris
(202) 512-4456
harriscc@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

What GAO Found

The Transportation Security Administration's (TSA) new strategy for the Technology Infrastructure Modernization (TIM) program includes using Agile software development, but the program only fully implemented two of six leading practices necessary to ensure successful Agile adoption. Specifically, the Department of Homeland Security (DHS) and TSA leadership fully committed to adopt Agile and TSA provided Agile training. Nonetheless, the program had not defined key roles and responsibilities, prioritized system requirements, or implemented automated capabilities that are essential to ensuring effective adoption of Agile. Until TSA adheres to all leading practices for Agile implementation, the program will be putting at risk its ability to deliver a quality system that strengthens and enhances the sophistication of TSA's security threat assessments and credentialing programs.

TSA and DHS fully implemented one of the key practices for overseeing the TIM program, by establishing a process for ensuring corrective actions are identified and tracked. However, TSA and DHS did not fully implement the remaining three key practices, which impede the effectiveness of their oversight. Specifically,

  • TSA and DHS documented selected policies and procedures for governance and oversight of the TIM program, but they did not develop or finalize other key oversight and governance documents. For example, TSA officials developed a risk management plan tailored for Agile; however, they did not update the TIM system life-cycle plan to reflect the Agile governance framework they were using.
  • The TIM program management office conducted frequent performance reviews, but did not establish thresholds or targets for oversight bodies to use to ensure that the program was meeting acceptable levels of performance. In addition, department-level oversight bodies have focused on reviewing selected program life-cycle metrics for the TIM program; however, they did not measure the program against the rebaselined cost, or important Agile release-level metrics.
  • TIM's reported performance data were not always complete and accurate. For example, program officials reported that they were testing every line of code, even though they were unable to confirm that they were actually doing so, thus calling into question the accuracy of the data reported.

These gaps in oversight and governance of the TIM program were due to, among other things, TSA officials not updating key program management documentation and DHS leadership not obtaining consensus on needed oversight and governance changes related to Agile programs. Given that TIM is a historically troubled program and is at least 6 months behind its rebaselined schedule, it is especially concerning that TSA and DHS have not fully implemented oversight and governance practices for this program. Until TSA and DHS fully implement these practices to ensure the TIM program meets its cost, schedule, and performance targets, the program is at risk of repeating past mistakes and not delivering the capabilities that were initiated 9 years ago to protect the nation's transportation infrastructure.

Why GAO Did This Study

TSA conducts security threat assessment screening and credentialing activities for millions of workers and travelers in the maritime, surface, and aviation transportation industries that are seeking access to transportation systems. In 2008, TSA initiated the TIM program to enhance the sophistication of its security threat assessments and to improve the capacity of its supporting systems. However, the program experienced significant cost and schedule overruns, and performance issues, and was suspended in January 2015 while TSA established a new strategy. The program was rebaselined in September 2016 and is estimated to cost approximately $1.27 billion and be fully operational by 2021 (about $639 million more and 6 years later than originally planned).

GAO was asked to review the TIM program's new strategy. This report determined, among other things, the extent to which (1) TSA implemented selected key practices for transitioning to Agile software development for the program; and (2) TSA and DHS are effectively overseeing the program's cost, schedule, and performance. GAO compared program documentation to key practices identified by the Software Engineering Institute and the Office of Management and Budget, as being critical to transitioning to Agile and for overseeing and governing programs.

What GAO Recommends

GAO is making 14 recommendations, including that DHS should prioritize requirements and obtain leadership consensus on oversight and governance changes. DHS concurred with all 14 recommendations.

For more information, contact Carol Harris at (202) 512-4456 or harriscc@gao.gov.

Recommendations for Executive Action

  1. Status: Open

    Comments: TSA concurred with this recommendation. In November 2018, the agency provided updated documentation on the TIM program's plans for transitioning from the current state to the final TIM state. We are in the process of reviewing this information.

    Recommendation: The TSA Administrator should ensure that the TIM program management office establishes and implements specific time frames for determining key strategic implementation details, including how the program will transition from the current state to the final TIM state. (Recommendation 1)

    Agency Affected: Department of Homeland Security

  2. Status: Open

    Comments: TSA concurred with this recommendation. In November 2018, the agency provided updated documentation on the TIM program's efforts to estimate how long it will take to deliver capabilities and complete the program's milestones. We are in the process of reviewing this information.

    Recommendation: The TSA Administrator should ensure that the TIM program management office establishes a schedule that provides planned completion dates based on realistic estimates of how long it will take to deliver capabilities. (Recommendation 2)

    Agency Affected: Department of Homeland Security

  3. Status: Open

    Comments: TSA concurred with this recommendation. In November 2018, TIM program officials stated that, while they had not yet completed the actions identified in the organizational change management strategy, such as filling a communications lead position, they had improved coordination and communication with stakeholders through TSA's security threat assessment portfolio management process. We are in the process of reviewing the documentation that was provided by TSA.

    Recommendation: The TSA Administrator should ensure that the TIM program management office establishes new time frames for implementing the actions identified in the organizational change management strategy and effectively executes against these time frames. (Recommendation 3)

    Agency Affected: Department of Homeland Security

  4. Status: Closed - Implemented

    Comments: In response to our recommendation, TSA further clarified and documented the roles and responsibilities of key TIM program stakeholders in prioritizing and approving the program's Agile software development work. As a result, TSA has better assurance that the TIM program's key stakeholders can effectively establish priorities, approve user stories, and decide whether completed work meets the acceptance criteria for the TIM program.

    Recommendation: The TSA Administrator should ensure that the TIM program management office defines and documents the roles and responsibilities among product owners, the solution team, and any other relevant stakeholders for prioritizing and approving Agile software development work. (Recommendation 4)

    Agency Affected: Department of Homeland Security

  5. Status: Open

    Comments: TSA concurred with this recommendation. In November 2018, the agency provided documentation on the TIM program's backlog of prioritized requirements (features and user stories). We are in the process of reviewing this information.

    Recommendation: The TSA Administrator should ensure that the TIM program management office establishes specific prioritization levels for current and future features and user stories. (Recommendation 5)

    Agency Affected: Department of Homeland Security

  6. Status: Open

    Comments: TSA concurred with this recommendation. In November 2018, the agency provided documentation to show that the TIM program had implemented all of its automated Agile management testing and deployment tools. We are in the process of reviewing this information.

    Recommendation: The TSA Administrator should ensure that the TIM program management office implements automated Agile management testing and deployment tools, as soon as possible. (Recommendation 6)

    Agency Affected: Department of Homeland Security

  7. Status: Open

    Comments: TSA concurred with this recommendation. In November 2018, the agency provided a revised version of the TIM program's Systems Engineering Life Cycle Tailoring Plan. We are in the process of reviewing this information.

    Recommendation: The TSA Administrator should ensure that the TIM program management office updates the Systems Engineering Life Cycle Tailoring Plan to reflect the current governance framework and milestone review processes. (Recommendation 7)

    Agency Affected: Department of Homeland Security

  8. Status: Open

    Comments: TSA concurred with this recommendation. In November 2018, the agency provided updated documentation on the TIM program's performance monitoring and reporting. We are in the process of reviewing this information.

    Recommendation: The TSA Administrator should ensure that the TIM program management office establishes thresholds or targets for acceptable performance-levels. (Recommendation 8)

    Agency Affected: Department of Homeland Security

  9. Status: Open

    Comments: TSA concurred with this recommendation. In November 2018, the agency provided updated documentation on the TIM program's efforts to report on its planned and actual costs as a result of the ongoing Agile development activities. We are in the process of reviewing this information.

    Recommendation: The TSA Administrator should ensure that the TIM program management office begins collecting and reporting on Agile-related cost metrics. (Recommendation 9)

    Agency Affected: Department of Homeland Security

  10. Status: Closed - Implemented

    Comments: In response to our recommendation, TSA provided several performance reports, ranging from June through November 2017, that demonstrated consistent reporting of the TIM program's velocity. As a result, TSA has better assurance that it is using accurate program velocity data to more reliably forecast the TIM program's ability to meet its cost, schedule, and performance targets.

    Recommendation: The TSA Administrator should ensure that the TIM program management office ensures that program velocity is measured and reported consistently. (Recommendation 10)

    Agency Affected: Department of Homeland Security

  11. Status: Open

    Comments: TSA concurred with this recommendation. In November 2018, TSA provided documentation on its methods for automatically measuring unit test coverage. We are in the process of reviewing this information.

    Recommendation: The TSA Administrator should ensure that the TIM program management office ensures that unit test coverage for software releases is measured and reported accurately. (Recommendation 11)

    Agency Affected: Department of Homeland Security

  12. Status: Open

    Comments: DHS concurred with this recommendation. In August 2018, DHS OCIO officials stated that DHS leadership had reached consensus on oversight and governance changes related to the frequency of reviewing Agile programs. Officials said they intend to update DHS policies to reflect that Agile programs should be reviewed by the ARB every 6 months, but they did not have a timeframe for completion. Once the updated policies are available, we will review it.

    Recommendation: The Secretary of Homeland Security should direct the Under Secretary for Management to ensure that appropriate DHS leadership reaches consensus on needed oversight and governance changes related to the frequency of reviewing Agile programs, and then documents and implements associated changes. (Recommendation 12)

    Agency Affected: Department of Homeland Security

  13. Status: Closed - Implemented

    Comments: In October 2017, DHS provided its completed guidance which included recommended practices for collecting and reporting on agile performance metrics, as well as a set of core agile performance metrics that programs should report to the Department. As a result, DHS has better assurance that agile development programs will report informative performance metrics to oversight entities so that they can ensure the programs are effectively delivering their intended capabilities and outcomes.

    Recommendation: The Secretary of Homeland Security should direct the Under Secretary for Management to ensure that the Office of the Chief Technology Officer completes guidance for Agile programs to use for collecting and reporting on performance metrics. (Recommendation 13)

    Agency Affected: Department of Homeland Security

  14. Status: Open

    Comments: DHS concurred with this recommendation. In November 2018, the agency provided documentation on recent DHS efforts to review the TIM program's Agile performance and cost metrics and make informed management oversight decisions on the direction of the program. We are in the process of reviewing this information.

    Recommendation: The Secretary of Homeland Security should direct the Under Secretary for Management to ensure that DHS-level oversight bodies review key Agile performance and cost metrics for the TIM program and use them to inform management oversight decisions. (Recommendation 14)

    Agency Affected: Department of Homeland Security

 

Explore the full database of GAO's Open Recommendations »

Mar 18, 2019

Mar 14, 2019

Mar 7, 2019

Feb 27, 2019

Feb 22, 2019

Feb 21, 2019

Feb 7, 2019

Feb 5, 2019

Looking for more? Browse all our products here