Fast Facts

Many of the Department of Homeland Security’s IT acquisitions have taken longer than planned or failed to deliver desired results.

In April 2016, DHS started transitioning to Agile software development to help improve its IT acquisitions. Agile focuses on collaborative processes and workflows to quickly and frequently deliver working software.

DHS has made significant progress implementing leading practices during this transition but needs to take additional steps. For example, it needs to ensure all staff are trained in this new approach.

We recommended that DHS fully implement leading practices in its transition to Agile software development.

Homeland Security building

Homeland Security building

Skip to Highlights
Highlights

What GAO Found

The Department of Homeland Security (DHS) has taken steps to implement selected leading practices in its transition from waterfall, an approach that historically delivered useable software years after program initiation, to Agile software development, which is focused on incremental and rapid delivery of working software in small segments. As shown below, this quick, iterative approach is to deliver results faster and collect user feedback continuously.

Comparison of Agile and Waterfall Methods for Developing Software

Comparison of Agile and Waterfall Methods for Developing Software

DHS has fully addressed one of three leading practice areas for organization change management and partially addressed the other two. Collectively, these practices advise an organization to plan for, implement, and measure the impact when undertaking a significant change. The department has fully defined plans for transitioning to Agile development. DHS has partially addressed implementation—the department completed 134 activities but deferred roughly 34 percent of planned activities to a later date. These deferred activities are in progress or have not been started. With respect to the third practice, DHS clarified expected outcomes for the transition, such as reduced risk of large, expensive IT failures. However, these outcomes are not tied to target measures. Without these, DHS will not know if the transition is achieving its desired results.

DHS has also addressed four of the nine leading practices for adopting Agile software development. For example, the department has modified its acquisition policies to support Agile development methods. However, it needs to take additional steps to, among other things, ensure all staff are appropriately trained and establish expectations for tracking software code quality. By fully addressing leading practices, DHS can reduce the risk of continued problems in developing and acquiring current, as well as, future IT systems.

Why GAO Did This Study

Many of DHS's major IT acquisition programs have taken longer than expected to develop or failed to deliver the desired value. In April 2016, to help improve the department's IT acquisition and management, DHS identified Agile software development as the preferred approach for all of its IT programs and projects.

GAO was asked to examine DHS's adoption of Agile software development. The objective of this review was to assess the extent to which DHS has addressed selected leading practices for its transition to the use of Agile software development.

GAO identified leading practices for planning, implementing, and measuring organizational change that apply to DHS's transition to Agile through its review of guidance published by the Project Management Institute and GAO. GAO also reviewed work it performed to develop leading practices for Agile software development adoption. GAO analyzed DHS documentation, such as policies, guidance, plans, and working group artifacts and assessed them against the selected leading practices. GAO also reviewed the implementation of selected practices within individual IT projects. Finally, GAO interviewed DHS officials to discuss any practices that were not fully implemented.

Skip to Recommendations

Recommendations

GAO is making 10 recommendations to DHS to implement selected leading practices for its transition to Agile software development. DHS agreed with GAO's recommendations and described actions taken and planned to address them.

Recommendations for Executive Action

Agency Affected Recommendation Status
Department of Homeland Security 1. The Secretary should ensure that the Director of Strategic Technology Management (STM), in collaboration with other members of the Information Technology Program Management Center of Excellence (ITPM COE), identifies the skills and resources needed to complete the work intended for the upcoming fiscal year, including the availability of supplementary staff, such as subject matter experts. (Recommendation 1)
Open
In its 180-day letter provided in response to our report, DHS stated that during the first quarter of each fiscal year, DHS Office of the Chief Information Officer (OCIO) staff host a planning session to review ongoing and upcoming tasks related to 18 action plans, while the Information Technology Program Management Center of Excellence (ITPM COE) reviews its charter annually to ensure it is properly aligned to the ITPM COE's scope and objectives. During these planning sessions, criteria for completing each of the outstanding tasks is developed and potential leads are identified. Following this planning, DHS OCIO works with members of the ITPM COE to ensure personnel who have the necessary skills and subject matter expertise address each task. As the vehicle for completion of these tasks, ITPM COE defines workloads within existing resources. DHS subsequently provided presentation slides from the planning session held for fiscal year 2020 (FY20) and a recap session to revisit accomplishments achieved in FY20. These slides address the fiscal year core strategies for each office that makes up the ITPM COE, any new tasks, and demonstrates that the ITPM COE completed some of the tasks, while the majority were still in progress or unachieved. While DHS officials stated that the skills and resources needed to complete the work are discussed during the planning session to inform the intended workload for the upcoming fiscal year, the department has not demonstrated that such discussions have led the ITPM COE to be more successful in completing the tasks intended for that fiscal year. Until the department can demonstrate an ability to complete the work intended for the upcoming fiscal year, this recommendation will remain open.
Department of Homeland Security 2. The Secretary should ensure that the Executive Steering Committee overseeing the activities of the ITPM COE establishes target measures for the department's desired outcomes of its transition to Agile development. (Recommendation 2)
Open
In its 180-day letter provided in response to our report, the Department of Homeland Security (DHS) stated that DHS's Office of the Chief Technology Officer (OCTO) Directorate (CTOD) is updating the Agile Core Metrics to include the data collected through the Agile Software Delivery Maturity Model, in order to provide additional details around each of the five intended outcomes cited for the transition to Agile. Furthermore, the Agile Core Metrics are scheduled to be published to DHS's Investment Evaluation, Submission, and Tracking (INVEST) system in the first quarter of FY 2021. In addition, DHS stated that the Agile Software Delivery Maturity Model was administered to Major IT programs from July to September 2020 to assess their Agile maturity. Although the Agile Software Delivery Maturity Model was not used to assess Level 3 programs, CTOD plans to work with the Solutions Development Directorate to establish processes and procedures to improve Systems Engineering Life Cycle (SELC) and Agile Alignment at DHS Headquarters. DHS expects to complete the activities necessary to implement this recommendation by June 30, 2021.
Department of Homeland Security 3. The Secretary should ensure that the DHS Chief Information Officer (CIO) defines a process and associated set of controls to ensure that Agile programs and projects are reporting a set of core required performance metrics for monitoring and measuring Agile adoption. (Recommendation 3)
Open
In a 180-day letter, provided in response to our report, DHS stated that the DHS OCIO's Agile Core Metrics, Program Health Assessments (PHA), and Agile Software Delivery Maturity Model (SDMM) collectively define a process and are used as controls that ensure Agile programs and projects are reporting a set of core required performance metrics for monitoring and measuring Agile adoption. DHS provided the Program Assessment Process Guide, approved by the CIO and reviewed and signed by the Chief Technology Officer (CTO), on September 29, 2020, formalizing the PHA process. The department also provided a narrative explanation for the Quality Assurance (QA) activities applied to the PHA process and intended to provide the SDMM used to monitor and measure Major IT Programs' Agile adoption. However, DHS did not provide evidence that these QA procedures were formalized and documented, thereby requiring compliance, or provide the SDMM as intended. Moreover, as stipulated in its responses to other recommendations, the department has not finalized and published the Agile Core Metrics, scheduled to take place in the first quarter of 2021. Until these actions occur, the recommendation will remain open.
Department of Homeland Security 4. The Secretary should ensure that the ITPM COE, in coordination with the CIO, begins measuring results associated with the transition to Agile and the success of the transition based on its impact on the department. (Recommendation 4)
Open
In its 180-day letter provided in response to our report, DHS stated that the Agile Core Metrics are scheduled to be published to DHS's INVEST system in the first quarter of FY 2021. In addition, the department stated that the DHS Office of the Chief Information Officer (OCIO) began using its Agile Software Delivery Maturity Model to monitor and measure Agile adoption in July 2020. Additionally, DHS stated that the Program Health Assessment (PHA) Process Guide was reviewed by the CIO, and reviewed and signed by the CTO on September 29, 2020. DHS added that the PHAs are ongoing, at this time, using the new process, and anticipates completing actions necessary to implement this recommendation by June 30, 2021.
Department of Homeland Security 5. The Secretary should ensure that the CIO, in collaboration with the Chief Procurement Officer, through the Homeland Security Acquisition Institute, establish Agile training requirements for senior stakeholders. (Recommendation 5)
Open
In its 180-day letter provided in response to our report, DHS stated that on September 1, 2020, DHS OCIO met with members from the the Homeland Security Acquisition Institute (HSAI) and the Information Technology Program Management Center of Excellence (ITPM COE) to identify Agile training requirements for software development. During that time, DHS OCIO conducted research and identified several training opportunities that can be leveraged across the Department. DHS OCIO is also researching the possibility of incorporating Agile Software Development training in the current ITPM certification program that is available across the Department. Finally, DHS OCIO identified Development/Security/Operations (DevSecOps) training that is being made available to DHS employees via the Performance and Learning Management System. DHS OCIO will continue working on defining an Agile training Strategy and anticipates completing the actions necessary to implement this recommendation by September 30, 2021.
Department of Homeland Security 6. The Secretary should ensure that the Chief Human Capital Officer, in collaboration with the CIO, consider modifications to the current employee recognition and performance management governance to ensure that teamwork and team performance of Agile programs and projects are incentivized. (Recommendation 6)
Open
In its 180-day letter provided in response to our report, DHS stated that on on January 28, 2020, the OCIO Employee Awards Program was launched. DHS stated that this program added guidance to incentivize teamwork, team performance, and IT programs (including Agile). OCIO anticipates completing the actions necessary to implement this recommendation by March 31, 2021.
Department of Homeland Security 7. The Secretary should ensure that the CIO, in collaboration with the Chief Procurement Officer, through the Homeland Security Acquisition Institute, establish Agile training requirements for staff outside of the acquisition workforce but assigned to Agile programs. (Recommendation 7)
Open
In its 180-day letter provided in response to our report, DHS stated that DHS OCIO met with members of HSAI, ITPM COE and the Agile Center of Excellence (COE), and continues to identify Agile training requirements for staff outside of the acquisition workforce but assigned to Agile programs. DHS stated that OCIO completed an assessment of Project Management courses on September 1, 2020, at HSAI and identified the gaps in Agile coverage. DHS stated that its OCIO will continue to develop a strategy to address those gaps and share the training resources that are available across the Department. OCIO anticipates completing the actions necessary to implement this recommendation by September 30, 2021.
Department of Homeland Security 8. The Secretary should ensure that the CIO, upon establishing a set of core performance metrics, tracks and monitors the pace of Agile team development. (Recommendation 8)
Open
In its 180-day letter provided in response to our report, DHS stated that the Agile Core Metrics are scheduled to be published to INVEST in the first quarter of FY 2021. DHS added that OCIO will commence gathering and analyzing the data submitted to INVEST in order to track and monitor the pace of Agile team development and anticipates completing the actions necessary to implement this recommendation by June 30, 2021.
Department of Homeland Security 9. The Secretary should ensure that the CIO, in collaboration with the Executive Director of the Office of Program Accountability and Risk Management (PARM), update or develop new guidance on Agile methodologies to describe how Agile teams can estimate the relative complexity of user stories. (Recommendation 9)
Open
In its 180-day letter provided in response to our report, DHS stated that DHS OCIO is on track to publish guidance on how Agile teams can estimate relative complexity of user stories in the updated Agile Guidebook. In addition, in December 2020, DHS updated its Agile Guidebook. The guidebook now includes a definition for relative complexity as it pertains to user stories. However, the guidebook still does not provide techniques on how Agile teams can estimate complexity of user stories and measure program performance. As of March 2021, DHS had not provided additional documentation that described such techniques.
Department of Homeland Security 10. The Secretary should ensure that the CIO, upon establishing a set of core performance metrics, sets expectations for automated testing and code quality, and tracks and monitors against those expectations. (Recommendation 10)
Open
In its 180-day letter provided in response to our report, DHS stated that the Agile Core Metrics are scheduled to be published to INVEST in the first quarter of FY 2021. DHS stated that OCIO will commence gathering and analyzing the data submitted to INVEST in order to track and monitor the pace of Agile team development, and anticipates completing the actions necessary to implement this recommendation by June 30, 2021.

Full Report

GAO Contacts