Immigration Benefits System:

U.S. Citizenship and Immigration Services Can Improve Program Management

GAO-16-467: Published: Jul 7, 2016. Publicly Released: Jul 15, 2016.

Additional Materials:

Contact:

Carol C. Harris
(202) 512-4456
chac@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

What GAO Found

U.S. Citizenship and Immigration Services (USCIS) created a reliable updated estimate to project the Transformation Program's cost, but has experienced program management challenges. In particular, the program's cost estimate was well-documented and substantially comprehensive, accurate, and credible. However, among other things, software development and systems integration and testing for USCIS's Electronic Immigration System (USCIS ELIS) have not consistently been managed in line with the program's policies and guidance or with leading practices.

Regarding software development, the Transformation Program has produced some software increments, but is not consistently following its own guidance and leading practices. The software development model (Agile) adopted by the USCIS Transformation Program in 2012 includes practices aimed at continuous, incremental release of segments of software. Important practices for Agile defined in program policies, guidance, and leading practices include ensuring that the software meets expectations prior to being deployed, teams adhere to development principles, and development outcomes are defined. The table below lists the program's status in addressing eight key Agile development practices. For example, the program has committed to a specific framework for software development, referred to as Scrum, but has deviated from the underlying practices and principles of this framework.

Table: Implementation of Key Agile Practices for USCIS ELIS

Practice

Rating

Completing planning for software releases prior to initiating development and ensuring software meets business expectations prior to deployment

Adhering to the principles of the framework adopted for implementing Agile software development

Defining and consistently executing appropriate roles and responsibilities for individuals responsible for development activities

Identifying users of the system and involving them in release planning activities

Writing user stories that identify user roles, include estimates of complexity, take no longer than one sprint to complete, and describe business value

Prioritizing user stories to maximize the value of each development cycle

Setting outcomes for Agile software development

Monitoring and reporting on program performance through the collection of reliable metrics

Source: GAO analysis of USCIS documentation. I GAO-16-467.

Note: ● yes ◐ partial ○ noc

The Transformation Program has established an environment that allows for effective systems integration and testing and has planned for and performed some system testing. However, the program needs to improve its approach to system testing to help ensure that USCIS ELIS meets its intended goals and is consistent with agency guidance and leading practices. Among other things, the program needs to improve testing of the software code that comprises USCIS ELIS and ensure its approaches to interoperability and end user testing, respectively, meet leading practices. Collectively, these limitations have contributed to issues with USCIS ELIS after new software is released into production.

Why GAO Did This Study

Each year, USCIS processes millions of applications for persons seeking to study, work, visit, or live in the United States, and for persons seeking to become a U.S. citizen. In 2006, USCIS began the Transformation Program to enable electronic adjudication and case management tools that would allow users to apply and track their applications online. In 2012, to address performance concerns, USCIS changed its acquisition strategy to improve system development.

In May 2015, GAO reported that USCIS expected the program to cost up to $3.1 billion and be fully operational by March 2019. This includes more than $475 million that was invested in the initial version of the program's key case management component, USCIS ELIS, which has since been decommissioned.

This report evaluates the extent to which the program is using information technology program management leading practices. To perform this work, GAO identified agency policy and guidance and leading practices in, among other things, cost estimation, Agile software development, and systems integration and testing, and compared these with practices being used by the program.

What GAO Recommends

GAO is making 12 recommendations to improve Transformation Program management, including ensuring alignment among policy, guidance, and leading practices in areas such as Agile software development and systems integration and testing. DHS concurred with the recommendations.

For more information, contact Carol C. Harris at (202) 512-4456 or chac@gao.gov.

Recommendations for Executive Action

  1. Status: Open

    Comments: As of July 2017, the U.S. Citizenship and Immigration Services (USCIS) within the Department of Homeland Security (DHS) had taken steps to address this recommendation. In particular, in June 2017, USCIS provided an updated policy, dated April 2017, governing planning and deploying software releases. USCIS also demonstrated partial compliance with that policy. For example, it provided some release planning review documentation for recent releases that are required by the updated policy, including readiness review memos for releases 7.2 and 8.1. However, USCIS did not demonstrate that the program responsible for developing the USCIS Electronic Immigration System (USCIS ELIS) was consistently following its updated policy. For example, USCIS did not demonstrate that the program was completing all planning activities prior to initiating development, as called for in its updated policy. Moreover, the agency did not demonstrate compliance with its previous policy for all software releases planned and deployed since our July 2016 report. We will continue to work with USCIS to monitor actions the agency is taking to address this recommendation.

    Recommendation: To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of the Department of Homeland Security (DHS) should direct the Director of USCIS to direct the USCIS Chief Information Officer (CIO), in coordination with the DHS CIO and the Chief of the Office of Transformation Coordination (OTC), to review and update, as needed, existing policies and guidance and consider additional controls to complete planning for software releases prior to initiating development and ensure software meets business expectations prior to deployment.

    Agency Affected: Department of Homeland Security

  2. Status: Open

    Comments: As of July 2017, USCIS had taken steps to address this recommendation. For example, in May 2017, USCIS provided updated policy governing the development of software releases, dated April 2017, along with release planning artifacts specific to USCIS ELIS. The updated policy included an appendix devoted to generally accepted agency practices and applying Agile principles in the agency. However, USCIS had not clearly indicated if USCIS ELIS was to implement the practices described in the policy. For example, the updated policy did not require program compliance with the generally accepted agency practices. Moreover, supporting artifacts from the release planning process did not always define a commitment to a particular development methodology or set of development practices. For example, the team process agreements, which describe how members of individual teams will work with each other, did not indicate if developers were to adhere to the practices described in updated USCIS policy. We will continue to work with USCIS to obtain additional documentation about actions it is taking to address this recommendation.

    Recommendation: To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to consistently implement the principles of the framework adopted for Agile software development.

    Agency Affected: Department of Homeland Security

  3. Status: Open

    Comments: As of July 2017, USCIS had taken steps to address this recommendation. For example, in June 2017, USCIS provided updated policy, dated April 2017, governing the development of software releases and release planning artifacts. The updated policy and release documentation defined some roles and responsibilities that were previously only described by USCIS in its informal November 2014 management model, such as the authority and responsibility of a product owner. However, program documentation and policy did not define all of the roles and responsibilities. For example, program documentation and policy did not define the roles and responsibilities of a facilitator, or Scrum Master, which is a position identified in leading practices for software development using Scrum, the development methodology previously identified by the program. In addition, USCIS did not demonstrate that it had defined and committed to an updated development methodology for software releases. Such a defined methodology will impact expectations for the roles and responsibilities in software development. Without such a defined methodology or approach to Agile software development, it is not clear if roles and responsibilities defined by previously documented approach to Agile software development are still applicable for the current development approach. Moreover, documentation associated with program releases and updated policy did not define all of the roles and responsibilities for positions described by USCIS in its May 2017 written response to GAO. We will continue to work with USCIS to obtain additional documentation about actions it is taking to address this recommendation.

    Recommendation: To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to define and consistently execute appropriate roles and responsibilities for individuals responsible for development activities consistent with its selected development framework.

    Agency Affected: Department of Homeland Security

  4. Status: Open

    Comments: As of July 2017, DHS and USCIS had not provided information demonstrating that the department has addressed this recommendation. In October 2016, DHS provided a written response stating that the USCIS Office of Information Technology and Office of Transformation Coordination were working closely with the various USCIS directorates to obtain and integrate feedback through regular review sessions with the end users and through additional end user testing. However, as of July 2017, DHS and USCIS have not provided new information about the status of this recommendation.

    Recommendation: To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to identify all system users and involve them in release planning activities.

    Agency Affected: Department of Homeland Security

  5. Status: Open

    Comments: As of July 2017, USCIS had provided GAO with documentation intended to demonstrate that the agency had taken steps to address this recommendation. For example, in May 2017, USCIS provided updated policy governing the development of software releases along with release planning artifacts specific to USCIS ELIS and an Independent Verification and Validation assessment. The agency also provided a series of backlogs that captured user stories for some software releases. In addition, the Independent Verification and Validation assessment indicated that the program was tracking user story quality as part of assessing whether value was continuously discovered and aligned to the mission. However, the assessment report provided to GAO indicated a negative trend for this outcome. Moreover, USCIS policy no longer set expectations regarding user story development. In addition, supporting artifacts from the release planning process did not always define a commitment to a particular development methodology, which is turn impacts the expectations for writing user stories. Finally, backlogs provided by USCIS did not cover all releases in development since our July 2016 report and did not include enough detail to assess all aspects of the user story process (e.g., story size and user involvement). We will continue to work with USCIS to obtain additional documentation about actions it is taking to address this recommendation.

    Recommendation: To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to write user stories that identify user roles, include estimates of complexity, take no longer than one sprint to complete, and describe business value.

    Agency Affected: Department of Homeland Security

  6. Status: Open

    Comments: As of July 2017, USCIS had taken steps to address this recommendation. For example, in April 2017, USCIS issued updated policy governing software development at the agency. The updated policy included an appendix devoted to generally accepted agency practices and applying Agile principles in the agency. This appendix also included a set of ten outcomes associated with using Agile practices at USCIS. For example, outcomes included that value is continuously discovered and aligned to the mission. However, the updated policy did not require program compliance with the practices and principles described in the appendix. Moreover, the agency did not demonstrate that USCIS ELIS had committed to achieving a specific set of outcomes for Agile software development, such as the outcomes described in the USCIS policy. We will continue to work with USCIS to obtain additional documentation about actions it is taking to address this recommendation.

    Recommendation: To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to establish outcomes for Agile software development.

    Agency Affected: Department of Homeland Security

  7. Status: Open

    Comments: As of July 2017, USCIS had taken steps to address this recommendation. For example, in May 2017, USCIS provided updated policy governing the development of software that called for teams to prepare an Operations Monitoring Plan or dashboard showing the practices, tools, and measures that will monitor applications in production. The agency also provided a series of documents from internal systems and processes intended to monitor performance, such as a product dashboard for analyzing code quality (i.e., SonarQube) and a report from its Independent Verification and Validation team. However, the program was undergoing a re-baseline and had yet to document updated cost, schedule, and performance expectations against which to monitor. Moreover, the agency did not demonstrate that other metrics, such as customer satisfaction and team velocity, were being reliably collected. We will continue to work with USCIS to obtain additional documentation about actions it is taking to address this recommendation.

    Recommendation: To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to monitor program performance and report to appropriate entities through the collection of reliable metrics.

    Agency Affected: Department of Homeland Security

  8. Status: Open

    Comments: As of July 2017, USCIS had taken steps to address this recommendation. For example, in May 2017, USCIS provided artifacts from internal systems in place to monitor software development performance. These metrics monitored aspects of testing, such as code quality and code coverage. However, the program did not provide an updated Test and Evaluation Master Plan, which is a document it will produce as part of its ongoing effort to re-baseline. A Test and Evaluation Master Plan sets the testing expectations for the program as agreed upon with its stakeholders in DHS and USCIS. The updated plan will provide a basis for further evaluation of the steps DHS and USCIS have taken to address this recommendation. Moreover, the agency did not demonstrate that functional acceptance tests were being conducted in accordance with stated program goals. For example, the agency did not provide acceptance criteria or the associated tests demonstrating that user stories passed the defined acceptance criteria. We will continue to work with USCIS to obtain additional documentation about actions it is taking to address this recommendation.

    Recommendation: To help manage the USCIS ELIS system, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update existing policies and guidance and consider additional controls to conduct unit and integration, and functional acceptance tests, and code inspection consistent with stated program goals.

    Agency Affected: Department of Homeland Security

  9. Status: Open

    Comments: As of July 2017, DHS and USCIS had not provided information demonstrating that they had addressed this recommendation. In October 2016, DHS provided a written response indicating that an internal process for revisiting the USCIS ELIS Test and Evaluation Master Plan had been initiated, with participation from all relevant stakeholder groups. A Test and Evaluation Master Plan sets the testing expectations for the program as agreed upon with its stakeholders in DHS and USCIS. The updated plan will provide a basis for further evaluation of the steps DHS and USCIS have taken to address this recommendation. The letter also stated that USCIS had begun to work on a policy for new interoperability test procedures. Moreover, the letter added that end user testing is a continuing activity, including providing feedback of observed issues into the development queue, with the slow launch of the naturalization capabilities in USCIS ELIS being a model. However, as of July 2017, DHS and USCIS had not provided new information about the status of this recommendation. We will continue to work with DHS and USCIS to obtain additional documentation about actions they are taking to address this recommendation.

    Recommendation: To help manage the USCIS ELIS system, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update existing policies and guidance and consider additional controls to develop complete test plans and cases for interoperability and end user testing, as defined in the USCIS Transformation Program Test and Evaluation Master Plan, and document the results.

    Agency Affected: Department of Homeland Security

  10. Status: Closed - Implemented

    Comments: As of March 2017, USCIS has demonstrated that it has taken sufficient steps to close this recommendation. For example, USCIS demonstrated that it has provided training to both contracting officers (COs) and contracting officer representatives (CORs) to further clarify the intention of a Quality Assurance Surveillance Plan, when such a plan is expected, and what such a plan should document. This training included a discussion of performance measures. Further, the program office demonstrated that it included missing performance criteria (e.g. productivity, standards adherence, and test quality and test coverage) in its monthly scorecard and is now assessing Flexible Agile Development Services (FADS) contractors against all performance criteria identified in the contract. Also, in January 2017, USCIS provided two Quality Assurance Surveillance Plans for its new FADS contracts, referred to as FADS II, signed by the COR. These surveillance plans clearly defined measures against which to analyze differences between services expected and those delivered. In addition to defining performance measures and expectations for all of the performance criteria defined in the contract, the surveillance plans also set the expectation that all contract deliverables will be delivered on time. USCIS further articulated in the Quality Assurance Surveillance Plans the ratings that will be used to evaluate the extent to which contractor deliverables meet government expectations and the process for performing those evaluations. In addition, the plans included templates for performance reports and contract discrepancy reports that are to be completed by the COR. In July 2016, we had reported that these two items were not being properly maintained in the COR file. Including the templates will assist USCIS in improving the consistency and documentation of its monitoring of contractor performance. Finally, in March 2017, USCIS provided the signed FADS II contracts along with the associated performance work statements, confirming the contract awards and the applicability of the Quality Assurance Surveillance Plans.

    Recommendation: To help improve oversight of selected Transformation Program contracts, the Secretary of DHS should direct the Director of USCIS to direct the Chief of the USCIS Contracting Office, in coordination with the appropriate contracting officer, to consider inconsistencies between policy and leading practices in contract administration and, as needed, institute controls to clearly define measures against which to analyze differences between services expected and those delivered.

    Agency Affected: Department of Homeland Security

  11. Status: Closed - Implemented

    Comments: As of April 2017, USCIS has demonstrated that it has taken sufficient steps to close this recommendation. For example, in March 2017, USCIS provided evidence that it completed an internal review of all USCIS Transformation Program contracting officer representatives (COR) files. Based on our review of the inspection results, the additional training for contracting officer representatives (COR) and contracting officers on the proper maintenance of COR files, and the COR checklist previously provided by USCIS (see recommendation 10), we determined that USCIS had taken sufficient steps to close this recommendation. These actions should help to ensure that CORs maintain complete contract files.

    Recommendation: To help improve oversight of selected Transformation Program contracts, the Secretary of DHS should direct the Director of USCIS to direct the Chief of the USCIS Contracting Office, in coordination with the appropriate contracting officer, to consider inconsistencies between policy and leading practices in contract administration and, as needed, institute controls to ensure contracting officer's representatives are maintaining complete contract files.

    Agency Affected: Department of Homeland Security

  12. Status: Closed - Implemented

    Comments: As of March 2017, USCIS has demonstrated that it has taken sufficient steps to close this recommendation. For example, in February 2017, USCIS provided evidence that it completed an internal review of all 46 contracts that it identified as performance based acquisitions. Based on our review of the inspection results, the additional Quality Assurance Surveillance Plan training for contracting officer representatives (COR) and contracting officers, and the COR checklist previously provided by USCIS (see recommendation 10), we determined that USCIS had taken sufficient steps to close this recommendation. These actions should help to ensure that quality assurance surveillance plans are developed when appropriate.

    Recommendation: To help improve oversight of selected Transformation Program contracts, the Secretary of DHS should direct the Director of USCIS to direct the Chief of the USCIS Contracting Office, in coordination with the appropriate contracting officer, to consider inconsistencies between policy and leading practices in contract administration and, as needed, institute controls to ensure quality assurance surveillance plans are developed when appropriate.

    Agency Affected: Department of Homeland Security

 

Explore the full database of GAO's Open Recommendations »

Jul 27, 2017

Dec 8, 2016

Nov 16, 2016

Oct 7, 2016

Sep 20, 2016

Sep 16, 2016

Sep 8, 2016

Aug 18, 2016

Aug 3, 2016

Looking for more? Browse all our products here