Reports & Testimonies

  • GAO’s recommendations database contains report recommendations that still need to be addressed.

    GAO’s recommendations help congressional and agency leaders prepare for appropriations and oversight activities, as well as help improve government operations. Recommendations remain open until they are designated as Closed-implemented or Closed-not implemented. You can explore open recommendations by searching or browsing.

    GAO's priority recommendations are those that we believe warrant priority attention. We sent letters to the heads of key departments and agencies, urging them to continue focusing on these issues. These recommendations are labeled as such. You can find priority recommendations by searching or browsing our open recommendations below, or through our mobile app.

  • Browse Open Recommendations

    Explore priority recommendations by subject terms or browse by federal agency

    Search Open Recommendations

    Search for a specific priority recommendation by word or phrase



  • Governing on the go?

    Our Priorities for Policy Makers app makes it easier for leaders to search our recommendations on the go.

    See the November 10th Press Release


  • Have a Question about a Recommendation?

    • For questions about a specific recommendation, contact the person or office listed with the recommendation.
    • For general information about recommendations, contact GAO's Audit Policy and Quality Assurance office at (202) 512-6100 or apqa@gao.gov.
  • « Back to Results List Sort by   

    Results:

    Subject Term: "Integrated software"

    1 publication with a total of 9 open recommendations
    Director: Carol C. Harris
    Phone: (202) 512-4456

    9 open recommendations
    Recommendation: To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of the Department of Homeland Security (DHS) should direct the Director of USCIS to direct the USCIS Chief Information Officer (CIO), in coordination with the DHS CIO and the Chief of the Office of Transformation Coordination (OTC), to review and update, as needed, existing policies and guidance and consider additional controls to complete planning for software releases prior to initiating development and ensure software meets business expectations prior to deployment.

    Agency: Department of Homeland Security
    Status: Open

    Comments: As of July 2017, the U.S. Citizenship and Immigration Services (USCIS) within the Department of Homeland Security (DHS) had taken steps to address this recommendation. In particular, in June 2017, USCIS provided an updated policy, dated April 2017, governing planning and deploying software releases. USCIS also demonstrated partial compliance with that policy. For example, it provided some release planning review documentation for recent releases that are required by the updated policy, including readiness review memos for releases 7.2 and 8.1. However, USCIS did not demonstrate that the program responsible for developing the USCIS Electronic Immigration System (USCIS ELIS) was consistently following its updated policy. For example, USCIS did not demonstrate that the program was completing all planning activities prior to initiating development, as called for in its updated policy. Moreover, the agency did not demonstrate compliance with its previous policy for all software releases planned and deployed since our July 2016 report. We will continue to work with USCIS to monitor actions the agency is taking to address this recommendation.
    Recommendation: To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to consistently implement the principles of the framework adopted for Agile software development.

    Agency: Department of Homeland Security
    Status: Open

    Comments: As of July 2017, USCIS had taken steps to address this recommendation. For example, in May 2017, USCIS provided updated policy governing the development of software releases, dated April 2017, along with release planning artifacts specific to USCIS ELIS. The updated policy included an appendix devoted to generally accepted agency practices and applying Agile principles in the agency. However, USCIS had not clearly indicated if USCIS ELIS was to implement the practices described in the policy. For example, the updated policy did not require program compliance with the generally accepted agency practices. Moreover, supporting artifacts from the release planning process did not always define a commitment to a particular development methodology or set of development practices. For example, the team process agreements, which describe how members of individual teams will work with each other, did not indicate if developers were to adhere to the practices described in updated USCIS policy. We will continue to work with USCIS to obtain additional documentation about actions it is taking to address this recommendation.
    Recommendation: To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to define and consistently execute appropriate roles and responsibilities for individuals responsible for development activities consistent with its selected development framework.

    Agency: Department of Homeland Security
    Status: Open

    Comments: As of July 2017, USCIS had taken steps to address this recommendation. For example, in June 2017, USCIS provided updated policy, dated April 2017, governing the development of software releases and release planning artifacts. The updated policy and release documentation defined some roles and responsibilities that were previously only described by USCIS in its informal November 2014 management model, such as the authority and responsibility of a product owner. However, program documentation and policy did not define all of the roles and responsibilities. For example, program documentation and policy did not define the roles and responsibilities of a facilitator, or Scrum Master, which is a position identified in leading practices for software development using Scrum, the development methodology previously identified by the program. In addition, USCIS did not demonstrate that it had defined and committed to an updated development methodology for software releases. Such a defined methodology will impact expectations for the roles and responsibilities in software development. Without such a defined methodology or approach to Agile software development, it is not clear if roles and responsibilities defined by previously documented approach to Agile software development are still applicable for the current development approach. Moreover, documentation associated with program releases and updated policy did not define all of the roles and responsibilities for positions described by USCIS in its May 2017 written response to GAO. We will continue to work with USCIS to obtain additional documentation about actions it is taking to address this recommendation.
    Recommendation: To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to identify all system users and involve them in release planning activities.

    Agency: Department of Homeland Security
    Status: Open

    Comments: As of July 2017, DHS and USCIS had not provided information demonstrating that the department has addressed this recommendation. In October 2016, DHS provided a written response stating that the USCIS Office of Information Technology and Office of Transformation Coordination were working closely with the various USCIS directorates to obtain and integrate feedback through regular review sessions with the end users and through additional end user testing. However, as of July 2017, DHS and USCIS have not provided new information about the status of this recommendation.
    Recommendation: To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to write user stories that identify user roles, include estimates of complexity, take no longer than one sprint to complete, and describe business value.

    Agency: Department of Homeland Security
    Status: Open

    Comments: As of July 2017, USCIS had provided GAO with documentation intended to demonstrate that the agency had taken steps to address this recommendation. For example, in May 2017, USCIS provided updated policy governing the development of software releases along with release planning artifacts specific to USCIS ELIS and an Independent Verification and Validation assessment. The agency also provided a series of backlogs that captured user stories for some software releases. In addition, the Independent Verification and Validation assessment indicated that the program was tracking user story quality as part of assessing whether value was continuously discovered and aligned to the mission. However, the assessment report provided to GAO indicated a negative trend for this outcome. Moreover, USCIS policy no longer set expectations regarding user story development. In addition, supporting artifacts from the release planning process did not always define a commitment to a particular development methodology, which is turn impacts the expectations for writing user stories. Finally, backlogs provided by USCIS did not cover all releases in development since our July 2016 report and did not include enough detail to assess all aspects of the user story process (e.g., story size and user involvement). We will continue to work with USCIS to obtain additional documentation about actions it is taking to address this recommendation.
    Recommendation: To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to establish outcomes for Agile software development.

    Agency: Department of Homeland Security
    Status: Open

    Comments: As of July 2017, USCIS had taken steps to address this recommendation. For example, in April 2017, USCIS issued updated policy governing software development at the agency. The updated policy included an appendix devoted to generally accepted agency practices and applying Agile principles in the agency. This appendix also included a set of ten outcomes associated with using Agile practices at USCIS. For example, outcomes included that value is continuously discovered and aligned to the mission. However, the updated policy did not require program compliance with the practices and principles described in the appendix. Moreover, the agency did not demonstrate that USCIS ELIS had committed to achieving a specific set of outcomes for Agile software development, such as the outcomes described in the USCIS policy. We will continue to work with USCIS to obtain additional documentation about actions it is taking to address this recommendation.
    Recommendation: To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to monitor program performance and report to appropriate entities through the collection of reliable metrics.

    Agency: Department of Homeland Security
    Status: Open

    Comments: As of July 2017, USCIS had taken steps to address this recommendation. For example, in May 2017, USCIS provided updated policy governing the development of software that called for teams to prepare an Operations Monitoring Plan or dashboard showing the practices, tools, and measures that will monitor applications in production. The agency also provided a series of documents from internal systems and processes intended to monitor performance, such as a product dashboard for analyzing code quality (i.e., SonarQube) and a report from its Independent Verification and Validation team. However, the program was undergoing a re-baseline and had yet to document updated cost, schedule, and performance expectations against which to monitor. Moreover, the agency did not demonstrate that other metrics, such as customer satisfaction and team velocity, were being reliably collected. We will continue to work with USCIS to obtain additional documentation about actions it is taking to address this recommendation.
    Recommendation: To help manage the USCIS ELIS system, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update existing policies and guidance and consider additional controls to conduct unit and integration, and functional acceptance tests, and code inspection consistent with stated program goals.

    Agency: Department of Homeland Security
    Status: Open

    Comments: As of July 2017, USCIS had taken steps to address this recommendation. For example, in May 2017, USCIS provided artifacts from internal systems in place to monitor software development performance. These metrics monitored aspects of testing, such as code quality and code coverage. However, the program did not provide an updated Test and Evaluation Master Plan, which is a document it will produce as part of its ongoing effort to re-baseline. A Test and Evaluation Master Plan sets the testing expectations for the program as agreed upon with its stakeholders in DHS and USCIS. The updated plan will provide a basis for further evaluation of the steps DHS and USCIS have taken to address this recommendation. Moreover, the agency did not demonstrate that functional acceptance tests were being conducted in accordance with stated program goals. For example, the agency did not provide acceptance criteria or the associated tests demonstrating that user stories passed the defined acceptance criteria. We will continue to work with USCIS to obtain additional documentation about actions it is taking to address this recommendation.
    Recommendation: To help manage the USCIS ELIS system, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update existing policies and guidance and consider additional controls to develop complete test plans and cases for interoperability and end user testing, as defined in the USCIS Transformation Program Test and Evaluation Master Plan, and document the results.

    Agency: Department of Homeland Security
    Status: Open

    Comments: As of July 2017, DHS and USCIS had not provided information demonstrating that they had addressed this recommendation. In October 2016, DHS provided a written response indicating that an internal process for revisiting the USCIS ELIS Test and Evaluation Master Plan had been initiated, with participation from all relevant stakeholder groups. A Test and Evaluation Master Plan sets the testing expectations for the program as agreed upon with its stakeholders in DHS and USCIS. The updated plan will provide a basis for further evaluation of the steps DHS and USCIS have taken to address this recommendation. The letter also stated that USCIS had begun to work on a policy for new interoperability test procedures. Moreover, the letter added that end user testing is a continuing activity, including providing feedback of observed issues into the development queue, with the slow launch of the naturalization capabilities in USCIS ELIS being a model. However, as of July 2017, DHS and USCIS had not provided new information about the status of this recommendation. We will continue to work with DHS and USCIS to obtain additional documentation about actions they are taking to address this recommendation.