Since its creation in 2003, the Department of Homeland Security (DHS) has been developing new information technology (IT) systems to perform both mission-critical and support functions; however, it has faced challenges in developing these systems. One way to manage the inherent risks of developing and acquiring systems is through independent verification and validation (IV&V)--a process conducted by a party independent of the development effort that provides an objective assessment of a project's processes, products, and risks throughout its life cycle and helps ensure that program performance, schedule, and budget targets are met. GAO was asked to determine (1) how DHS's IV&V policies and procedures for IT acquisitions compare with leading practices and (2) the extent to which DHS has implemented IV&V on its large IT system acquisitions. To do so, GAO assessed DHS's policy against industry standards and leading practice guides, as well as analyzed how eight selected IT programs had implemented IV&V.
DHS recognizes the importance of IV&V and recommends its use on major IT programs. Nevertheless, its acquisition policy does not address the elements of leading practices for IV&V. Specifically, the department has not established risk-based decision making criteria for determining whether, or the extent to which, programs should utilize IV&V. In addition, department policy does not define the degree of independence required of agents and does not require that programs determine and document the planned scope of their efforts, including the program activities subject to review; the resources required; roles and responsibilities; and how the results will be reported and acted upon. Moreover, the policy does not address overseeing DHS's investment in IV&V. Thus, officials were unaware of the extent to which it was being used on major IT acquisition programs, associated expenditures, or if those expenditures are producing satisfactory results. Absent such policy elements and more effective oversight, the department's investments in IV&V efforts are unlikely to provide optimal value for the department and, in some cases, may even fail to deliver any significant benefits. Many large IT acquisition programs across DHS reported using IV&V as part of their acquisition and/or development processes. Nevertheless, the eight major IT acquisition programs that GAO analyzed did not consistently implement the elements of leading practice. For example, the eight did not fully apply a structured, risk-based decision making process when deciding if, when, and how to utilize IV&V. In part, these weaknesses are due to the lack of clear departmentwide guidance regarding the use of such practices. As a result, the department's IV&V efforts may not consistently contribute toward meeting IT acquisition cost, schedule, and mission goals. GAO recommends that DHS (1) update its acquisition policy to reflect elements of effective IV&V, (2) monitor and ensure implementation of this policy on applicable new and ongoing IT programs, and (3) collect data on IV&V usage and use it to evaluate the effectiveness of these investments. DHS concurred with GAO's recommendations and described actions planned or under way to address them.
Recommendations for Executive Action
|Department of Homeland Security||To help guide consistent and effective execution of IV&V at DHS, the Secretary of Homeland Security should direct the department Chief Information Officer (CIO) and Chief Procurement Officer (CPO) to revise DHS acquisition policy such that it establishes (1) risk-based criteria for (1) determining which major and other high- risk IT acquisition programs should conduct IV&V and (2) selecting appropriate activities for independent review of these programs; (2) requirements for technical, financial, and managerial independence of agents; (3) standards and guidance for defining and documenting plans and products; (4) controls for planning, managing, and overseeing efforts; (5) mechanisms to ensure that plans and significant findings inform DHS acquisition program reviews and decisions, including those of the DHS's Acquisition Review Board (ARB); and (6) mechanisms to monitor and ensure implementation of this policy on applicable new IT acquisition programs.|
|Department of Homeland Security||To help guide consistent and effective execution of IV&V at DHS, the Secretary of Homeland Security should direct the department CIO and CPO to reevaluate the approach to IV&V for ongoing programs (including the eight programs featured in this report) and ensure that appropriate actions are taken to bring each of them into alignment with the elements of leading practice.|
|Department of Homeland Security||To help guide consistent and effective execution of IV&V at DHS, the Secretary of Homeland Security should direct the department CIO and CPO to collect and analyze data on IV&V efforts for major IT acquisition programs to facilitate the development of lessons learned and evaluation of the effectiveness of DHS's investments, and establish a process that uses the results to inform the department's IT investment decisions.|