This is the accessible text file for GAO report number GAO-14-361 entitled 'Information Technology: Agencies Need to Establish and Implement Incremental Development Policies' which was released on May 8, 2014. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: GAO: Report to the Committee on Homeland Security and Governmental Affairs, U.S. Senate: May 2014: Information Technology: Agencies Need to Establish and Implement Incremental Development Policies: GAO-14-361: GAO Highlights: Highlights of GAO-14-361, a report to the Committee on Homeland Security and Governmental Affairs, U.S. Senate. Why GAO Did This Study: Federal agencies plan to spend at least $82 billion on IT in fiscal year 2014. However, prior IT expenditures have often produced disappointing results. Thus, OMB has called for agencies to deliver investments in smaller parts or increments. In 2010, it called for IT investments to deliver capabilities every 12 months and now requires investments to deliver capabilities every 6 months. GAO was asked to review agencies' incremental development approaches. Among other things, this report (1) assesses whether selected agencies have established policies for incremental IT development; and (2) determines whether selected agencies are using incremental development approaches to manage their IT investments. To do so, GAO selected five agencies—-Defense, HHS, DHS, Transportation, and VA—-and 89 total investments at these agencies. GAO then reviewed the agencies' incremental development policies and plans. What GAO Found: All five agencies in GAO's review—the Departments of Defense (Defense), Health and Human Services (HHS), Homeland Security (DHS), Transportation (Transportation), and Veterans Affairs (VA)—have established policies that address incremental development; however, the policies usually did not fully address three key components for implementing the Office of Management and Budget's (OMB) guidance (see table). Specifically, only VA fully addressed the three components. Among other things, agencies cited the following reasons that contributed to these weaknesses: (1) the guidance was not feasible because not all types of investments should deliver functionality in 6 months, and (2) the guidance did not identify what agencies' policies are to include or time frames for completion. GAO agrees these concerns have merit. Until OMB issues realistic and clear guidance and agencies address the weaknesses in their incremental development policies, it will be difficult to deliver project capability more rapidly. Table: Assessment of Selected Agencies' Incremental Development Policies: Component: Require delivery of functionality every 6 months Defense: Not met; HHS: Not met; DHS: Not met; Transportation: Not met; VA: Fully met. Component: Define functionality Defense: Partially met; HHS: Not met; DHS: Not met; Transportation: Not met; VA: Fully met. Component: Define a process for enforcing compliance Defense: Partially met; HHS: Partially met; DHS: Not met; Transportation: Not met; VA: Fully met. Source: GAO analysis of agency documentation. [End of table] The weaknesses in agency policies have enabled inconsistent implementation of incremental development approaches: almost three- quarters of the selected investments did not plan to deliver functionality every 6 months, and less than half planned to deliver functionality in 12-month cycles (see table). Without consistent use of incremental development approaches, information technology (IT) expenditures are more likely to continue producing disappointing results. Table: Number of Selected Investments Planning to Incrementally Deliver Functionality: Agency: Defense; Total number of selected investments: 37; Investments planning to deliver functionality every 6 months: 1; Investments planning to deliver functionality every 12 months: 11. Agency: HHS; Total number of selected investments: 14; Investments planning to deliver functionality every 6 months: 9; Investments planning to deliver functionality every 12 months: 11. Agency: DHS; Total number of selected investments: 12; Investments planning to deliver functionality every 6 months: 2; Investments planning to deliver functionality every 12 months: 6. Agency: Transportation; Total number of selected investments: 20; Investments planning to deliver functionality every 6 months: 5; Investments planning to deliver functionality every 12 months: 7. Agency: VA; Total number of selected investments: 6; Investments planning to deliver functionality every 6 months: 6; Investments planning to deliver functionality every 12 months: 6. Agency: Totals; Total number of selected investments: 89; Investments planning to deliver functionality every 6 months: 23; Investments planning to deliver functionality every 12 months: 41. Source: GAO analysis of agency data. [End of table] What GAO Recommends: Among other things, GAO recommends that OMB develop and issue realistic and clear guidance on incremental development and that the selected agencies update and implement their incremental development policies to reflect OMB's guidance. OMB partially disagreed, believing its guidance is realistic. Four agencies generally agreed with the report or had no comments, and one agency did not agree that its recommendations should be dependent on OMB first taking action. GAO continues to believe that its recommendations are valid, as discussed in this report. View [hyperlink, http://www.gao.gov/products/GAO-14-361]. For more information, contact David A. Powner at (202) 512-9286 or pownerd@gao.gov. [End of section] Contents: Letter: Background: Majority of Selected Agencies' Incremental Development Policies Have Significant Weaknesses: Most Selected Investments Do Not Plan to Deliver Functionality Every 6 or 12 Months: Multiple Factors Were Commonly Identified as Enabling and Inhibiting Incremental Development: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Objectives, Scope, and Methodology: Appendix II: Selected Investments' Plans for Delivering Functionality: Appendix III: Comments from the Department of Defense: Appendix IV: Comments from the Department of Homeland Security: Appendix V: Comments from the Department of Veterans Affairs: Appendix VI: GAO Contact and Staff Acknowledgments: Tables: Table 1: Assessment of Selected Agencies' Incremental Development Policies: Table 2: Number and Percentage of Selected Investments That Planned to Deliver Functionality Every 6 Months, by Agency, for Fiscal Years 2013 and 2014: Table 3: Number and Percentage of Remaining Selected Investments That Planned to Deliver Functionality Every 6 Months, by Agency, for Fiscal Years 2013 and 2014: Table 4: Number and Percentage of Selected Investments That Planned to Deliver Functionality Every 12 Months, by Agency, for Fiscal Years 2013 and 2014: Table 5: Commonly Identified Factors Enabling Incremental Development during a 6-month Period: Table 6: Commonly Identified Factors Inhibiting Incremental Development during a 6-month Period: Table 7: Selected Investments and Associated Plans for Delivering Functionality Every 6 and 12 Months, for Fiscal Years 2013 and 2014: Abbreviations: CIO: chief information officer: CMMI-ACQ: Capability Maturity Model® Integration for Acquisition: Defense: Department of Defense: DHS: Department of Homeland Security: FAA: Federal Aviation Administration: HHS: Department of Health and Human Services: IT: information technology: OMB: Office of Management and Budget: SEI: Software Engineering Institute: Transportation: Department of Transportation: VA: Department of Veterans Affairs: [End of section] United States Government Accountability Office: GAO: 441 G St. N.W. Washington, DC 20548: May 1, 2014: The Honorable Thomas R. Carper: Chairman: The Honorable Tom Coburn, M.D. Ranking Member: Committee on Homeland Security and Governmental Affairs: United States Senate: Information systems are critical to the health, economy, and security of the nation. To support agency missions, the federal government plans to spend at least $82 billion on information technology (IT) investments in fiscal year 2014. However, as we have previously reported, prior IT expenditures too often have produced failed projects--that is, projects with multimillion dollar cost overruns and schedule delays measured in years, with questionable mission-related achievements.[Footnote 1] Many of these investments have been broadly scoped projects that aim to deliver their capabilities several years after initiation. To help resolve these issues, Congress and the Office of Management and Budget (OMB) have called for agencies to deliver investments in smaller parts, or increments, in order to reduce investment risk, deliver capabilities more quickly, and facilitate the adoption of emerging technologies.[Footnote 2] In 2010, OMB placed a renewed emphasis on incremental IT development by calling for major IT investments to deliver functionality at least every 12 months. Subsequently, OMB has made this guidance more stringent, and annual budget guidance now states that each project associated with major IT investments is to deliver functionality every 6 months.[Footnote 3] This report responds to your request that we review agencies' incremental IT development. Specifically, we (1) assessed whether selected agencies have established policies for incremental IT development, (2) determined whether selected agencies are using incremental development approaches to manage their IT investments, and (3) identified the key factors that enabled and inhibited the selected agencies' abilities to effectively use incremental development approaches to manage their IT investments. In conducting our review, we selected five agencies and 89 investments to review. We selected the five agencies with the largest IT budgets for development, modernization, and enhancement on major IT investments[Footnote 4] in fiscal years 2013 and 2014: the Departments of Defense (Defense), Health and Human Services (HHS), Homeland Security (DHS), Transportation (Transportation), and Veterans Affairs (VA). We then selected the major IT investments at these agencies that planned to spend more than 50 percent of the investments' fiscal year 2013 and 2014 budgets on development, modernization, and enhancement. These 89 investments are identified in appendix II. To address our first objective, we reviewed OMB guidance related to the use of incremental development and identified three key components of incremental development that agencies should include in their policies. Specifically, in order to identify these components, we reviewed OMB's Circular A-130[Footnote 5]--which requires agencies to develop incremental development policies, but does not provide guidance on what the policies are to include--as well as more recent OMB guidance[Footnote 6] on incremental development and leading industry guidance on institutionalizing processes throughout an organization.[Footnote 7] We then compared the selected agencies' incremental development policies to the components we identified. To address our second objective, we analyzed information obtained from data collection instruments describing how often the selected investments planned to deliver functionality. Specifically, we administered a data collection instrument to officials responsible for each of the selected investments that asked how often they planned to deliver functionality during fiscal years 2013 and 2014. Using these data, we determined the extent to which the selected investments and their projects are meeting OMB's guidance on incremental development. To address our third objective, officials from the five selected agencies and investments identified the key factors that have both enabled and inhibited their efforts to deliver functionality for their major IT investments, consistent with OMB guidance. We then determined the key factors that were commonly identified by the selected agencies and investments. We conducted this performance audit from May 2013 to May 2014, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Details of our objectives, scope, and methodology are contained in appendix I. Background: If done correctly, investments in IT have the potential to make organizations more efficient in fulfilling their missions. For example, we recently reported that Defense officials stated that an IT system supporting military logistics has improved the organization's performance by providing real-time information about road conditions, construction, incidents, and weather to facilitate rapid deployment of military assets.[Footnote 8] However, as we have previously reported, investments in federal IT too frequently result in failed projects that incur cost overruns and schedule slippages while contributing little to mission-related outcomes. For example: * In January 2011, the Secretary of Homeland Security ended the Secure Border Initiative Network program after obligating more than $1 billion for the program because it did not meet cost-effectiveness and viability standards. Since 2007, we have identified a range of issues and made several recommendations to improve this program.[Footnote 9] For example, in May 2010, we reported that the final acceptance of the first two deployments had slipped from November 2009 and March 2010 to September 2010 and November 2010, respectively, and that the cost- effectiveness of the system had not been justified.[Footnote 10] We concluded that DHS had not demonstrated that the considerable time and money being invested to acquire and deploy the program were a wise and prudent use of limited resources. As a result, we recommended that the department (1) limit near-term investment in the program, (2) economically justify any longer-term investment in it, and (3) improve key program management disciplines. This work contributed to the department's decision to cancel the program. * In February 2011, the Office of Personnel Management canceled its Retirement Systems Modernization program after several years of trying to improve the implementation of this investment.[Footnote 11] According to the Office of Personnel Management, it spent approximately $231 million on this investment. We issued a series of reports on the agency's efforts to modernize its retirement system and found that the Office of Personnel Management was hindered by weaknesses in several important management disciplines that are essential to successful IT modernization efforts.[Footnote 12] Accordingly, we made recommendations in areas such as project management, organizational change management, testing, and cost estimating. In May 2008, an Office of Personnel Management official cited the issues that we identified as justification for issuing a stop work order to the system contractor, and the agency subsequently terminated the contract. * In December 2012, Defense canceled the Air Force's Expeditionary Combat Support System after having spent more than a billion dollars and missing multiple milestones, including failure to achieve deployment within 5 years of obligating funds. We issued several reports on this system and found that, among other things, the program was not fully following best practices for developing reliable schedules and cost estimates.[Footnote 13] Agencies have reported that poor-performing projects have often used a "big bang" approach--that is, projects that are broadly scoped and aim to deliver functionality several years after initiation. For example, in 2009 the Defense Science Board reported that Defense's acquisition process for IT systems--which was rooted in the "waterfall" development model[Footnote 14]--was too long, ineffective, and did not accommodate the rapid evolution of IT.[Footnote 15] The board reported that the average time to deliver an initial program capability for a major IT system acquisition at Defense was over 7 years. Also in 2009, VA's former chief information officer (CIO) reported that many of its projects exceeded cost estimates by more than 50 percent and missed scheduled completion dates by more than a year. That official concluded that VA needed to make substantial changes to its acquisition process in order to eliminate project failures associated with the "big bang" approach. One approach to reducing the risks from broadly scoped, multiyear projects is to divide investments into smaller parts--a technique long advocated by Congress and OMB.[Footnote 16] By following this approach, agencies can potentially: * deliver capabilities to their users more rapidly, giving them more flexibility to respond to changing agency priorities; * increase the likelihood that each project will achieve its cost, schedule, and performance goals; * obtain additional feedback from users, increasing the probability that each successive increment and project will meet user needs; * more easily incorporate emerging technologies; and: * terminate poorly performing investments with fewer sunk costs. OMB has placed a renewed emphasis on such development. Since 2000, OMB Circular A-130 has required agencies to incorporate incremental development approaches into their policies and ensure that investments implement them.[Footnote 17] More recently, in its 2010 IT Reform Plan, OMB called for IT programs to deliver functionality at least every 12 months and complete initial deployment to end users no later than 18 months after the start of the program.[Footnote 18] In 2011, as part of its budget guidance, OMB first recommended that projects associated with major IT investments deliver functionality every 6 months.[Footnote 19] OMB's latest guidance now makes this mandatory; specifically, in 2012, OMB began requiring that functionality be delivered at least every 6 months.[Footnote 20] Roles and Responsibilities for Overseeing Federal IT Investments: Over the last three decades, Congress has enacted several laws to assist agencies and the federal government in managing IT investments. For example, the Paperwork Reduction Act of 1995 requires that OMB develop and oversee policies, principles, standards, and guidelines for federal agency IT functions, including periodic evaluations of major information systems.[Footnote 21] In addition, to assist agencies in managing their investments, Congress enacted the Clinger- Cohen Act of 1996.[Footnote 22] Among other things, the act requires agency heads to appoint CIOs and specifies many of their responsibilities.[Footnote 23] With regard to IT management, CIOs are responsible for implementing and enforcing applicable governmentwide and agency IT management principles, standards, and guidelines; assuming responsibility and accountability for IT investments; and monitoring the performance of IT programs and advising the agency head whether to continue, modify, or terminate such programs.[Footnote 24] Additionally, with regard to incremental development, Clinger-Cohen calls for provisions in the Federal Acquisition Regulation that encourage agencies to structure their IT contracts such that the capabilities are delivered in smaller increments. The Federal Acquisition Regulation provisions are also to provide, to the maximum extent practicable, that the increment should be delivered within 18 months of the contract solicitation.[Footnote 25] As set out in these laws, OMB is to play a key role in helping federal agencies manage their investments by working with them to better plan, justify, and determine how much they need to spend on projects and how to manage approved projects. Within OMB, the Office of E-Government and Information Technology, headed by the Federal CIO,[Footnote 26] directs the policy and strategic planning of federal IT investments and is responsible for oversight of federal technology spending. In carrying out its responsibilities, OMB uses several data collection mechanisms to oversee federal IT spending during the annual budget formulation process. Specifically, OMB requires federal departments and agencies to provide information related to their IT investments (called exhibit 53s) and capital asset plans and business cases (called exhibit 300s). * Exhibit 53. The purpose of the exhibit 53 is to identify all IT investments--both major and nonmajor--and their associated costs within a federal organization. Information included in agency exhibit 53s is designed, in part, to help OMB better understand agencies' spending on IT investments. * Exhibit 300. The purpose of the exhibit 300 is to provide a business case for each major IT investment and to allow OMB to monitor IT investments once they are funded. An IT investment may include one or more projects that are to develop, modernize, enhance, or maintain a single IT asset or group of IT assets with related functionality. Agencies are required to provide information on each major investment's projects, including cost, schedule, and performance information. For example, in order to measure compliance with its requirement that projects deliver functionality in 6-month cycles, OMB requires agencies to break their projects into activities, and describe when the activities are to deliver functionality. OMB's Recent Major Initiatives for Overseeing IT Investments: OMB has implemented a series of initiatives to improve the oversight of underperforming investments and more effectively manage IT. These efforts include the following: * IT Dashboard. In June 2009, to further improve the transparency into and oversight of agencies' IT investments, OMB publicly deployed the IT Dashboard.[Footnote 27] As part of this effort, OMB issued guidance directing federal agencies to report, via the Dashboard, the performance of their IT investments. Currently, the Dashboard publicly displays information on the cost, schedule, and performance of over 700 major federal IT investments at 26 federal agencies. Further, the public display of these data is intended to allow OMB, other oversight bodies, and the general public to hold the government agencies accountable for results and progress. * TechStat reviews. In January 2010, the Federal CIO began leading TechStats sessions--face-to-face meetings to terminate or turn around IT investments that are failing or are not producing results. These meetings involve OMB and agency leadership and are intended to increase accountability and transparency and improve performance. For example, the Federal CIO testified in June 2013 that he holds TechStat meetings on large investments that are not being acquired incrementally.[Footnote 28] More recently, OMB empowered agency CIOs to hold their own TechStat sessions within their respective agencies. In doing so, OMB has called for agencies to use their TechStat processes to identify investments that are not being acquired incrementally and undertake corrective actions.[Footnote 29] * IT Reform Plan. In December 2010, OMB released its 25-point plan to reform federal IT.[Footnote 30] This document established an ambitious plan for achieving operational efficiencies and effectively managing large-scale IT programs. In particular, as part of its effort to effectively manage IT acquisitions, the plan calls for federal IT programs to deploy functionality in release cycles no longer than 12 months, and ideally, less than 6 months. The plan also identifies key actions that can help agencies implement this incremental development guidance, such as working with Congress to develop IT budget models that align with incremental development, and issuing contracting guidance and templates to support incremental development. GAO Has Reported on Efforts to Improve IT Acquisitions: In April 2012, we reported on OMB's efforts to implement the actions called for in its IT Reform Plan and found that it had partially completed work on two key action items relating to incremental development--issuing contracting guidance and templates to support incremental development and working with Congress to create IT budget models that align with incremental development.[Footnote 31] With respect to the contracting guidance and templates, we found that, although OMB worked with the IT and acquisition community to develop guidance, it had not yet issued this guidance or the templates. Regarding the IT budget models, we found that, although OMB worked to promote ideas for IT budget flexibility (such as multiyear budgets or revolving funds) with congressional committees, there has not yet been any new legislation to create budget models, and OMB has not identified options to increase transparency for programs that would fall under these budgetary flexibilities. We recommended that the Director of OMB ensure that all action items called for in the IT Reform Plan are completed. OMB agreed with this recommendation. OMB has since issued contracting guidance for incremental development, [Footnote 32] but, as of January 2014, a staff member from the OMB Office of E-Government and Information Technology stated that activities to address the development of new IT budget models are still ongoing. * Additionally, in 2011, we identified seven successful investment acquisitions and nine common factors critical to their success. [Footnote 33] Specifically, we reported that department officials identified seven successful investment acquisitions, in that they best achieved their respective cost, schedule, scope, and performance goals.[Footnote 34] Notably, all of these were smaller increments, phases, or releases of larger projects. For example, the Defense investment in our sample was the seventh increment of an ongoing investment; the Department of Energy system was the first of two phases; the DHS investment was rolled out to two locations prior to deployment to 37 additional locations; and the Transportation investment had been part of a prototype deployed to four airports. In addition, common factors critical to the success of three or more of the seven investments were: (1) program officials were actively engaged with stakeholders, (2) program staff had the necessary knowledge and skills, (3) senior department and agency executives supported the programs, (4) end users and stakeholders were involved in the development of requirements, (5) end users participated in testing of system functionality prior to formal end user acceptance testing, (6) government and contractor staff were stable and consistent, (7) program staff prioritized requirements, (8) program officials maintained regular communication with the prime contractor, and (9) programs received sufficient funding. These critical factors support OMB's objective of improving the management of large-scale IT acquisitions across the federal government, and wide dissemination of these factors could complement OMB's efforts. Further, in 2012, we identified 32 practices and approaches as effective for applying Agile software development[Footnote 35] methods to IT projects.[Footnote 36] Officials from five agencies who had used Agile methods on federal projects cited beneficial practices, such as obtaining stakeholder and customer feedback frequently, managing requirements, and ensuring staff had the proper knowledge and experience. We also identified 14 challenges with adapting and applying Agile in the federal environment, including agencies having difficulty with committing staff to projects, procurement practices that did not support Agile projects, and compliance reviews that were difficult to execute within an iterative time frame. We noted that the effective practices and approaches identified in the report, as well as input from others with broad Agile experience, could help agencies in the initial stages of adopting Agile. Majority of Selected Agencies' Incremental Development Policies Have Significant Weaknesses: Since 2000, OMB Circular A-130 has required agencies to (1) develop policies that require their major investments to deliver functionality incrementally and (2) ensure that investments comply with their policies.[Footnote 37] In addition, as part of its recent budget guidance, OMB has defined how often investments must deliver functionality. Specifically, each project associated with major IT investments is to deliver functionality at least once every 6 months. [Footnote 38] Further, through the President's Budget, OMB has provided additional guidance on how incremental development is to be enforced by requiring agencies to use their TechStat processes to identify investments that are not being acquired incrementally and undertake corrective actions.[Footnote 39] Although OMB's guidance requires agencies to develop incremental development policies, it does not specify what those policies are to include. Absent this detail and in reviewing the previously mentioned guidance and leading practices on institutionalizing processes throughout an organization,[Footnote 40] we identified three components that agencies should include in their policies in order to effectively carry out OMB's incremental development guidance: * require that all projects associated with major IT investments deliver functionality in cycles that are not more than 6-months long; * define functionality--that is, what the projects are to deliver at the end of a 6-month cycle; and: * define a process for ensuring that major IT investments and their projects deliver functionality every 6 months. This should include identifying investments that are not being acquired incrementally through agency TechStat processes and undertaking corrective actions. Although all five selected agencies developed policies that address incremental development, the majority of the agencies' policies did not fully address all three components. Specifically, only VA fully addressed the three components; Defense partially addressed the majority of them; and HHS, DHS, and Transportation did not address the majority of components. Table 1 provides a detailed assessment of each agency's policies against the three key components of an incremental development policy. In addition, a discussion of each policy component follows the table. Table 1: Assessment of Selected Agencies' Incremental Development Policies: Component: Require delivery of functionality every 6 months Defense: Not met; HHS: Not met; DHS: Not met; Transportation: Not met; VA: Fully met. Component: Define functionality Defense: Partially met; HHS: Not met; DHS: Not met; Transportation: Not met; VA: Fully met. Component: Define a process for enforcing compliance Defense: Partially met; HHS: Partially met; DHS: Not met; Transportation: Not met; VA: Fully met. Source: GAO analysis of agency documentation. Key: Fully met--the agency provided evidence that addressed the component. Partially met--the agency provided evidence that addressed about half or a large portion of the component. Not met--the agency did not provide evidence that addressed the component or provided evidence that minimally addressed the component. [End of table] * Require delivery of functionality every 6 months. Only one of the five agencies--VA--fully addressed this policy component by clearly requiring that its projects be completed in increments that must not exceed 6 months.[Footnote 41] The other four agencies did not address this policy component. Three of these agencies--Defense, HHS, and DHS-- all developed policies that promote the use of incremental development, but these policies do not require functionality to be delivered every 6 months. Specifically, with regard to Defense, although the department's acquisition framework calls for investments to use incremental development,[Footnote 42] its policy on IT budget submissions encourages investments to deliver functionality every 12- 18 months--not every 6 months.[Footnote 43] According to officials of the Defense Office of the CIO, 12-18 month incremental development is better aligned with the acquisition framework that many of its IT acquisitions have used.[Footnote 44] For HHS, although its policy requires incremental development, its policy recommends--but does not require--that all projects deliver functionality every 3-6 months. According to an HHS Office of the CIO official, HHS does not require its projects to deliver functionality every 6 months because it wants to provide projects with flexibility. For DHS, in June 2012, the former DHS CIO issued a draft policy encouraging IT projects to move towards Agile development approaches. Additionally, with respect to financial systems modernization programs, DHS's policy calls for providing financial capabilities to the customer in small increments of 6-12 months.[Footnote 45] According to DHS officials representing the Office of the CIO, the department is currently developing a departmentwide policy on incremental development; however, they said that the draft currently encourages investments to deliver the first release 18 months after program initiation and thereafter deploy functionality in cycles no longer than 12 months, but ideally less than 6 months. Lastly, Transportation also did not address the component because, although Transportation has a policy that calls for projects to deliver functionality every 6 months, officials from the Office of the CIO explained that this policy does not apply to the Federal Aviation Administration (FAA).[Footnote 46] These officials explained that how often FAA projects deliver functionality depends on their size, scope, risk, visibility, and interdependencies with other programs. * Define functionality. Only one of the five agencies--VA--fully addressed this policy component. VA has a policy that defines what it means to deliver functionality--both in terms of what constitutes an increment and what should be delivered at the end of an increment. For example, VA defines an increment as the segment of the project that produces, in a cycle of 6 months or less, a deliverable that can be used by customers in an operational environment.[Footnote 47] For the agency that partially addressed the component--Defense--although it has defined functionality for purposes of its acquisition framework, it has not defined the functionality that its IT budget submission policy encourages projects to deliver every 12-18 months. The department stated that it is working with OMB to define this term. Lastly, three agencies--HHS, DHS, and Transportation--had not defined functionality in terms of what they expected projects to deliver at the end of a development cycle. Officials representing these agencies' respective Office of the CIO acknowledged that they have not defined functionality. These officials told us that they would update their policies to define the term, but officials from HHS and Transportation did not provide a time frame for doing so. DHS officials stated that, although they did not have a definitive timeframe, they hoped to finalize the policy in 2014. Until the agencies define this term, investments may create definitions that are inconsistent with the intent of OMB's policy. * Define a process for enforcing compliance. Only one of the five agencies--VA--fully addressed this policy component by defining processes for ensuring that increments are structured to deliver functionality every 6 months or less and for reviewing projects that fall behind schedule. In particular, VA's policy requires the agency to hold a TechStat session when any increment delivery date has been or will be missed. Two agencies partially addressed this component-- Defense and HHS--because, although they established processes for ensuring that IT is acquired incrementally, these processes do not (1) require enforcement of incremental development within the specific time frames consistent with OMB guidance (12-18 months for Defense and 3-6 months for HHS) or (2) include using TechStat processes to identify investments that are not being acquired incrementally. Finally, two agencies--DHS and Transportation--have not established processes for enforcing compliance with their incremental development policies. Officials from their respective Office of the CIO told us that they are updating their policies to address this issue. Transportation officials representing the Office of the CIO stated that it would update its policy later this year; DHS Office of the CIO officials stated that, although they did not have a definitive timeframe, they hoped to finalize their policy in 2014. Agencies cited several underlying reasons that contributed to these weaknesses: (1) they were not always aware of OMB guidance, (2) they did not believe that the guidance was realistic, and (3) they said the guidance was not always clear. Regarding agency awareness of the guidance, since the 2010 IT Reform Plan, OMB has communicated changes to incremental development requirements, such as the change from 12 to 6 months, through budget guidance.[Footnote 48] However, selected agency officials said they were not always aware of this guidance. For example, DHS Office of the CIO officials told us that did not know about OMB's requirement to deliver functionality every 6 months. Additionally, Transportation officials representing the Office of the CIO were not aware that in 2012 OMB had changed its guidance from recommending to requiring that projects deliver in 6-month cycles. With respect to whether OMB's guidance is realistic, officials from Defense, HHS, DHS, and Transportation explained that they do not want to require all of their projects to deliver functionality every 6 months because it may not be reasonable for all investments to do so. Defense, DHS, and Transportation officials said that delivering every 12 months, as advocated in OMB's IT Reform Plan, is more reasonable. [Footnote 49] According to Defense officials from the Office of the CIO, 12-18 month incremental development is better aligned with the acquisition framework that many of its IT investments have used. DHS and Transportation Office of the CIO officials stated that, depending on program size, scope, complexity, budget, schedule, and expertise, it may be more reasonable to deliver functionality every 12 months. As discussed later in this report, we agree that OMB's requirement to deliver functionality every 6 months is unrealistic. OMB staff members from the Office of E-Government and Information Technology noted that it will take time for agencies to embrace delivering functionality every 6 months because of the perceived risk of adopting such approaches. Those staff members explained that agencies, such as Defense, have been using the waterfall development method for many years, and they perceive less risk in continuing with that method than with changing to a method that produces functionality more rapidly. Lastly, two key components of OMB's guidance are not clear. First, in revising Circular A-130 in 2000, OMB did not identify the minimum requirements of what the agencies' policies are to include and did not specify when the policies are to be completed. Although OMB issued later guidance on incremental development, it has not yet specified what agencies' incremental development policies are to include. Second, OMB's guidance did not provide a complete definition of the functionality it expects to be delivered every 6 months. According to staff from the Office of E-Government, OMB intends for agencies to deliver functionality that can be tested by business users; nevertheless, they noted that they left this definition out of their guidance so that agencies could develop a definition that would be flexible enough to meet their needs. However, in the absence of further guidance from OMB, agencies may continue to not define this term or may create definitions that are inconsistent with the intent of OMB's policy. For example, HHS officials from the Office of the CIO told us that the completion of requirements documents could meet OMB's definition of delivering functionality. Additionally, an FAA Office of the CIO official explained that some investments have classified the delivery of requirements documentation as functionality. These two examples are not consistent with OMB's intent since they do not deliver functionality that can be tested, but instead only plan to deliver project documentation. Until OMB explicitly issues realistic and clear guidance and Defense, HHS, DHS, and Transportation address the identified weaknesses in their incremental development policies, it will be difficult to deliver project functionality more rapidly, measure how often projects are delivering functionality, and enforce compliance with the delivery time frames called for in their policies. Most Selected Investments Do Not Plan to Deliver Functionality Every 6 or 12 Months: In its 2010 IT Reform Plan, OMB called for IT programs to deliver functionality at least every 12 months.[Footnote 50] Subsequently, OMB has made this requirement more stringent in that it now requires projects associated with major IT investments to deliver functionality every 6 months.[Footnote 51] The majority of the selected investments we reviewed did not plan to deliver functionality every 6 months. Specifically, only 23 of the selected 89 investments had one or more projects that, when taken collectively, planned to deliver functionality every 6 months. To VA's credit, all six of the department's selected investments planned to deliver functionality every 6 months. The other agencies varied in the extent to which they met the standards established by OMB's guidance. Table 2 shows how many of the selected investments at each agency planned on delivering functionality every 6 months during fiscal years 2013 and 2014. Table 2: Number and Percentage of Selected Investments That Planned to Deliver Functionality Every 6 Months, by Agency, for Fiscal Years 2013 and 2014: Agency: Defense; Total number of selected investments: 37; Number of investments that planned to deliver functionality every 6 months: 1; Percentage of investments that planned to deliver functionality every 6 months: 3%. Agency: HHS; Total number of selected investments: 14; Number of investments that planned to deliver functionality every 6 months: 9; Percentage of investments that planned to deliver functionality every 6 months: 64%. Agency: DHS; Total number of selected investments: 12; Number of investments that planned to deliver functionality every 6 months: 2; Percentage of investments that planned to deliver functionality every 6 months: 17%. Agency: Transportation; Total number of selected investments: 20; Number of investments that planned to deliver functionality every 6 months: 5; Percentage of investments that planned to deliver functionality every 6 months: 25%. Agency: VA; Total number of selected investments: 6; Number of investments that planned to deliver functionality every 6 months: 6; Percentage of investments that planned to deliver functionality every 6 months: 100%. Agency: Totals; Total number of selected investments: 89; Number of investments that planned to deliver functionality every 6 months: 23; Percentage of investments that planned to deliver functionality every 6 months: 26%. Source: GAO analysis of agency data. [End of table] The variety of life-cycle cost estimates for these investments shows that incremental development can be applied to a wide variety of investment scopes. Specifically, of the 23 investments that planned to deliver functionality in 6-month cycles, 9 had cost estimates that were less than $250 million, 4 had estimates between $250 and $575 million, 5 had estimates between $575 million and $2 billion, and 5 had estimates greater than $2 billion. Twenty-seven of the 89 investments in our review (30 percent) reported using an Agile development methodology for one or more of their projects. Of those 27 investments, 14 (52 percent) planned to deliver functionality every 6 months.[Footnote 52] The other 13 investments using Agile did not plan on delivering functionality as frequently as OMB guidance requires. We have previously found that Agile projects typically produce working functionality every 1 to 8 weeks;[Footnote 53] as such, it appears that these investments may not be properly implementing an Agile development methodology. Agency officials cited three types of investments for which it may not always be practical or necessary to expect functionality to be delivered in 6-month cycles: (1) investments in life-cycle phases other than acquisition (i.e., planning and budgeting, management in- use, and disposition);[Footnote 54] (2) investments intended to develop IT infrastructure; and (3) research and development investments. * Life-cycle phases other than acquisition. Officials from Defense, HHS, DHS, and Transportation stated that it is not reasonable to expect investments to deliver functionality every 6 months when their investments' projects are not in the acquisition phase.[Footnote 55] Specifically, 24 investments did not have projects in the acquisition stage. Of those 24 investments, 22 did not plan to deliver functionality in 6-month cycles (10 from Defense, 3 from HHS, 1 from DHS, and 8 from Transportation), and 2 investments did plan to do so (2 from HHS). For the 2 investments that planned to deliver functionality every 6 months, both had at least one project in the management in-use phase, meaning that at least one of the investments' projects were beyond the planning and development stages and were being used to support agency operations. * Infrastructure investments. Officials from Defense, DHS, and Transportation explained that not all infrastructure investments can be expected to deliver functionality every 6 months. Specifically, 21 investments provide infrastructure, such as IT security, office automation, and telecommunications. For example, officials representing two of the DHS investments explained that, prior to deploying functionality, they need to acquire real estate, conduct environmental assessments, and perform construction work, such as digging trenches, burying cables, and building facilities. Of the 21 investments, 20 did not plan to deliver functionality every 6 months (17 from Defense, 1 from HHS, and 2 from DHS); however, 1 investment did plan to do so (1 from Defense). * Research and development investments. Officials from FAA's Office of the CIO explained that FAA's research and development investments are not intended to deliver functionality. Those officials stated that, before the agency approves a technology for further development, it performs research and development to ensure that the technology meets safety standards. If those standards are met, FAA creates a new investment aimed at deploying that technology. Consistent with this, none of FAA's six research and development investments planned to deliver any functionality during fiscal years 2013 and 2014. These concerns have merit. For example, with respect to investments in life-cycle phases other than acquisition, at the outset of a new investment, an agency may need more than 6 months to, among other things, define high-level requirements and find a contractor to help develop the system. In addition, it may not be necessary for an investment to continue delivering new functionality when all planned functionality has been fully deployed. Regarding infrastructure investments, it may not be practical or cost- effective for an investment to refresh fully functioning hardware every 6 months. Additionally, it may not be feasible for an agency to build a physical facility (e.g., data center) within 6 months. Further, for research and development investments, industry practices[Footnote 56] and our work on best practices[Footnote 57] support FAA's efforts to thoroughly validate the feasibility and cost- effectiveness of new technology prior to making significant investments. Although OMB requires all investments to deliver functionality every 6 months, an OMB staff member from the Office of E-Government explained that not all investments will be able to meet this goal. Instead, that staff member said that about half of the federal government's major IT investments will deliver functionality in 6 months or less, and the other half will have longer development cycles. However, OMB's guidance does not make this distinction. As a result, agencies may be confused about whether OMB's incremental development guidance applies to all investments. If these three types of investments, which account for 40 of the selected 89 investments, are not considered,[Footnote 58] 29 of the remaining 49 investments did not plan to deliver functionality in 6- month cycles. Table 3 shows, after removing the three types of investments discussed above, how many of the selected investments at each agency planned to deliver functionality every 6 months during fiscal years 2013 and 2014. Table 3: Number and Percentage of Remaining Selected Investments That Planned to Deliver Functionality Every 6 Months, by Agency, for Fiscal Years 2013 and 2014: Agency: Defense; Total number of remaining investments: 14; Number of remaining investments that planned to deliver functionality every 6 months: 0; Percentage of remaining investments that planned to deliver functionality every 6 months: 0%. Agency: HHS; Total number of remaining investments: 8; Number of remaining investments that planned to deliver functionality every 6 months: 7; Percentage of remaining investments that planned to deliver functionality every 6 months: 88%. Agency: DHS; Total number of remaining investments: 9; Number of remaining investments that planned to deliver functionality every 6 months: 2; Percentage of remaining investments that planned to deliver functionality every 6 months: 22%. Agency: Transportation; Total number of remaining investments: 12; Number of remaining investments that planned to deliver functionality every 6 months: 5; Percentage of remaining investments that planned to deliver functionality every 6 months: 42%. Agency: VA; Total number of remaining investments: 6; Number of remaining investments that planned to deliver functionality every 6 months: 6; Percentage of remaining investments that planned to deliver functionality every 6 months: 100%. Agency: Totals; Total number of remaining investments: 49; Number of remaining investments that planned to deliver functionality every 6 months: 20; Percentage of remaining investments that planned to deliver functionality every 6 months: 41%. Source: GAO analysis of agency data. [End of table] Considering agencies' concerns about delivering functionality every 6 months for the three types of investments discussed above and OMB's own expectations that many investments will not meet this goal, it is unclear whether this is the most appropriate governmentwide goal, and it raises the question of whether OMB should consider a longer time frame, such as 12 months, as called for in OMB's IT Reform Plan. However, even using the time frame of 12 months as the target, less than half of the selected investments planned to deliver functionality in 12-month cycles. Specifically, 41 of the 89 selected investments planned to deliver functionality every 12 months during fiscal years 2013 and 2014 and 48 did not. Most notably, the preponderance of Defense and Transportation investments (70 percent for Defense and 65 percent for Transportation) did not plan to deliver functionality every 12 months. Table 4 shows how many of the selected investments at each agency planned on delivering functionality every 12 months during fiscal years 2013 and 2014. Table 4: Number and Percentage of Selected Investments That Planned to Deliver Functionality Every 12 Months, by Agency, for Fiscal Years 2013 and 2014: Agency: Defense; Total number of selected investments: 37; Number of investments that planned to deliver functionality every 12 months: 11; Percentage of investments that planned to deliver functionality every 12 months: 30%. Agency: HHS; Total number of selected investments: 14; Number of investments that planned to deliver functionality every 12 months: 11; Percentage of investments that planned to deliver functionality every 12 months: 79%. Agency: DHS; Total number of selected investments: 12; Number of investments that planned to deliver functionality every 12 months: 6; Percentage of investments that planned to deliver functionality every 12 months: 50%. Agency: Transportation; Total number of selected investments: 20; Number of investments that planned to deliver functionality every 12 months: 7; Percentage of investments that planned to deliver functionality every 12 months: 35%. Agency: VA; Total number of selected investments: 6; Number of investments that planned to deliver functionality every 12 months: 6; Percentage of investments that planned to deliver functionality every 12 months: 100%. Agency: Totals; Total number of selected investments: 89; Number of investments that planned to deliver functionality every 12 months: 41; Percentage of investments that planned to deliver functionality every 12 months: 46%. Source: GAO analysis of agency data. [End of table] The previously discussed weaknesses in agency policies have permitted the inconsistent implementation of incremental development approaches. OMB staff members from the Office of E-Government acknowledged that inconsistent implementation of OMB's guidance can be at least partially attributed to challenges in ensuring that agencies develop consistent definitions of and approaches to incremental development. Although OMB has led the government's recent effort to improve incremental development, it has not completed a key commitment aimed at improving incremental development--namely, OMB has rarely used the TechStat process to turn around or cancel investments that are not using incremental development. As previously mentioned, TechStat sessions are intended to terminate or turn around IT investments that are failing or are not producing results. Additionally, many failed IT investments[Footnote 59] have not used incremental development approaches. Therefore, OMB could use TechStat sessions as a powerful tool to turn around investments that are not being acquired incrementally. However, OMB staff members from the Office of E- Government said that OMB has only held one such TechStat.[Footnote 60] OMB staff from the Office of E-Government explained that, in order to select investments in need of a TechStat session, agency exhibit 300 data are reviewed for evidence of poor performance. However, the usefulness of the exhibit 300 data for the purpose of identifying whether investments are using incremental approaches is limited. For example, OMB does not require agencies to explicitly identify whether their investments are using incremental approaches. Additionally, of the 89 selected investments, 34 had activities in their exhibit 300 submissions that were inaccurately classified as delivering functionality (9 from Defense, 8 from HHS, 5 from DHS, 9 from Transportation, and 3 from VA). For example, one Defense investment indicated that awarding a contract constituted a delivery of functionality. OMB staff from the Office of E-Government acknowledged that the exhibit 300 data are not as helpful as possible in addressing incremental development. Consequently, for its TechStat reviews, OMB's insight into investments' use of incremental development approaches is limited. Officials from Defense, HHS, DHS, and Transportation attributed the problem to a lack of guidance from OMB on what is to be delivered every 6 months.[Footnote 61] Nevertheless, officials from Defense, DHS, and VA stated that they would properly classify activities in future exhibit 300 submissions.[Footnote 62] However, without additional guidance from OMB, agencies may continue to improperly classify activities. Additionally, as previously mentioned, four of the five selected agencies--Defense, HHS, DHS, and Transportation--have not updated their TechStat policies to include identifying investments that are not being acquired incrementally. Without better implementation of incremental development approaches and identification of investments using incremental development, IT expenditures will likely continue to produce disappointing results-- including large cost overruns, long schedule delays, and questionable mission-related achievements. Further, without useful information, including whether investments are following an incremental approach, on projects associated with major IT investments, OMB does not have the necessary information to oversee the extent to which projects and investments are implementing its guidance. Multiple Factors Were Commonly Identified as Enabling and Inhibiting Incremental Development: Multiple factors were identified by the five agencies as enabling and inhibiting incremental development during a 6-month period. Specifically, eight factors were identified by three or more of the five agencies in our review as enabling incremental development of IT systems, and seven factors were identified by three or more agencies as inhibiting incremental development. The enabling factor identified by all of five of the agencies was active engagement of program officials with stakeholders. The inhibiting factors identified by all five agencies were (1) the lack of sufficient, timely funding; (2) program characteristics that made rapid incremental development infeasible; and (3) the lack of stable, prioritized requirements.[Footnote 63] Eight Factors Were Commonly Identified as Enabling Incremental Development during a 6-month Period: Eight factors were identified by three or more of the five agencies in our review as contributing to the successful development of functionality in 6-month cycles. All five of the agencies in our review cited active program engagement with stakeholders. Table 5 shows the distribution of the eight factors among the agencies, and examples of how the agencies implemented them are discussed following the table. Table 5: Commonly Identified Factors Enabling Incremental Development during a 6-month Period: Enabling factor for incremental development: 1: Program officials actively engaged with stakeholders; Defense: [Check]; HHS: [Check]; DHS: [Check]; Transportation: [Check]; VA: [Check]. Enabling factor for incremental development: 2: Programs used an Agile development methodology; Defense: [Check]; HHS: [Check]; DHS: [Check]; Transportation: [Empty]; VA: [Check]. Enabling factor for incremental development: 3: Programs successfully prioritized, managed, and tested requirements; Defense: [Check]; HHS: [Check]; DHS: [Check]; Transportation: [Check]; VA: [Empty]. Enabling factor for incremental development: 4: Staff had the necessary skills and experience; Defense: [Check]; HHS: [Check]; DHS: [Check]; Transportation: [Empty]; VA: [Check]. Enabling factor for incremental development: 5: Programs successfully implemented cost and schedule estimating best practices; Defense: [Check]; HHS: [Check]; DHS: [Check]; Transportation: [Check]; VA: [Empty]. Enabling factor for incremental development: 6: Officials used various contracting strategies to increase flexibility and improve contractor oversight; Defense: [Check]; HHS: [Check]; DHS: [Check]; Transportation: [Empty]; VA: [Check]. Enabling factor for incremental development: 7: Staff used key technologies to accelerate development work; Defense: [Check]; HHS: [Check]; DHS: [Check]; Transportation: [Empty]; VA: [Empty]. Enabling factor for incremental development: 8: Programs successfully implemented risk management best practices; Defense: [Check]; HHS: [Check]; DHS: [Empty]; Transportation: [Empty]; VA: [Check]. Source: GAO analysis of agency data. Note: Where factors were cited by the same number of agencies, the factors are listed in descending order from the greatest amount of investments that cited them to the least amount. [End of table] * Program officials actively engaged with stakeholders. Officials from all five agencies explained that active engagement with program stakeholders--individuals or groups with an interest in the success of the investment--was a factor that enabled the development of functionality in 6-month cycles. For example, officials from one of the HHS investments that we reviewed stated that having strong communication between the business program, the IT program, and contractors enabled projects to move forward on schedule in a cohesive manner with clear goals and objectives. * Programs used an Agile development methodology. Officials from four of the five agencies indicated that the use of an Agile development methodology helped them to deliver functionality every 6 months. For example, Defense officials explained that the use of Agile development processes allowed software to be broken into smaller releases that were easily achieved, tested, and fielded. Additionally, VA officials stated that the agency has embraced Agile development, which has helped the department deliver functionality more quickly. Those officials also noted that merely forcing investments to use 6-month waterfall iterations would not have resulted in the same success. * Programs successfully prioritized, managed, and tested requirements. Officials from four of the five agencies identified implementation of requirements management practices--including requirements prioritization, management, and testing--as a factor that enabled the agencies to deliver functionality every 6 months. For example, Transportation officials explained that the primary factor that enabled incremental development has been obtaining clear requirements. Further, HHS officials stated that the use of a prioritized product backlog[Footnote 64] has helped teams make decisions about which requirements should be allocated to future releases. * Staff had the necessary skills and experience. Officials from four of the five agencies stated that consistent and stable program staff with experience and expertise allowed them to deliver functionality frequently. For example, DHS officials representing one of the selected investments indicated that having skilled program managers has a large impact on the success of a program. In addition, VA officials reported that their ability to retain key personnel with both functional and technical expertise has enabled them to deliver high-quality functionality rapidly. * Programs successfully implemented cost and schedule estimating best practices. Officials from four of the five agencies cited the implementation of cost and schedule estimating practices as enabling the frequent delivery of functionality. For example, Defense officials told us that development of realistic cost estimates and comprehensive program schedules helped programs to deliver functionality while meeting established performance goals. Further, Transportation officials stated that improved guidance and training on cost estimating and scheduling helped them to deliver functionality in 6- month cycles. * Officials used various contracting strategies to increase flexibility and improve contractor oversight. Officials from four of the five agencies cited the use of specific contracting strategies. For example, DHS officials indicated that they worked with contractors to modify their contracts from the traditional waterfall approach to a structure that allows for Agile development. In addition, officials from VA stated that the use of performance-based acquisitions assisted them in monitoring contractor progress towards achieving actual results against planned objectives. * Staff used key technologies to accelerate development work. Officials from three of the five agencies indicated that the use of key technologies enabled them to deliver functionality more quickly. For example, HHS officials explained that having an established cloud environment has enabled them to reduce deployment time for releases, while also providing the needed flexibility to meet their customers' changing needs. Additionally, DHS officials explained that projects, especially those using an Agile development methodology, have saved time using automated testing tools. * Programs successfully implemented risk management best practices. Officials from three agencies explained that the successful implementation of risk management practices helped them to deliver functionality more rapidly. For example, officials from one of the selected VA investments told us that the organization's risk management process allowed teams to quickly escalate risks to senior leadership so that they could be managed before their impacts were fully realized. Many of the factors that the five agencies cited are consistent with our 2011 work on factors critical to successful IT acquisitions and our 2012 work on effective practices in implementing Agile software development.[Footnote 65] In particular, our work in both areas discussed the importance of active engagement with program stakeholders, such as actively engaging with stakeholders to obtain customer feedback, and effectively managing requirements. Additionally, the eight commonly identified factors that enable the delivery of functionality in 6 months are consistent with OMB's IT Reform Plan.[Footnote 66] In particular, as previously mentioned, one high-level objective of the plan--effectively managing large-scale IT programs--aims to improve areas that impact the success rates of large IT programs by, among other things, enabling programs to use incremental development approaches. As part of this high-level objective, the plan addresses the importance of actively engaging with stakeholders, ensuring that program management professionals have proper skills and experience, and developing new contracting strategies that more effectively support incremental development. Seven Factors Were Commonly Identified as Inhibiting Incremental Development during a 6-month Period: Seven factors were identified by three or more of the five agencies as inhibiting the development of IT functionality every 6 months. The factors most commonly cited include (1) programs did not receive sufficient funding or receiving funding later than needed, (2) program characteristics made rapid delivery of functionality infeasible or impractical, (3) programs did not have stable and prioritized requirements. Table 6 shows the distribution of the seven factors among the agencies, and examples of how the factors impacted the agencies are discussed following the table. Table 6: Commonly Identified Factors Inhibiting Incremental Development during a 6-month Period: Inhibiting factor for incremental development: 1: Programs did not receive sufficient funding or received funding later than needed; Defense: [Check]; HHS: [Check]; DHS: [Check]; Transportation: [Check]; VA: [Check]. Inhibiting factor for incremental development: 2: Program characteristics made rapid delivery of functionality infeasible or impractical; Defense: [Check]; HHS: [Check]; DHS: [Check]; Transportation: [Check]; VA: [Check]. Inhibiting factor for incremental development: 3: Programs did not have stable, prioritized requirements; Defense: [Check]; HHS: [Check]; DHS: [Check]; Transportation: [Check]; VA: [Check]. Inhibiting factor for incremental development: 4: Development work was slowed by inefficient governance and oversight processes; Defense: [Check]; HHS: [Empty]; DHS: [Check]; Transportation: [Empty]; VA: [Check]. Inhibiting factor for incremental development: 5: Development schedules were impeded by procurement delays; Defense: [Check]; HHS: [Check]; DHS: [Check]; Transportation: [Empty]; VA: [Empty]. Inhibiting factor for incremental development: 6: Program staff were overutilized or lacked the necessary skills and experience; Defense: [Empty]; HHS: [Check]; DHS: [Check]; Transportation: [Empty]; VA: [Check]. Inhibiting factor for incremental development: 7: Incremental development was impeded by select technologies; Defense: [Empty]; HHS: [Check]; DHS: [Check]; Transportation: [Check]; VA: [Empty]. Source: GAO analysis of agency data. Note: Where factors were cited by the same number of agencies, the factors are listed in descending order from the greatest amount of investments that cited them to the least amount. [End of table] * Programs did not receive sufficient funding or received funding later than needed. Officials from all five departments cited insufficient funding, such as reductions caused by the fiscal year 2013 sequester, or receiving funding later than needed because of continuing resolutions. For example, Defense officials representing one of the investments we reviewed stated that furloughs brought on by the 2013 sequester significantly impacted program schedules, both in terms of lost work days and the inability to coordinate integration between software providers, resulting in overall inefficiencies. In addition, several FAA officials explained that the delivery of planned functionality was adversely affected by the uncertainty brought about by the 2013 sequester, and that future funding instability has impacted the agency's ability to plan when functionality is to be delivered. * Program characteristics made rapid delivery of functionality infeasible or impractical. Officials from all five agencies indicated that some of their programs have certain characteristics--such as the deployment of new physical infrastructure, human health and safety concerns, or external dependencies--that make the delivery of functionality in 6-month time frames infeasible or impractical. For example, DHS officials explained that their infrastructure projects cannot deliver functionality until all key activities (e.g., land acquisition, environmental assessments, site preparation, and construction) have been completed and the new infrastructure has been fully deployed, tested, and accepted. Additionally, Transportation officials reported that air traffic control systems require years of development and testing in order to ensure that the systems do not compromise airspace security. * Programs did not have stable, prioritized requirements. Officials from all five agencies stated that not having complete requirements at the beginning of their investments' development or having changes made to requirements and their relative priorities during development was a factor that negatively affected their programs' delivery of incremental functionality in 6-month periods. For example, HHS officials representing one of the selected investments explained that final rules detailing eligibility for a medical program were not completed until late in the development cycle; consequently, the program could not define the corresponding business requirements in time to meet the development schedule. Further, VA officials stated that schedules are disrupted when stakeholders identify new program requirements that must be delivered before others. * Development work was slowed by inefficient governance and oversight processes. Officials from three of the five agencies cited inefficient governance and oversight processes. For example, DHS officials representing the Office of the CIO stated that the current DHS governance model was not conducive to frequent delivery of functionality. To illustrate, those officials noted that it can take up to 2 months to schedule a meeting with DHS review boards prior to releasing functionality. However, a DHS official from Program Accountability and Risk Management disagreed with this statement, explaining that DHS's acquisition review boards perform reviews very quickly, and that any delays in completing these reviews are attributable to investments being unprepared. Further, DHS Office of the CIO officials suggested that governance over programs using an Agile development methodology should be performed at the lowest practicable level of the organization. In addition, VA officials explained that, although the periodic approvals required of various department groups are useful, these approvals can slow the testing and release processes and could use further streamlining. * Development schedules were impeded by procurement delays. Officials from three of the five agencies explained that procurement delays-- such as delays in getting contracts awarded or approving contract modifications--contributed to difficulties in delivering incremental functionality. For example, DHS officials explained the process of planning for an acquisition, developing solicitations, selecting contractors, and, in some cases, addressing protests, are not conducive to delivering functionality in 6-month cycles. * Program staff were overutilized or lacked the necessary skills and experience. Officials from three of the five agencies indicated that they did not have enough program staff with the expertise and experience necessary to deliver functionality every 6 months. For example, officials representing one of the HHS investments stated that the loss of key personnel and the lack of staff with knowledge and skill in key disciplines, such as incremental development, have negatively impacted the agency's ability to deliver functionality. * Incremental development was impeded by select technologies. Officials from three of the five agencies in our review explained that the software and tools they selected for their programs either introduced delays in development or were ultimately not usable by the program. Transportation officials representing one of the selected investments reported that their chosen technology added a new level of complexity to storage and data processing that required all development and testing to be completed before the implementation could occur. As a result, those officials told us that all functionality had to be deployed in one release, instead of being spread out in stages. Many of the factors that the five agencies identified as inhibiting delivery of functionality in 6-month increments were consistent with our work on the challenges we identified in the application of Agile software development methods.[Footnote 67] For example, we noted that one challenge in deploying software developed using an Agile development method is that traditional governance and oversight activities were difficult to execute within an iterative time frame. In particular, one agency official stated that completing the necessary reviews within the short, fixed time frame of an Agile iteration was difficult because reviewers followed a slower waterfall schedule with reviews that could take months to perform after the completion of an iteration. This caused delays for iterations that needed such reviews within the few weeks of the iteration. We also noted that procurement practices sometimes delay Agile development. Further, OMB's IT Reform Plan also identified some of the same problems that inhibit incremental development and described solutions to address them.[Footnote 68] In particular, as previously mentioned, one item of the plan was to improve funding of programs by working with Congress to create IT budget models that promote IT budget flexibility (such as multiyear budgets or revolving funds). In addition, the plan addresses the importance of streamlining IT governance by strengthening the quality and timing of oversight activities. The factors identified in this report as enabling and inhibiting incremental development could help agencies address the challenges they face in incrementally acquiring IT. Conclusions: Given the enormous size of the federal government's investment in IT and the often disappointing results from IT development efforts--many of which are attributable to the use of a "big bang" approach--finding ways to improve the quality and timeliness of agencies' investments is important. Congress, OMB, and our work support the use of incremental development practices. However, although the selected agencies have developed policies that address incremental development, most of the agencies' policies have significant weaknesses. With the exception of VA, these policies do not fully define functionality, and do not have a complete process for ensuring that the agencies' investments are developed incrementally, including the use of TechStat sessions to enforce compliance with incremental development policies. In the absence of such policies, agencies continue to run the risk of failing to deliver major investments in a cost-effective and efficient manner. Regarding implementation of incremental development, slightly more than one-fourth of selected investments planned to deliver functionality every 6 months--and less than one-half planned to do so every 12 months. Thus, delivering functionality every 6 months is not an appropriate requirement for all agencies given current performance. Requiring the delivery of functionality every 12 months, consistent with OMB's IT Reform Plan, would be an appropriate starting point and be a substantial improvement. Further, since there are three types of investments for which it may not always be practical or necessary to expect functionality to be delivered in 6-month cycles, this raises questions about whether OMB's requirement for shorter delivery cycles should be applied to all investments or whether investments should be allowed the latitude to determine a more appropriate time frame. The lack of progress in updating policies and implementing OMB's guidance was enabled by weaknesses in OMB's guidance. In the absence of agency and OMB use of TechStat sessions to ensure compliance with incremental development policy, investments will continue to be at risk of not delivering promised capabilities on time and within budget. Until OMB clearly and explicitly disseminates guidance with realistic goals and clear expectations, and agencies update their policies to reflect this guidance, agencies may not consistently adopt incremental development approaches, and IT expenditures will continue to produce disappointing results--including sizable cost overruns and schedule slippages, and questionable progress in meeting mission goals and outcomes. Additionally, without useful information on how often projects are delivering functionality, it will be difficult for OMB to identify investments that are not implementing its guidance. Further, dissemination of the factors identified in this report as enabling and inhibiting incremental development may help federal agencies address the challenges they face in acquiring IT investments faster and more efficiently. Recommendations for Executive Action: We recommend that the Director of the Office of Management and Budget direct the Federal Chief Information Officer to take the following two actions. * Update, and clearly and explicitly issue incremental development guidance that addresses the following three components: - requires projects associated with major IT investments to deliver incremental functionality at least every 12 months, with the exception of the three types of investments identified in this report; - specifies how agencies are to define the project functionality that is to be delivered; and: - requires agencies to define a process for enforcing compliance with incremental functionality delivery, such as the use of TechStat sessions. * Require agencies to clearly identify on exhibit 300 submissions whether, for each project, functionality will be delivered within the time frames called for by this incremental development guidance, and to provide justification for projects that do not plan to do so. We further recommend that the Secretaries of Defense, Health and Human Services, Homeland Security, and Transportation take the following two actions: * modify, finalize, and implement their agencies' policies governing incremental development to ensure that those policies comply with OMB's guidance, once that guidance is made available; and: * when updating their policies, consider the factors identified in this report as enabling and inhibiting incremental development. We also recommend that the Secretary of Veterans Affairs consider incorporating the factors identified in this report as enabling and inhibiting incremental development in the department's related policy. Agency Comments and Our Evaluation: We received comments on a draft of this report from OMB and the five agencies in our review. OMB agreed with one recommendation and partially disagreed with the other; Defense generally concurred with the report; HHS neither agreed nor disagreed with the report's recommendations; DHS agreed with our recommendations; Transportation did not agree with the recommendations in that it did not believe the department should be dependent on OMB first taking action; and VA generally agreed with the report's conclusions and concurred with our recommendation. Each agency's comments that we received are discussed in more detail below. * In comments provided via e-mail on April 15, 2014, staff in OMB's Office of General Counsel, on behalf of OMB, stated that the agency agreed with one of our recommendations and partially disagreed with the other. Specifically, OMB agreed with our recommendation to require agencies to clearly identify on exhibit 300 submissions whether functionality will be delivered within the time frames called for by this incremental development guidance. OMB stated that it agreed with our recommendation to update and issue incremental development guidance, but did not agree that this guidance should require major IT investments to deliver incremental functionality at least every 12 months. OMB explained that changing the requirement from 6 to 12 months would reduce the emphasis on incremental development that it has been advocating. OMB also noted that it believes requiring investments to deliver functionality every 6 months is an appropriate governmentwide goal and said that updating and clarifying its guidance on incremental development will make it easier for agencies to meet this target. However, as we state in this report, slightly more than one-fourth of selected investments planned to deliver functionality every 6 months--and less than one-half planned to do so every 12 months. Additionally, there are three types of investments for which it may not always be practical or necessary to expect functionality to be delivered in 6-month cycles. Thus, we continue to believe that delivering functionality every 6 months is not an appropriate requirement for all agencies and that requiring the delivery of functionality every 12 months, consistent with OMB's IT Reform Plan, is a more appropriate starting point. We therefore maintain that OMB should require projects associated with major IT investments to deliver functionality at least every 12 months. * In written comments, Defense stated that it generally concurred with the report and outlined planned actions to address the recommendations. However, Defense explained that many of its IT investments are consistent with the three types of investments for which it may not always be practical or necessary to expect functionality to be delivered in 6-month cycles or have other exceptional circumstances. Given these issues, Defense stated that requiring the delivery of functionality every 12 months is not a useful management constraint. However, as we state in our report, many failed projects have been broadly scoped in that they aim to deliver their capabilities several years after initiation. Additionally, the Defense Science Board reported that Defense's acquisition process for IT systems was too long, was ineffective, and did not accommodate the rapid evolution of IT. In order to resolve these issues, we continue to believe that investments and their projects should deliver functionality more frequently. In addition, the existence of many investments that are consistent with the three types of investments identified in this report for which it may not be practical to deliver functionality quickly does not excuse Defense from the requirement of delivering functionality every 12 months. Consistent with our recommendation to OMB, investments and projects that meet these three exceptions should be exempt from the requirement, but all other investments and projects should aim to meet this requirement. Further, we expect that Defense's actions to implement our recommendation to consider the factors identified in this report that enable and inhibit incremental development will help its investments and projects deliver functionality more rapidly. As such, we continue to believe that requiring the delivery of functionality every 12 months is an appropriate goal for all federal agencies, including Defense. Defense's comments are reprinted in appendix III. The department also provided technical comments, which we have incorporated in the report as appropriate. * In comments provided via e-mail on April 15, 2014, an official from HHS's Office of the Assistant Secretary for Legislation, on behalf of HHS, stated that the department had no comments. * In written comments, DHS stated that it concurred with our recommendations and outlined planned actions to address the recommendations. DHS's comments are reprinted in appendix IV. The department also provided technical comments, which we have incorporated in the report as appropriate. * In comments provided via e-mail on April 15, 2014, Transportation's Deputy Director of Audit Relations, on behalf of Transportation, stated that the department would prefer to have specific recommendations and deliverables so that it can achieve success in closing them. Specifically, the department explained that relying on another agency to concur with one of our recommendations before Transportation can take action leaves the department with the potential challenge of a recommendation that cannot be implemented. However, as previously stated, OMB agrees with our recommendation to update and issue incremental guidance, meaning that OMB has committed to taking the actions necessary to enable Transportation to begin addressing our recommendation. Additionally, our recommendation to consider the factors identified in this report as enabling and inhibiting incremental development when updating incremental development policies does not require another agency to take action before Transportation can implement it. Accordingly, we continue to believe that our recommendations are warranted and can be implemented. * In written comments, VA stated that it generally agreed with the report's conclusions, that it concurred with our recommendation to the department, and that it will review its existing policy for incremental development and consider incorporating the factors identified as enabling and inhibiting incremental development. VA's comments are reprinted in appendix V. The department also provided technical comments, which we have incorporated in the report as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to interested congressional committees; the Director of the Office of Management and Budget; and the Secretaries of the Departments of Defense, Health and Human Services, Homeland Security, Transportation, and Veterans Affairs. In addition, the report will also be available at no charge on our website at [hyperlink, http://www.gao.gov/]. If you or your staffs have any questions on matters discussed in this report, please contact me at (202) 512-9286 or pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VI. Signed by: David A. Powner: Director: Information Technology Management Issues: [End of section] Appendix I: Objectives, Scope, and Methodology: Our objectives for this review were to (1) assess whether selected agencies have established policies for incremental information technology (IT) development, (2) determine whether selected agencies are using incremental development approaches to manage their IT investments, and (3) identify the key factors that enabled and inhibited the selected agencies' abilities to effectively use incremental development approaches to manage their IT investments. In conducting our review, we selected five agencies and a total of 89 investments from those agencies. To choose the agencies, we identified the five agencies with the largest IT budgets for development, modernization, and enhancement on major IT investments in fiscal years 2013 and 2014 as reported in the Office of Management and Budget's (OMB) fiscal year 2014 exhibit 53.[Footnote 69] Those agencies are the Departments of Defense (Defense), Health and Human Services (HHS), Homeland Security (DHS), Transportation (Transportation), and Veterans Affairs (VA). To choose the agencies' investments, we identified the 98 major IT investments for which the selected agencies planned to spend more than 50 percent of the investments' fiscal year 2013 and 2014 budgets on development, modernization, and enhancement as reported in OMB's fiscal year 2014 exhibit 53. We removed 9 investments because, after reporting their planned budgets for OMB's fiscal year 2014 exhibit 53, these investments changed their budgets so that they no longer planned to spend more than 50 percent of the investments' fiscal year 2013 and 2014 budgets on development, modernization, and enhancement.[Footnote 70] The final 89 investments are identified in appendix II. The investments selected for the review account for about 58 percent of the development, modernization, and enhancement spending on all federal agencies' major IT investments for fiscal years 2013 and 2014 reported in OMB's exhibit 53 for fiscal year 2014. To address our first objective, we reviewed OMB guidance related to the use of incremental development, as well as industry guidance, and identified three key components of incremental development that agencies should include in their policies. In order to identify these components, we reviewed OMB's guidance[Footnote 71] and leading industry guidance on institutionalizing processes throughout an organization.[Footnote 72] Our analysis identified three key policy components that will help agencies effectively implement OMB's requirement for incremental development. * Require delivery of functionality every 6 months. According to the Software Engineering Institute's (SEI) Capability Maturity Model® Integration for Acquisition (CMMI-ACQ), as part of institutionalizing a process, organizations should document their processes, to include defining standards and requirements.[Footnote 73] According to OMB budget guidance, projects associated with major IT investments must deliver functionality every 6 months.[Footnote 74] * Define functionality. As previously stated, according to SEI's CMMI- ACQ, as part of institutionalizing a process, organizations should document that process, to include defining standards and requirements.[Footnote 75] * Define a process for enforcing compliance. According to SEI's CMMI- ACQ, as part of institutionalizing a process, organizations should document that process, including management review activities (e.g., taking corrective action when requirements and objectives are not being satisfied).[Footnote 76] Additionally, OMB Circular A-130 requires agencies to ensure that investments comply with their policies.[Footnote 77] Further, according to the President's Fiscal Year 2014 Budget, agencies are to use their TechStat processes to identify investments that are not being acquired incrementally and undertake corrective actions.[Footnote 78] At each selected agency, we then analyzed agency policies for incremental development and compared these policies to the three components identified above. For each agency, each policy component was assessed as either being not met--the agency did not provide evidence that addressed the component or provided evidence that minimally addressed the component; partially met--the agency provided evidence that addressed about half or a large portion of the component; or fully met--the agency provided evidence that addressed the component. We also interviewed officials from OMB and the five selected agencies to obtain information about their current and future incremental development policies. To address our second objective, we administered a data collection instrument to each of the selected investments about how often each investment planned to deliver functionality during fiscal years 2013 and 2014. We then analyzed information obtained from data collection instruments describing how often the selected investments planned to deliver functionality. We prepopulated these instruments with data obtained from OMB's fiscal year 2014 exhibit 53, as well as each investment's exhibit 300 data, which describe the investments' projects and how often each project plans to deliver functionality. We asked officials from each investment to verify the accuracy and completeness of the data and to make corrections where needed. Because the exhibit 300 data did not always describe the investments' plans for delivering functionality in fiscal year 2014, we asked the officials to indicate for each project whether they planned to deliver functionality in the first half of fiscal year 2014 (i.e., from October 1, 2013, to March 31, 2014) and in the second half of fiscal year 2014 (i.e., from April 1, 2014, to September 30, 2014). We also asked the investments to provide their life-cycle cost estimates and, for each project, the development methodology used and phase of the acquisition life cycle. Using the information obtained through the data collection instruments, we determined the extent to which the selected investments and their projects planned to meet OMB's guidance on incremental development. To assess whether investments had planned to deliver functionality every 6 months, we determined whether the selected investments planned to deliver functionality in each of the following four time frames: (1) the first half of fiscal year 2013 (i.e., from October 1, 2012, to March 31, 2013), (2) the second half of fiscal year 2013 (i.e., from April 1, 2013, to September 30, 2013), (3) the first half of fiscal year 2014, and (4) the second half of fiscal year 2014. To determine whether investments had planned to deliver functionality every 12 months, we analyzed whether the selected investments planned to deliver functionality in each of the following two time frames: (1) fiscal year 2013 and (2) fiscal year 2014. We presented our results to the five selected agencies and OMB and solicited their input and explanations for the results. To determine the reliability of the exhibit 300 data, we performed three steps. First, as previously mentioned, we asked officials from each investment to verify the accuracy and completeness of these data and provide the correct information where needed. Second, we removed projects that were not intended to deliver functionality and activities that were inaccurately classified as resulting in the delivery of functionality. To do so, we compared the descriptions of the projects and activities with a definition of functionality that we developed. To develop this definition, we reviewed OMB[Footnote 79] and agency guidance,[Footnote 80] as well as leading practices. [Footnote 81] We defined functionality as follows: the implementation of IT requirements that is intended to either (1) ultimately be used by one or more customers in production (actual deployment may occur at a later date than when the functionality is reported as being delivered) or (2) be a delivery of a prototype or pilot. We then assessed the descriptions of the activities and projects agencies reported in their exhibit 300 submissions against our definition. We presented the activities and projects that did not meet our definition to the selected agencies and solicited their input and explanations. Third, where there was a conflict between the exhibit 300 data and agencies' answers to our questions regarding plans for delivering functionality in fiscal year 2014, we presented the conflict to the agencies and obtained clarification. We determined that the data were sufficiently reliable for the purpose of this report, which is to determine the extent to which the selected investments planned to deliver functionality every 6 and 12 months, during fiscal years 2013 and 2014. To address our third objective, as part of the previously mentioned data collection instrument, we asked the five selected agencies and investments to identify the key factors that have both enabled and inhibited their efforts to deliver functionality for their major IT investments every 6 months, consistent with OMB guidance. We also asked these questions of officials from each of the selected agencies' Office of the Chief Information Officer (CIO). Because of the open- ended nature of our questions, we conducted a content analysis of the information we received in order to identify common factors. We then totaled the number of times each factor was mentioned by department and agency officials, choosing to report on the factors that were identified by three or more agencies. We then compared these factors to our recent work on critical factors underlying the success of major IT acquisitions,[Footnote 82] as well as practices for and challenges in effectively applying Agile development methods.[Footnote 83] Additionally, we compared the factors to OMB's 25 Point Implementation Plan to Reform Federal Information Technology Management.[Footnote 84] Further, because Defense guidance encourages investments to deliver functionality every 12-18 months where possible,[Footnote 85] we asked the selected investments and officials from Defense's Office of the CIO to identify the key factors that have both enabled and inhibited their efforts to deliver functionality for their major IT investments every 12-18 months. We compared the information we received to the eight factors the five selected agencies commonly identified as enabling incremental development during a 6-month period and the seven factors commonly identified by the five agencies as inhibiting incremental development during a 6-month period. We also performed a content analysis of the information we received in order to identify additional factors. We conducted this performance audit from May 2013 to May 2014, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of section] Appendix II: Selected Investments' Plans for Delivering Functionality: Below is the list of investments that are included in this review, as well as whether each planned to deliver functionality every 6 and 12 months for fiscal years 2013 and 2014. Table 7: Selected Investments and Associated Plans for Delivering Functionality Every 6 and 12 Months, for Fiscal Years 2013 and 2014: Agency: Defense; Investment name: Advanced Field Artillery Tactical Data System; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: Air Force Nuclear Command, Control and Communications Minimum Essential Emergency Communications Network Modernization; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Air Force Integrated Personnel and Pay System; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Area Common User System Modernization; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Aviation Tactical Communication Systems; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: Base Information Transport Infrastructure Wired; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Battle Control System Fixed; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Common Aviation Command and Control System; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Consolidated Afloat Networks Enterprise Service; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Defense Agencies Initiative; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Defense Enterprise Accounting and Management System; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: Distributed Common Ground System Army; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Energy Convergence; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Global Combat Support System Army; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: Global Command And Control System Army; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: High Performance Computing Modernization Program; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Installation Information Infrastructure Modernization Program; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Integrated Electronic Health Record; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: Joint Battle Command-Platform; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Joint Personnel Identification Version 2; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Joint Precision Approach and Landing System; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Joint Tactical Radio System Airborne and Maritime/Fixed Station; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Joint Tactical Radio System Handheld, Manpack, and Small Form Fit Radios; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Joint Tactical Radio System Network Enterprise Domain; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: Joint Space Operations Center Mission System; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Launch Test Range System; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Maneuver Control System, V6.4; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: Mid-tier Networking Vehicular Radio; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Mission Planning System Increment IV; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: Multifunctional Information Distribution System; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Check]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: Navstar Global Positioning System; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Tactical Data Link System; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Teleport Generation 3; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Theater Battle Management Core System; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: Voice Switching System; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Warfighter Information Network Tactical Increment 2; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: Warfighter Information Network Tactical Increment 3; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Agency: Health and Human Services; Investment name: Centers for Disease Control and Prevention Public Health Information Network: BioSense; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Centers for Medicare and Medicaid Services Business Rules Enterprise Service; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Check]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: Centers for Medicare and Medicaid Services Center for Consumer Information and Insurance Oversight Enterprise IT Data Management Investment; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Check]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: Centers for Medicare and Medicaid Services Center for Consumer Information and Insurance Oversight Healthcare Insurance Exchange IT Investment; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Check]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: Centers for Medicare and Medicaid Services Healthcare Quality End Stage Renal Disease Systems; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Check]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: Centers for Medicare and Medicaid Services International Classification of Diseases, 10th Revision Initiative; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Centers for Medicare and Medicaid Services Innovation Core Systems; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Centers for Medicare and Medicaid Services Medicaid and Children's Health Insurance Program Business Information and Solutions; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: Centers for Medicare and Medicaid Services Master Data Management; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: U.S. Food and Drug Administration Center for Drug Evaluation and Research MedWatch Plus; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Check]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: U.S. Food and Drug Administration's Office of Regulatory Affairs Automated Laboratory Management; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Check]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: U.S. Food and Drug Administration's Office of Regulatory Affairs Mission Accomplishments and Regulatory Compliance Services; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Check]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: Federal Health Architecture; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Check]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: Heath Resources and Services Administration Bureau of Clinician Recruitment and Service Management Information System Solution; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Check]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Agency: Homeland Security; Investment name: U.S. Customs and Border Protection Integrated Fixed Towers; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: U.S. Customs and Border Protection Tactical Communications Modernization; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Mount Weather Information Technology Services; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Federal Emergency Management Agency Disaster Assistance Improvement Plan; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: Federal Emergency Management Agency Integrated Public Alert and Warning System; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Check]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: Federal Emergency Management Agency National Flood Insurance Program Information Technology Phoenix; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Immigration and Customs Enforcement TECS Modernization; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: National Protection and Programs Directorate Continuous Diagnostics and Mitigation; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: National Protection and Programs Directorate Next Generation Networks Priority Services; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: Transportation Security Administration Technology Infrastructure Modernization Program; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: U.S. Coast Guard Logistics Information Management System; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: U.S. Citizenship and Immigration Services Transformation; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Check]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Agency: Transportation; Investment name: Delphi Version Two; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: Federal Aviation Administration (FAA) Terminal Automation Modernization and Replacement Program; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Check]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: FAA Wide Area Augmentation System; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: FAA En Route Automation Modernization D Position Upgrade and System Enhancements; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: FAA Aviation Safety Knowledge Management; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: FAA System Approach for Safety Oversight; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: FAA Traffic Flow Management; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Check]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: FAA System Wide Information Management; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Transportation; Investment name: FAA Automatic Dependent Surveillance Broadcast; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Check]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: FAA Runway Status Lights; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Check]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: FAA Data Communications Next Generation Air Transportation System Support; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: FAA Next Generation Air Transportation System National Airspace System Voice System; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: FAA Common Support Services Weather; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: FAA Next Generation Air Transportation System Air/Ground Communications Segment 2; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Check]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: FAA Next Generation Air Transportation System Research and Development Trajectory Based Operations; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: FAA Next Generation Air Transportation System Research and Development Arrivals/Departures at High Density Airport; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: FAA Next Generation Air Transportation System Research and Development Flexibility in Terminal Environment; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: FAA Next Generation Air Transportation System Research and Development Collaborative Air Traffic Management; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: FAA Next Generation Air Transportation System Research and Development Reduce Weather Impact; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Investment name: FAA Next Generation Air Transportation System Research and Development Demonstrations and Infrastructure; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Empty]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Empty]. Agency: Veterans Affairs; Investment name: Benefits Legacy Veterans Services Network; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Check]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: InterAgency 21st Century Core; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Check]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: InterAgency 21st Century-One Vet; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Check]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: Medical 21st Century My HealtheVet; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Check]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: Medical 21st Century TeleHealth; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Check]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Investment name: Medical Legacy; Plans to deliver functionality every 6 months in fiscal years 2013 and 2014: [Check]; Plans to deliver functionality every 12 months in fiscal years 2013 and 2014: [Check]. Source: GAO analysis of agency data. [End of table] [End of section] Appendix III: Comments from the Department of Defense: Assistant Secretary of Defense: Acquisition: 3600 Defense Pentagon: Washington, DC 20301-3600: April 21, 2014: Mr. David A. Powner: Director, Information Technology Management Issues: U.S. Government Accountability Office: 441 G Street, N.W. Washington, DC 20548: Dear Mr. Powner: This is the Department of Defense (000) response to the Government Accountability Office (GAO) Draft Report, GAO-14-361, "Information Technology: Agencies Need to Establish and Implement Incremental Development Policies," dated March 19, 2014 (GAO Code 311289). Your staff conducted a thorough study and produced a useful report with which we generally concur. Comments on the report recommendations are enclosed. Sincerely, Signed by: Katrina McFarland: Enclosure: As stated: GAO-14-361 (GAO Code 311289): "Information Technology: Agencies Need To Establish And Implement Incremental Development Policies" Department Of Defense Comments To The GAO Recommendation: Recommendation 1: The GAO recommends that the Secretary of Defense modify, finalize, and implement policies governing incremental development to ensure that those policies comply with OMB's guidance, once that guidance is made available. DoD Response: The GAO has recommended that OMB issue guidance for agencies to define delivered functionality, and to delineate a compliance enforcement process. GAO also recommends the OMB guidance require delivery of incremental functionality at least every 12 months, with three exceptions: 1) investments in life-cycle phases other than acquisition, 2) IT infrastructure investments, and 3) research and development investments. Many DoD IT investments fit those exceptions or have other exceptional circumstances that suggest a 12 month functional delivery requirement is not a useful management constraint. For example, operational testing of each delivered increment adds significant burden on DoD resources and challenges most program schedules. Iterations of that testing process should occur only when it benefits the goal to deliver effective, suitable, and survivable capability, and not to satisfy a calendar requirement. DoD will coordinate with OMB to develop guidance with the latitude to determine appropriate delivery time frames and a responsive implementing policy. Recommendation 2: The GAO recommends that the Secretary of Defense, when updating the policies, consider the factors identified in this report as enabling and inhibiting incremental development. DoD Response: The GAO identified eight enabling and seven inhibiting factors that commonly contribute to the success or failure to deliver functionality in 6-month cycles. These factors are cited as consistent with other GAO work and other works in the literature of IT management. They are no less applicable to lengthier delivery cycles and DoD will consider them when updating IT development policy. [End of section] Appendix IV: Comments from the Department of Homeland Security: U.S. Department of Homeland Security: Washington, DC 20528: April 23, 2014: David A. Powner: Director, Information Technology Management Issues: U.S. Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Re: Draft Report GAO-14-361, "Information Technology: Agencies Need to Establish and Implement Incremental Development Policies" Dear Mr. Powner: Thank you for the opportunity to review and comment on this draft report. The U.S. Department of Homeland Security (DHS) appreciates the U.S. Government Accountability Office's (GAO's) work in planning and conducting its review and issuing this report. The Department is pleased to note GAO's positive acknowledgment that multiple factors were commonly identified as enabling and inhibiting incremental development. Specifically, eight factors were identified by three or more of the five agencies during GAO's review as enabling incremental development of Information Technology (IT) systems and seven factors were identified by three or more agencies as inhibiting incremental development. The enabling factor identified by all of five of the agencies was active engagement of program officials with stakeholders. The inhibiting factors identified by all five agencies were: (1) the lack of sufficient, timely funding; (2) program characteristics that made rapid incremental development infeasible; and (3) the lack of stable, prioritized requirements. The draft report contained two recommendations directed to DRS with which the Department concurs. Specifically, GAO recommended that the Secretary of Homeland Security: Recommendation 1: Modify, finalize, and implement its agencies' policies governing incremental development to ensure that those policies comply with OMB's guidance, once that guidance is made available. Response: Concur. The DRS Office of the Chief Information Officer (OCIO) will modify, finalize, and implement DRS policies and guidance governing incremental development to ensure that those policies and guidance comply with the Office of Management and Budget's guidance, once that guidance is made available, so long as it aligns with the findings of the GAO report. The DRS Agile Instruction is being delivered to the DRS Directives Manager for DRS Component review and approval in quarter (Q) 3 of fiscal year (FY) 2014 with an anticipated implementation in Q4 FY 2014. Estimated Completion Date (ECD): September 30, 2014. Recommendation 2: When updating its policies, consider the factors identified in this report as enabling and inhibiting incremental development. Response: Concur. DRS OCIO will update DRS modular development guidance to encourage the factors identified in the report as enabling incremental development, and will include strategies to discourage or minimize the factors identified in the report as inhibiting incremental development. The DRS Agile Instruction is being delivered to the DRS Directives Manager for DRS Component review and approval in Q3 FY 2014 with an anticipated implementation in Q4 FY 2014. ECD: September 30,2014. Again, thank you for the opportunity to review and comment on this draft report. Technical comments were previously submitted under separate cover. Please feel free to contact me if you have any questions. We look forward to working with you in the future. Sincerely, Signed by: Jim H. Crumpacker, CIA, CFE: Director: Departmental GAO-OIG Liaison Office: [End of section] Appendix V: Comments from the Department of Veterans Affairs: Department of Veterans Affairs: Washington DC 20420: April 17, 2014: Mr. David A. Powner: Director, Information Technology Management Issues: U.S. Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Dear Mr. Powner: The Department of Veterans Affairs (VA) has reviewed the Government Accountability Office's (GAO) draft report, "Information Technology: Agencies Need to Establish and Implement Incremental Development Policies" (GAO-14-361). VA generally agrees with GAO's conclusions and concurs with GAO's recommendation to the Department. The enclosure speCifically addresses GAO's recommendation and provides an action plan. VA appreciates the opportunity to comment on your draft report. Sincerely, Signed by: Jose Riojas: Chief of Staff: Enclosure: Department of Veterans Affairs (VA) Response to Government Accountability Office (GAO) Draft Report "Information Technology: Agencies Need to Establish and Implement Incremental Development Policies" (GAO-14-361) GAO Recommendation: We also recommend that the Secretary of Veterans Affairs consider incorporating the factors Identified in this report as enabling and inhibiting Incremental development in the department's related policy. VA Comment: Concur. VA's Department-wide policy for incremental development is VA Directive 6071, Project Management Accountability System (PMAS). This directive mandates the use of PMAS for all information technology development projects. VA's PMAS Business Office will review our existing directive and consider incorporating the factors identified as enabling and inhibiting incremental development. This review will be completed by June 30, 2014. [End of section] Appendix VI: GAO Contact and Staff Acknowledgments: GAO Contact: David A. Powner, (202) 512-9286 or pownerd@gao.gov: Staff Acknowledgments: In addition to the contact name above, individuals making contributions to this report included Dave Hinchman (Assistant Director), Deborah A. Davis (Assistant Director), Rebecca Eyler, Kaelin Kuhn, Jamelyn Payan, Meredith Raymond, Kevin Smith, Andrew Stavisky, and Kevin Walsh. [End of section] Footnotes: [1] GAO, OMB and Agencies Need to More Effectively Implement Major Initiatives to Save Billions of Dollars, [hyperlink, http://www.gao.gov/products/GAO-13-796T] (Washington, D.C.: July 25, 2013). [2] 41 U.S.C. § 2308; OMB, Management of Federal Information Resources, Circular No. A-130 Revised. [3] OMB, Guidance on Exhibits 53 and 300--Information Technology and E- Government (2013); Guidance on Exhibits 53 and 300--Information Technology and E-Government (2012); Guidance on Exhibit 300--Planning, Budgeting, Acquisition, and Management of Information Technology Capital Assets (2011). [4] "Major IT investment" means a system or an acquisition requiring special management attention because it has significant importance to the mission or function of the government; significant program or policy implications; high executive visibility; high development, operating, or maintenance costs; an unusual funding mechanism; or is defined as major by the agency's capital planning and investment control process. [5] OMB, Management of Federal Information Resources, Circular No. A- 130 Revised. [6] OMB, Guidance on Exhibits 53 and 300--Information Technology and E- Government (2013); Executive Office of the President of the United States, OMB, Fiscal Year 2014 Analytical Perspectives: Budget of the U.S. Government, 354; OMB, Contracting Guidance to Support Modular Development (June 14, 2012); Guidance on Exhibits 53 and 300-- Information Technology and E-Government (2012); and Guidance on Exhibit 300--Planning, Budgeting, Acquisition, and Management of Information Technology Capital Assets (2011). [7] Software Engineering Institute (SEI), Capability Maturity Model® Integration for Acquisition (CMMI-ACQ), Version 1.3 (November 2010). [8] GAO, Information Technology: Additional Executive Review Sessions Needed to Address Troubled Projects, [hyperlink, http://www.gao.gov/products/GAO-13-524] (Washington, D.C.: June 13, 2013). [9] See, for example, GAO, Secure Border Initiative: DHS Needs to Strengthen Management and Oversight of Its Prime Contractor, [hyperlink, http://www.gao.gov/products/GAO-11-6] (Washington, D.C.: Oct. 18, 2010); Secure Border Initiative: DHS Needs to Reconsider Its Proposed Investment in Key Technology Program, [hyperlink, http://www.gao.gov/products/GAO-10-340] (Washington, D.C.: May 5, 2010); Secure Border Initiative: DHS Needs to Address Testing and Performance Limitations That Place Key Technology Program at Risk, [hyperlink, http://www.gao.gov/products/GAO-10-158] (Washington, D.C.: Jan. 29, 2010); Secure Border Initiative: DHS Needs to Address Significant Risks in Delivering Key Technology Investment, [hyperlink, http://www.gao.gov/products/GAO-08-1086] (Washington, D.C.: Sept. 22, 2008); and Secure Border Initiative: SBInet Expenditure Plan Needs to Better Support Oversight and Accountability, [hyperlink, http://www.gao.gov/products/GAO-07-309] (Washington, D.C.: Feb. 15, 2007). [10] [hyperlink, http://www.gao.gov/products/GAO-10-340]. [11] GAO, OPM Retirement Modernization: Longstanding Information Technology Management Weaknesses Need to Be Addressed, [hyperlink, http://www.gao.gov/products/GAO-12-226T] (Washington, D.C.: Nov. 15, 2011). [12] GAO, Office of Personnel Management: Retirement Modernization Planning and Management Shortcomings Need to Be Addressed, [hyperlink, http://www.gao.gov/products/GAO-09-529] (Washington, D.C.: Apr. 21, 2009); Office of Personnel Management: Improvements Needed to Ensure Successful Retirement Systems Modernization, [hyperlink, http://www.gao.gov/products/GAO-08-345] (Washington, D.C.: Jan. 31, 2008); Comments on the Office of Personnel Management's February 20, 2008 Report to Congress Regarding the Retirement Systems Modernization, [hyperlink, http://www.gao.gov/products/GAO-08-576R] (Washington, D.C.: Mar. 28, 2008); and Office of Personnel Management: Retirement Systems Modernization Program Faces Numerous Challenges, [hyperlink, http://www.gao.gov/products/GAO-05-237] (Washington, D.C.: Feb. 28, 2005). [13] GAO, DOD Business Transformation: Improved Management Oversight of Business System Modernization Efforts Needed, [hyperlink, http://www.gao.gov/products/GAO-11-53] (Washington, D.C.: Oct. 7, 2010); and DOD Financial Management: Implementation Weaknesses in Army and Air Force Business Systems Could Jeopardize DOD's Auditability Goals, [hyperlink, http://www.gao.gov/products/GAO-12-134] (Washington, D.C.: Feb. 28, 2012). [14] The waterfall model begins with requirements development and continues sequentially through other phases--design, build, and testing--using the output of one phase as the input to the next to develop a finished product at the end. There is only one product release at the end of testing. [15] Defense, Defense Science Board, Report of the Defense Science Board Task Force on Department of Defense Policies and Procedures for the Acquisition of Information Technology (Washington, D.C.: March 2009). [16] See Clinger-Cohen Act of 1996, Pub. L. No. 104-106 § 5202, 110 Stat. 186, 690 (1996), codified at 41 U.S.C. § 2308; see also 48 C.F.R. § 39.103 (Federal Acquisition Regulation); OMB, Management of Federal Information Resources, Circular No. A-130 Revised. [17] OMB, Management of Federal Information Resources, Circular No. A- 130 Revised. OMB's 2012 and 2013 guidance reaffirmed and strengthened these requirements. Executive Office of the President of the United States, OMB, Fiscal Year 2014 Analytical Perspectives: Budget of the U.S. Government, 354; and OMB, Contracting Guidance to Support Modular Development (June 14, 2012). [18] OMB, 25 Point Implementation Plan to Reform Federal Information Technology Management (Washington, D.C.: Dec. 9, 2010). [19] OMB, Guidance on Exhibit 300--Planning, Budgeting, Acquisition, and Management of Information Technology Capital Assets (2011). [20] OMB, Guidance on Exhibits 53 and 300--Information Technology and E-Government (2013); Guidance on Exhibits 53 and 300--Information Technology and E-Government (2012). [21] 44 U.S.C. § 3501, et. seq. [22] 40 U.S.C. § 11101, et. seq. [23] 44 U.S.C. § 3506. [24] 40 U.S.C. § 11315. [25] 41 U.S.C. § 2308. The Federal Acquisition Regulation was amended in February 1998 to reflect this provision. 48 C.F.R. § 39.103. Of note, the Federal Aviation Administration (FAA), part of Transportation, is exempt from the Federal Acquisition Regulation. 49 U.S.C. § 40110 (d)(2)(G). Instead, FAA operates under an independent acquisition system, referred to as FAA's Acquisition Management System. [26] The Federal CIO is the presidential designation for the Administrator of the OMB Office of E-Government. [27] [hyperlink, http://www.itdashboard.gov]. [28] Reducing Duplication and Improving Outcomes in Federal Information Technology, Before the S. Comm. on Homeland Security and Governmental Affairs, 113 Cong. 23 (2013) (statement of Federal CIO Steven VanRoekel). [29] Executive Office of the President of the United States, OMB, Fiscal Year 2014 Analytical Perspectives: Budget of the U.S. Government, 354. [30] OMB, 25 Point Implementation Plan to Reform Federal Information Technology Management (Washington, D.C.: Dec. 9, 2010). [31] GAO, Information Technology Reform: Progress Made; More Needs to Be Done to Complete Actions and Measure Results, [hyperlink, http://www.gao.gov/products/GAO-12-461] (Washington, D.C.: Apr. 26, 2012). [32] OMB, Contracting Guidance to Support Modular Development (June 14, 2012). [33] GAO, Information Technology: Critical Factors Underlying Successful Major Acquisitions, [hyperlink, http://www.gao.gov/products/GAO-12-7] (Washington, D.C.: Oct. 21, 2011). [34] The seven investments were (1) Department of Commerce's Decennial Response Integration System, (2) Defense's Global Combat Support System Joint (Increment 7), (3) Department of Energy's Manufacturing Operations Management Project, (4) DHS's Western Hemisphere Travel Initiative, (5) Transportation's Integrated Terminal Weather System, (6) Internal Revenue Service's Customer Account Data Engine 2, and (7) VA's Occupational Health Record-keeping System. [35] Agile development calls for the delivery of software in small, short increments rather than in the typically long, sequential phases of a traditional waterfall approach. More a philosophy than a methodology, Agile emphasizes early and continuous software delivery, as well as using collaborative teams and measuring progress with working software. The Agile approach was first articulated in a 2001 document called the Agile Manifesto, which is still used today. The manifesto has four values: (1) individuals and interactions over processes and tools, (2) working software over comprehensive documentation, (3) customer collaboration over contract negotiation, and (4) responding to change over following a plan. [36] GAO, Software Development: Effective Practices and Federal Challenges in Applying Agile Methods, [hyperlink, http://www.gao.gov/products/GAO-12-681] (Washington, D.C.: July 27, 2012). [37] OMB, Management of Federal Information Resources, Circular No. A- 130 Revised. In 2012, OMB reaffirmed its call for agencies to develop policies on incremental development, but it did not specify what these policies are to include. OMB, Contracting Guidance to Support Modular Development (June 14, 2012). [38] OMB, Guidance on Exhibits 53 and 300--Information Technology and E-Government (2013); Guidance on Exhibits 53 and 300--Information Technology and E-Government (2012); Guidance on Exhibit 300--Planning, Budgeting, Acquisition, and Management of Information Technology Capital Assets (2011). [39] Executive Office of the President of the United States, OMB, Fiscal Year 2014 Analytical Perspectives: Budget of the U.S. Government, 354. [40] SEI, CMMI-ACQ, Version 1.3 (November 2010). [41] VA, Project Management Accountability System, Directive 6071 (Feb. 20, 2013). [42] Defense, Operation of the Defense Acquisition System, Interim Instruction No. 5000.02 (Nov. 26, 2013). [43] See, for example, Defense, Office of the Secretary of Defense, Guidance for Fiscal Year 2013 IT Budget Submissions (Aug. 9, 2011). [44] Using this same rationale, Defense has requested that OMB grant the department an exemption from OMB's 6-month requirement for the past 3 years. According to OMB staff members from the Office of E- Government, OMB has not granted these requests. [45] DHS, DHS Approach to Financial Systems Modernization, Version 1.3 (July 2012). [46] Transportation officials noted that FAA is exempt from the Federal Acquisition Regulation and operates under an independent acquisition system, referred to as FAA's Acquisition Management System. [47] According to VA guidance, an increment deliverable will be defined in most situations as a new or enhanced IT capability used by one or more customers in production. For some high-risk projects, delivery of a prototype or pilot may be an acceptable increment deliverable. VA's guidance also states that, for some complex systems where field deployment is resource intensive, the increment deliverable may be defined as the first production deployment. [48] OMB, Guidance on Exhibits 53 and 300--Information Technology and E-Government (2013); Guidance on Exhibits 53 and 300--Information Technology and E-Government (2012); and Guidance on Exhibit 300-- Planning, Budgeting, Acquisition, and Management of Information Technology Capital Assets (2011). [49] Officials from HHS explained that they are in the process of updating their IT governance guidance and, as part of this process, will consider whether the agency needs to make any changes to its incremental development policies. [50] OMB, 25 Point Implementation Plan to Reform Federal Information Technology Management (Washington, D.C.: Dec. 9, 2010). [51] OMB, Guidance on Exhibits 53 and 300--Information Technology and E-Government (2013); Guidance on Exhibits 53 and 300--Information Technology and E-Government (2012); and Guidance on Exhibit 300-- Planning, Budgeting, Acquisition, and Management of Information Technology Capital Assets (2011). [52] In contrast, of the 62 investments not using Agile, only 9 (15 percent) planned to deliver functionality every 6 months. [53] [hyperlink, http://www.gao.gov/products/GAO-12-681]. [54] During the planning and budgeting phase, the agency performs the steps needed to justify the new investment and identify the amount of funding needed to deliver the IT asset. The acquisition phase begins after the agency has received funding from Congress for a segment, module, or the entire asset, and ends when the asset is delivered and fully operational. The management in-use phase begins after the completion of the acquisition phase and involves the continuous monitoring of capital assets to ensure that they are maintained at the right size, cost, and condition to support agency mission and objectives. The disposition phase involves the disposal of the capital asset. OMB defines these terms in its Capital Programming Guide (July 2012). [55] VA's guidance on incremental development addresses this issue: its projects are always in one of six life-cycle stages (new start, planning, active, closed, provisioning, and paused) and are only required to deliver functionality every 6 months while in the active stage. [56] SEI, CMMI-ACQ, Version 1.3 (November 2010). [57] GAO, Information Technology Investment Management: A Framework for Assessing and Improving Process Maturity, version 1.1, [hyperlink, http://www.gao.gov/products/GAO-04-394G] (March 2004). [58] Five investments are both (1) in life-cycle phases other than acquisition and (2) intended to develop IT infrastructure, and 6 investments are both (1) in life-cycle phases other than acquisition and (2) research and development investments. As such, the total number of investments not considered is 40 rather than 51. [59] [hyperlink, http://www.gao.gov/products/GAO-13-796T]. [60] According to OMB staff members from the Office of E-Government, this investment was Transportation's Federal Motor Carrier Safety Administration Application and Information Services Modernization. OMB reported holding the TechStat session in August 2012. [61] A VA official from the Office of the CIO attributed the problems to confusion about how the 6-month requirement applies to projects that deploy functionality to sites across the country. [62] HHS officials representing the Office of the CIO explained that they will address this issue if they see patterns of problems with exhibit 300 submissions. Transportation Office of the CIO officials stated that a clear definition from OMB on what functionality is to be delivered every 6 months is needed to address this issue. [63] Defense also reported similar enabling and inhibiting factors for the 12-18 month period called for in its guidance. [64] The backlog is a list of customer requirements to be addressed by the functionality. [65] [hyperlink, http://www.gao.gov/products/GAO-12-681] and [hyperlink, http://www.gao.gov/products/GAO-12-7]. [66] OMB, 25 Point Implementation Plan to Reform Federal Information Technology Management (Washington, D.C.: Dec. 9, 2010). [67] [hyperlink, http://www.gao.gov/products/GAO-12-681]. [68] OMB, 25 Point Implementation Plan to Reform Federal Information Technology Management (Washington, D.C.: Dec. 9, 2010). [69] The purpose of the exhibit 53 is to identify all IT investments-- both major and nonmajor--and their associated costs. [70] Those investments are Defense's Air Force Intranet Increment 2; HHS's Federal Drug Administration Office of Regulatory Affairs Regulatory Business Information Services; DHS's TacNet, Immigration and Custom's Enforcement Student and Exchange Visitor Information System, and Federal Emergency Management Agency Infrastructure; and Transportation's Federal Motor Carrier Safety Administration Inspection Services, Federal Motor Carrier Safety Administration Compliance Services, Federal Motor Carrier Safety Administration Registration Services, and Pipeline and Hazardous Materials Safety Administration National Pipeline Information Exchange. [71] OMB, Guidance on Exhibits 53 and 300--Information Technology and E-Government (2013); Executive Office of the President of the United States, OMB, Fiscal Year 2014 Analytical Perspectives: Budget of the U.S. Government, 354; OMB, Contracting Guidance to Support Modular Development (June 14, 2012); Guidance on Exhibits 53 and 300-- Information Technology and E-Government (2012); Guidance on Exhibit 300--Planning, Budgeting, Acquisition, and Management of Information Technology Capital Assets (2011); and Management of Federal Information Resources, Circular No. A-130 Revised. [72] Software Engineering Institute, Capability Maturity Model® Integration for Acquisition, Version 1.3 (November 2010). [73] SEI, CMMI-ACQ, Version 1.3 (November 2010). [74] OMB, Guidance on Exhibits 53 and 300--Information Technology and E-Government (2013); Guidance on Exhibits 53 and 300--Information Technology and E-Government (2012); Guidance on Exhibit 300--Planning, Budgeting, Acquisition, and Management of Information Technology Capital Assets (2011). In 2011, OMB changed its guidance to recommend that projects deliver functionality every 6 months. Since 2012, OMB has required that projects deliver functionality in 6-month cycles. [75] SEI, CMMI-ACQ, Version 1.3 (November 2010). [76] SEI, CMMI-ACQ, Version 1.3 (November 2010). [77] OMB, Management of Federal Information Resources, Circular No. A- 130 Revised. In 2012, OMB reaffirmed its call for agencies to develop policies on incremental development, but it did not specify what these policies are to include. OMB, Contracting Guidance to Support Modular Development (June 14, 2012). [78] Executive Office of the President of the United States, OMB, Fiscal Year 2014 Analytical Perspectives: Budget of the U.S. Government, 354. [79] OMB, Guidance on Exhibits 53 and 300--Information Technology and E-Government (2013); Guidance on Exhibits 53 and 300--Information Technology and E-Government (2012); and Guidance on Exhibit 300-- Planning, Budgeting, Acquisition, and Management of Information Technology Capital Assets (2011). [80] VA, Project Management Accountability System Guide 4.0 (Nov. 7, 2012). [81] SEI, CMMI-ACQ, Version 1.3 (November 2010). [82] GAO, Information Technology: Critical Factors Underlying Successful Major Acquisitions, [hyperlink, http://www.gao.gov/products/GAO-12-7] (Washington, D.C.: Oct. 21, 2011). [83] GAO, Software Development: Effective Practices and Federal Challenges in Applying Agile Methods, [hyperlink, http://www.gao.gov/products/GAO-12-681] (Washington, D.C.: July 27, 2012). [84] OMB, 25 Point Implementation Plan to Reform Federal Information Technology Management (Washington, D.C.: Dec. 9, 2010). [85] See, for example, Defense, Office of the Secretary of Defense, Guidance for Fiscal Year 2013 IT Budget Submissions (Aug. 9. 2011). [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's website [hyperlink, http://www.gao.gov]. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Phone: The price of each GAO publication reflects GAO's actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO's website, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO: Connect with GAO on facebook, flickr, twitter, and YouTube. Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts. Visit GAO on the web at [hyperlink, http://www.gao.gov]. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; E-mail: fraudnet@gao.gov; Automated answering system: (800) 424-5454 or (202) 512-7470. Congressional Relations: Katherine Siggerud, Managing Director, siggerudk@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, DC 20548. Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, DC 20548. [End of document]