This is the accessible text file for GAO report number GAO-14-64 entitled 'IT Dashboard: Agencies Are Managing Investment Risk, but Related Ratings Need to Be More Accurate and Available' which was released on January 13, 2013. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: GAO: Report to Congressional Requesters: December 2013: IT Dashboard: Agencies Are Managing Investment Risk, but Related Ratings Need to Be More Accurate and Available: GAO-14-64: GAO Highlights: Highlights of GAO-14-64, a report to congressional requesters. Why GAO Did This Study: OMB launched the Dashboard in June 2009 as a public Web site that reports performance for major IT investments—on which the federal government plans to invest over $38 billion in fiscal year 2014. The Dashboard is to provide transparency for these investments and to facilitate public monitoring of them. After its launch, OMB began using it to identify at-risk investments. This report (1) characterizes the CIO ratings for selected federal agencies’ IT investments as reported on the Dashboard over time, (2) determines the extent to which selected agencies’ CIO ratings are consistent with investment risk, and (3) determines the extent to which selected agencies are addressing at-risk investments. GAO selected the eight agencies with the most reported major IT spending in fiscal year 2012 (excluding those GAO recently reviewed) and selected 10 investments at each agency. GAO reviewed the investments’ documentation, compared it to the CIO ratings, and reviewed processes used for the highest-risk investments. GAO also interviewed appropriate officials. What GAO Found: As of August 2013, the Chief Information Officers (CIO) at the eight selected agencies rated 198 of their 244 major information technology (IT) investments listed on the Federal IT Dashboard (Dashboard) as low risk or moderately low risk, 41 as medium risk, and 5 as high risk or moderately high risk. However, the total number of investments reported by these agencies has varied over time, which impacts the number of investments receiving CIO ratings. For example, Energy reclassified several of its supercomputer investments from IT to facilities and Commerce decided to reclassify its satellite ground system investments. Both decisions resulted in the removal of the investments from the Dashboard, even though the investments were clearly IT. In addition, the Office of Management and Budget (OMB) does not update the public version of the Dashboard as the President’s budget request is being created. As a result, the public version of the Dashboard was not updated for 15 of the past 24 months, and so was not available as a tool for investment oversight and decision making. Of the 80 investments reviewed, 53 of the CIO ratings were consistent with the investment risk, 20 were partially consistent, and 7 were inconsistent (see table). Table: Consistency of Selected Agencies’ CIO Ratings with Supporting Documentation: Agency: Department of Agriculture; Rating consistency of each agency’s 10 selected investments: The investment’s ratings were consistent with supporting documents: 2; The investment’s ratings were partially consistent with supporting documents: 8. Agency: Department of Commerce; Rating consistency of each agency’s 10 selected investments: The investment’s ratings were consistent with supporting documents: 8; The investment’s ratings were partially consistent with supporting documents: 2. Agency: Department of Energy; Rating consistency of each agency’s 10 selected investments: The investment’s ratings were consistent with supporting documents: 5; The investment’s ratings were partially consistent with supporting documents: 5. Agency: Department of Justice; Rating consistency of each agency’s 10 selected investments: The investment’s ratings were consistent with supporting documents: 9; The investment’s ratings were partially consistent with supporting documents: 1. Agency: Department of Transportation; Rating consistency of each agency’s 10 selected investments: The investment’s ratings were consistent with supporting documents: 10. Agency: Department of the Treasury; Rating consistency of each agency’s 10 selected investments: The investment’s ratings were consistent with supporting documents: 10. Agency: Department of Veterans Affairs; Rating consistency of each agency’s 10 selected investments: The investment’s ratings were partially consistent with supporting documents: 3; The investment’s ratings were inconsistent with supporting documents: 7. Agency: Social Security Administration; Rating consistency of each agency’s 10 selected investments: The investment’s ratings were consistent with supporting documents: 9; The investment’s ratings were partially consistent with supporting documents: 1. Source: GAO analysis of OMB’s Dashboard and agency data. [End of table] While two agencies’ CIO ratings were entirely consistent, other agencies’ ratings were inconsistent for a variety of reasons, including delays in updating the Dashboard and how investment performance was tracked. For example, the Department of Justice downgraded an investment in July 2012, but the Dashboard was not updated to reflect this until April 2013. Further, the Social Security Administration resets investment cost and schedule performance baselines annually, an approach that increases the risk of undetected cost or schedule variances that will impact investment success. Of the eight investments that were at highest risk in 2012, seven were reviewed by their agencies using tools such as TechStat sessions— evidence-based reviews intended to improve investment performance and other high-level reviews. Each of these resulted in action items intended to improve performance. The final investment was scheduled to have a TechStat, but instead, according to department officials, a decision was made to modify its program cost and schedule commitments to better reflect the investment’s actual performance. What GAO Recommends: GAO recommends that OMB make Dashboard information available independent of the budget process, and that agencies appropriately categorize IT investments and address identified weaknesses. OMB neither agreed nor disagreed. Six agencies generally agreed with the report or had no comments and two others did not agree, believing their categorizations were appropriate. GAO continues to believe its recommendations remain valid, as discussed. View [hyperlink, http://www.gao.gov/products/GAO-14-64]. For more information, contact David A. Powner at (202) 512-9286 or pownerd@gao.gov. [End of section] Contents: Letter: Background: CIOs at Selected Agencies' Rated Majority of IT Investments as Low or Moderately Low Risk: Most CIO Ratings Are Consistent with Reported Investment Risk: Highest-Risk Investments Were Appropriately Addressed by Selected Agencies: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Objectives, Scope, and Methodology: Appendix II: Selected Investments: Appendix III: Comments from the Office of Management and Budget: Appendix IV: Comments from the Department of Agriculture: Appendix V: Comments from the Department of Commerce: Appendix VI: Comments from the Department of Energy: Appendix VII: Comments from the Department of the Treasury: Appendix VIII: Comments from the Department of Veterans Affairs: Appendix IX: Comments from the Social Security Administration: Appendix X: GAO Contact and Staff Acknowledgments: Tables: Table 1: Dashboard Variance and Rating Colors: Table 2: Investment Evaluation Factors Identified by OMB for Assigning CIO Ratings: Table 3: IT Dashboard CIO Rating Colors, Based on a Five-Point Scale for CIO Ratings: Table 4: Assessment of the Consistency of Selected Agencies' CIO Ratings with Supporting Documentation from January 2012 through December 2012: Table 5: Investments Selected for Review and Associated Fiscal Year 2012 Spending: Figures: Figure 1: Example of an Agency Portfolio Page as Reported on OMB's IT Dashboard, May 2013: Figure 2: Dashboard CIO Ratings over Time from the Eight Selected Agencies: Figure 3: Dashboard Rating Changes over Time and Associated Number of Investments from the Eight Selected Agencies between June 2009 and August 2013: Figure 4: Dashboard Rating Changes over Time and Associated Number of Investments by Agency between June 2009 and August 2013: Figure 5: Number of Investments the Eight Selected Agencies Reported on the Dashboard over Time: Figure 6: Dashboard Ratings in 2012 for the Eight At-Risk Investments: Abbreviations: Agriculture: Department of Agriculture: CIO: Chief Information Officer: Commerce: Department of Commerce: Energy: Department of Energy: FAA: Federal Aviation Administration: FBI: Federal Bureau of Investigation: Justice: Department of Justice: IT: Information technology: OMB: Office of Management and Budget: SSA: Social Security Administration: TechStats: TechStat Accountability Sessions: Transportation: Department of Transportation: Treasury: Department of the Treasury: VA: Department of Veterans Affairs: [End of section] GAO: United States Government Accountability Office: 441 G St. N.W. Washington, DC 20548: December 12, 2013: The Honorable Thomas R. Carper: Chairman: The Honorable Tom Coburn, M.D. Ranking Member: Committee on Homeland Security and Governmental Affairs: United States Senate: The Honorable Fred Upton: Chairman: Committee on Energy and Commerce: House of Representatives: In accordance with its responsibilities set forth in law,[Footnote 1] the Office of Management and Budget (OMB) launched the Federal Information Technology (IT) Dashboard in June 2009 as a public Web site that reports performance and supporting data for major IT investments[Footnote 2]--on which the federal government plans to invest over $38 billion in fiscal year 2014. The Dashboard is to provide transparency for these investments in order to facilitate public monitoring of government operations and accountability for investment performance by the federal Chief Information Officers (CIO) who oversee them. According to OMB, it began using the Dashboard to identify at-risk investments with its launch in June 2009. These investments became the focus of joint OMB-agency TechStat Accountability Sessions (TechStats)--evidence-based reviews intended to improve investment performance through concrete actions. This report responds to your request that we review the consistency of investment risk information submitted to the Dashboard. Our objectives for this engagement were to (1) characterize the CIO ratings for selected federal agencies' IT investments as reported on the Dashboard over time, (2) determine the extent to which selected agencies' CIO ratings are consistent with reported investment risk, and (3) determine the extent to which selected agencies are addressing at-risk investments. To address our first objective, we selected the eight agencies [Footnote 3] with the most reported major IT spending in fiscal year 2012, after excluding agencies reviewed in our most recent report on the Dashboard.[Footnote 4] We then reviewed publicly available data from the Dashboard from its inception in June 2009 through August 2013. We also interviewed officials from OMB and the selected agencies. To accomplish the second objective, we selected the top 10 major investments with the highest reported IT spending at each of the 8 selected agencies (for a total of 80 investments). We reviewed investment documentation (such as executive-level briefings, operational analyses, and the results of performance reviews) and interviewed officials from each of the agencies to understand how they rated and monitored investments. We compared these documents to the CIO ratings submitted to the Dashboard in calendar year 2012 to determine whether the ratings on the Dashboard were accurately portraying the risk of these investments. We elected to review the investments' ratings during calendar year 2012 so that our assessment would span multiple fiscal years. Finally, for our third objective, we reviewed and described each of the processes used by the eight selected agencies to address the highest-risk major investments (such as TechStat guides and investment review board results). We also identified investments within the 80 investments selected for the second objective which had received high or moderately high ratings of risk on the Dashboard during 2012. We analyzed both the processes used to address these investments and the results of performance reviews. We also interviewed officials to determine whether the agencies implemented the appropriate risk management processes to oversee and review the selected investments. We conducted this performance audit from February to December 2013 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Further details of our objectives, scope, and methodology are provided in appendix I. Background: While IT can enrich people's lives and improve organizational performance, we have previously reported that federal IT projects too frequently incur cost overruns and schedule slippages while contributing little to mission-related outcomes. For example, in July 2013, we testified that the federal government continued to spend billions of dollars on troubled IT investments and we identified billions of dollars worth of failed and challenged IT projects. [Footnote 5] OMB plays a key role in overseeing how federal agencies manage their IT investments by working with them to better plan, justify, and determine how to manage them. Each year, OMB and federal agencies work together to determine how much the government plans to spend on IT projects and how these funds are to be allocated. OMB also guides agencies in developing sound business cases for IT investments and establishing management processes for overseeing these investments throughout their life cycles. The scope of this undertaking is quite large: in planning for fiscal year 2014, 27 federal agencies reported plans to spend about $82 billion on IT.[Footnote 6] Of that, $38.7 billion is to be spent on major IT investments, $37.6 billion is to be spent on non-major IT investments, and $5.5 billion is to be spent on classified Department of Defense IT investments. Within OMB, the Office of E-Government and Information Technology, headed by the Federal CIO, directs the policy and strategic planning of federal IT investments and is responsible for oversight of federal information technology spending. OMB's Dashboard Is Intended to Provide Visibility into the Performance of Federal IT Investments: In June 2009, OMB deployed a public Web site to further improve the transparency and oversight of agencies' IT investments. Known as the IT Dashboard,[Footnote 7] this site displays federal agencies' cost, schedule, and performance data for over 700 major federal IT investments at 27 federal agencies, accounting for $38.7 billion of those agencies' planned $82 billion budget for fiscal year 2014. According to OMB, these data are intended to provide a near-real-time perspective on the performance of these investments, as well as a historical perspective. Further, the public display of these data is intended to allow OMB; other oversight bodies, including Congress; and the general public to hold the government agencies accountable for progress and results. The Dashboard visually presents performance ratings for agencies and for individual investments using metrics that OMB has defined--cost, schedule, and CIO evaluation. The Web site also provides the capability to download certain data. Figure 1 is an example of VA's portfolio page as recently depicted on the Dashboard. (The ratings are explained in the narrative following the figure.) Figure 1: Example of an Agency Portfolio Page as Reported on OMB's IT Dashboard, May 2013: [Refer to PDF for image: IT Dashboard page illustration] Depicts: Investment Evaluation by Agency CIO: Current; Trend. Project Cost: Current; Trend. Project schedule: Current; Trend. Source: VA’s portfolio page as reported on OMB’s IT Dashboard on May 22, 2013. [End of figure] The Dashboard's data spans the period from its June 2009 inception to the present, and is based, in part, on agency assessments of individual investment performance and each agency's budget submissions to OMB.[Footnote 8] Cost and Schedule Ratings: The cost and schedule data agencies are required to submit to the Dashboard have changed over time, as have the related calculations. For example, in response to our recommendations (further discussed in the following section), OMB changed how the Dashboard calculates the cost and schedule ratings in July 2010 to include "in progress" milestones rather than just "completed" ones for a more accurate reflection of current investment status. While the required cost and schedule data have changed, the thresholds for assigning cost and schedule ratings have remained constant over the life of the Dashboard. Specifically, the Dashboard assigns rating colors (red, yellow, green) based on agency-submitted cost and schedule variances, as shown in table 1. Table 1: Dashboard Variance and Rating Colors: Project level cost and schedule variance rating: 30%; Rating color: Red. Project level cost and schedule variance rating: 10% and less than 30%; Rating color: Yellow. Project level cost and schedule variance rating: less than 10%; Rating color: Green. Source: OMB's IT Dashboard. [End of table] CIO Ratings: Unlike the variance-based cost and schedule ratings, the Dashboard's "Investment Evaluation by Agency CIO" (also called the CIO rating) is determined by agency officials. According to OMB's instructions, each agency CIO is to assess his or her IT investments against a set of six pre-established evaluation factors identified by OMB (shown in table 2) and then assign a rating of 1 (high risk) to 5 (low risk) based on the CIO's best judgment of the level of risk facing the investment. Table 2: Investment Evaluation Factors Identified by OMB for Assigning CIO Ratings: Evaluation factor: Risk management; Supporting examples: Risk management strategy exists; Risks are well understood by senior leadership; Risk log is current and complete; Risks are clearly prioritized; Mitigation plans are in place to address risks. Evaluation factor: Requirements management; Supporting examples: Investment objectives are clear and scope is controlled; Requirements are complete, clear, and validated; Appropriate stakeholders are involved in requirements definition. Evaluation factor: Contractor oversight; Supporting examples: Acquisition strategy is defined and managed via an Integrated Program Team; Agency receives key reports, such as earned value reports, current status, and risk logs; Agency is providing appropriate management of contractors such that the government is monitoring, controlling, and mitigating the impact of any adverse contract performance. Evaluation factor: Historical performance; Supporting examples: No significant deviations from planned cost and schedule; Lessons learned and best practices are incorporated and adopted. Evaluation factor: Human capital; Supporting examples: Qualified management and execution team for the IT investments and/or contracts supporting the investment; Low turnover rate. Evaluation factor: Other; Supporting examples: Other factors that the CIO deems important to forecasting future success. Source: OMB's IT Dashboard. [End of table] OMB recommends that CIOs consult with appropriate stakeholders in making their evaluation, including Chief Acquisition Officers, program managers, and other interested parties. According to an OMB staff member, agency CIOs are responsible for determining appropriate thresholds for the risk levels and for applying them to investments when assigning CIO ratings. OMB requires agencies to update these ratings as soon as new information becomes available which will affect an investment's assessment. After agencies assign a level of risk to each investment, the Dashboard assigns colors to CIO ratings according to a five-point scale, as illustrated in table 3. Table 3: IT Dashboard CIO Rating Colors, Based on a Five-Point Scale for CIO Ratings: Rating (by agency CIO): 1-High risk; Color: Red. Rating (by agency CIO): 2-Moderately high risk; Color: Red. Rating (by agency CIO): 3-Medium risk; Color: Yellow. Rating (by agency CIO): 4-Moderately low risk; Color: Green. Rating (by agency CIO): 5-Low risk; Color: Green. Source: OMB's IT Dashboard. [End of table] An OMB staff member from the Office of E-Government and Information Technology noted that the CIO rating should be a current assessment of future performance based on historical results and is the only Dashboard performance indicator that has been defined and produced the same way since the Dashboard's inception. Furthermore, OMB issued guidance in August 2011[Footnote 9] that stated, among other things, that agency CIOs shall be held accountable for the performance of IT program managers based on their governance process and the data reported on the IT Dashboard, which includes the CIO rating. According to OMB, the addition of CIO names and photos on Dashboard investments is intended to highlight this accountability and link it to the Dashboard's reporting on investment performance. GAO Has Previously Reported on the Dashboard's Value, Data Quality, and Improvements: We have previously reported that OMB has taken significant steps to enhance the oversight, transparency, and accountability of federal IT investments by creating its IT Dashboard, and by improving the accuracy of investment ratings. We also found issues with the accuracy and data reliability of cost and schedule data, and recommended steps that OMB should take to improve these data. * In July 2010, we reported[Footnote 10] that the cost and schedule ratings on OMB's Dashboard were not always accurate for the investments we reviewed, because these ratings did not take into consideration current performance. As a result, the ratings were based on outdated information. We recommended that OMB report on its planned changes to the Dashboard to improve the accuracy of performance information and provide guidance to agencies to standardize milestone reporting. OMB agreed with our recommendations and, as a result, updated the Dashboard's cost and schedule calculations to include both ongoing and completed activities. * Similarly, our report in March 2011[Footnote 11] noted that OMB had initiated several efforts to increase the Dashboard's value as an oversight tool, and had used its data to improve federal IT management. We also reported, however, that agency practices and the Dashboard's calculations contributed to inaccuracies in the reported investment performance data. For instance, we found missing data submissions or erroneous data at each of the five agencies we reviewed, along with instances of inconsistent program baselines and unreliable source data. As a result, we recommended that the agencies take steps to improve the accuracy and reliability of their Dashboard information, and that OMB improve how it rates investments relative to current performance and schedule variance. Most agencies generally concurred with our recommendations and three have taken steps to address our recommendation. OMB agreed with our recommendation for improving ratings for schedule variance. It disagreed with our recommendation to improve how it reflects current performance in cost and schedule ratings, but more recently made changes to Dashboard calculations to address this while also noting challenges in comprehensively evaluating cost and schedule data for these investments. * We also reported on OMB's guidance to agencies for reporting their IT investments in September 2011 and found that it did not ensure complete reporting.[Footnote 12] Specifically, agencies differed on what investments they included as an IT investment. Among other things, we recommended that OMB clarify its guidance on reporting IT investments to specify whether certain types of systems--such as space systems--are to be included. OMB did not agree that further efforts were needed to clarify reporting in regard to the types of systems. * Subsequently, in November 2011, we noted[Footnote 13] that the accuracy of investment cost and schedule ratings had improved since our July 2010 report because OMB had refined the Dashboard's cost and schedule calculations. Most of the ratings for the eight investments we reviewed as part of our November 2011 report were accurate, although we noted that more could be done to inform oversight and decision making by emphasizing recent performance in the ratings. We recommended that the General Services Administration comply with OMB's guidance for updating its ratings when new information becomes available (including when investments are rebaselined). The agency concurred and has since taken actions to address this recommendation. Since we previously recommended that OMB improve how it rates investments, we did not make any further recommendations. * More recently, in October 2012 we reported that CIOs at six agencies rated a majority of investments listed on the federal IT Dashboard as low or moderately low risk from June 2009 through March 2012.[Footnote 14] Additionally, two agencies, the Department of Defense and the National Science Foundation, rated no investments as high or moderately high risk during this time period. Agencies generally followed OMB's instructions for assigning CIO ratings, although the Department of Defense's ratings were unique in reflecting additional considerations, such as the likelihood of OMB review. Most of the selected agencies reported various benefits associated with producing and reporting CIO ratings, such as increased quality of their performance data and greater transparency and visibility of investments. We recommended that OMB analyze agencies' investment risk over time as reflected in the Dashboard's CIO ratings and present its analysis with the President's annual budget submission, and that the Department of Defense ensure that its CIO ratings reflect available investment performance assessments and its risk management guidance. Both agencies concurred with our recommendations, and OMB reported on CIO rating trends in the fiscal year 2014 budget submission; however, the Department of Defense has not updated any of its ratings since September 2012. * Further, we studied OMB's efforts to help agencies address IT projects with cost overruns, schedule delays, and performance shortfalls in June 2013.[Footnote 15] In particular, we reported that OMB used CIO ratings from the Dashboard, among other sources, to select at-risk investments for reviews known as TechStats.[Footnote 16] OMB initiated these reviews in January 2010 to further improve investment performance, and subsequently incorporated the TechStat model into its 25-point plan for reforming federal IT management. [Footnote 17] We reported that OMB and selected agencies had held multiple TechStat sessions but additional OMB oversight was needed to ensure that these meetings were having the appropriate impact on underperforming projects and that resulting cost savings were valid. Among other things, we recommended that OMB require agencies to address their highest-risk investments and to report on how they validated the outcomes. OMB generally agreed with our recommendations, and stated that it and the agencies were taking appropriate steps to address them. CIOs at Selected Agencies' Rated Majority of IT Investments as Low or Moderately Low Risk: As of August 2013, CIOs at the eight selected agencies rated 198 of their 244 major investments listed on the Dashboard as low risk or moderately low risk. Of the remaining 46 investments, 41 were rated medium risk, and 5 were rated high risk or moderately high risk. Historically, over the life of the Dashboard from June 2009 to August 2013, low or moderately low risk investments accounted for an average of 72 percent of all ratings at the eight agencies, medium risk ratings an average of 22 percent, and high risk or moderately high risk ratings an average of 6 percent.[Footnote 18] The CIO ratings and associated number of investments at each the eight agencies as reported on the Dashboard over the past 4 years are presented in figure 2. Figure 2: Dashboard CIO Ratings over Time from the Eight Selected Agencies: [Refer to PDF for image: 8 stacked vertical bar graphs] Depicts number of investments, with the following indications: Low risk; Moderately low risk; Medium risk; Moderately high rick; High risk. On selected dates, June, 2009 through August 2013; For the following agencies: Department of Agriculture; Department of Commerce; Department of Energy; Department of Justice; Social Security Administration; Department of Transportation; Department of the Treasury; Department of Veterans Affairs. Source: GAO analysis of data from OMB's dashboard. [End of figure] When comparing the investments' first and most recent ratings, the agencies we reviewed showed generally positive results. Specifically, of the 383 total investments listed on the Dashboard from June 2009 to August 2013 for the selected agencies,[Footnote 19] 121 increased (meaning risk is lower) and 81 decreased (meaning risk is higher). [Footnote 20] Additionally, a significant portion of investments had returned to their original rating (74) and about one third of investments' ratings had never changed (107). (See figure 3.) Figure 3: Dashboard Rating Changes over Time and Associated Number of Investments from the Eight Selected Agencies between June 2009 and August 2013: [Refer to PDF for image: pie-chart] 121 investments: Increased; 81 investments: Decreased; 107 investments: Never changed; 74 investments: No net change. Source: GAO analysis of data from OMB’s Dashboard. [End of figure] When considered individually, five of the eight selected agencies-- Agriculture, Commerce, Energy, Treasury, and VA--had more investments' ratings increase than decrease, whereas the other three--Justice, Transportation, and SSA--had more investments decrease. Figure 4 presents the net changes in investments' ratings for each agency during the reporting period of June 2009 to August 2013. Figure 4: Dashboard Rating Changes over Time and Associated Number of Investments by Agency between June 2009 and August 2013: [Refer to PDF for image: 3 vertical bar graphs] Increase: An increase in CIO rating is an indication of a reduction in risk level: Agency: Agriculture; Number of investments: 28. Agency: Commerce; Number of investments: 24 Agency: Energy; Number of investments: 16 Agency: Justice; Number of investments: 4 Agency: SSA; Number of investments: 1 Agency: Transportation; Number of investments: 6 Agency: Treasury; Number of investments: 24 Agency: VA; Number of investments: 18 No change: The initial CIO rating is equal to the most recent CIO rating: Agency: Agriculture; Number of investments: 14; No net change: 14; Never changed: 0. Agency: Commerce; Number of investments: 25; No net change: 22; Never changed: 3. Agency: Energy; Number of investments: 11; No net change: 5; Never changed: 6. Agency: Justice; Number of investments: 20; No net change: 3; Never changed: 17. Agency: SSA; Number of investments: 12; No net change: 2; Never changed: 10. Agency: Transportation; Number of investments: 33; No net change: 10; Never changed: 23. Agency: Treasury; Number of investments: 60; No net change: 25; Never changed: 35. Agency: VA; Number of investments: 6; No net change: 1; Never changed: 5. Decrease: A decrease in CIO rating is an indication of a rise in risk level; Agency: Agriculture; Number of investments: 9. Agency: Commerce; Number of investments: 17. Agency: Energy; Number of investments: 5. Agency: Justice; Number of investments: 6. Agency: SSA; Number of investments: 7. Agency: Transportation; Number of investments: 19. Agency: Treasury; Number of investments: 13. Agency: VA; Number of investments: 5. Source: GAO analysis of data from OMB’s Dashboard. [End of figure] The agencies cited additional oversight or program reviews as factors that contributed to improved ratings. Furthermore, Agriculture and Commerce both attributed improved ratings to enhanced timeliness and sufficiency of investment reporting. Both of these agencies factor investment reporting into CIO ratings and increased ratings as better performance and risk information was provided by investment management. This suggests that caution should be exercised when interpreting changing risk levels for investments, as rating changes by agencies may not be entirely due to investment performance. Agencies also commented that the CIO ratings and Dashboard reporting had spurred improved investment management and risk mitigation. Additionally, the total number of investments that the eight agencies reported on the Dashboard has varied over time, which impacts the number of investments receiving CIO ratings (see figure 5). Figure 5: Number of Investments the Eight Selected Agencies Reported on the Dashboard over Time: [Refer to PDF for image: line graph] June, 2009: 83; July, 2009: 237; August, 2009: 237; September, 2009: 237; October, 2009: 236; November, 2009: 236; December, 2009: 236. January, 2010: 236; February, 2010: 236; March, 2010: 241; April, 2010: 288; May, 2010: 289; June, 2010: 289; July, 2010: 289; August, 2010: 289; September, 2010: 289; October, 2010: 296; November, 2010: 315; December, 2010: 325. January, 2011: 325; February, 2011: 332; March, 2011: 332; April, 2011: 332; May, 2011: 324; June, 2011: 313; July, 2011: 301; August, 2011: 300; September, 2011: 330; October, 2011: 330; November, 2011: 330; December, 2011: 330. January, 2012: 329; February, 2012: 331; March, 2012: 331; April, 2012: 331; May, 2012: 266; June, 2012: 266; July, 2012: 266; August, 2012: 266; September, 2012: 284; October, 2012: 284; November, 2012: 284; December, 2012: 284. January, 2013: 284; February, 2013: 258; March, 2013: 258; April, 2013: 244; May, 2013: 244; June, 2013: 244; July, 2013: 244; August: 2013: 244. Source: GAO analysis of data from OMB’s Dashboard. [End of figure] The variation in the number of investments reported on the Dashboard can be attributed, in part, to agencies' ability to add, downgrade, and remove investments throughout the annual federal budget process. However, as we concluded in September 2011, OMB's guidance also did not ensure complete reporting of IT investments by agencies.[Footnote 21] Specifically, we found that OMB's definition of an IT investment is broad, and the 10 agencies we evaluated differed on what systems they included as IT investments. For example, 5 of the 10 agencies we reviewed consistently considered investments in research and development systems as IT, and 5 did not. Consequently, we recommended that OMB clarify its guidance on reporting IT investments to specify whether certain types of systems--such as space systems--were to be included. OMB did not agree that further efforts were needed to clarify reporting in regard to the types of systems. Now, 2 years later, and given the continuing lack of clarity in investment reporting guidance, agencies have elected to withdraw investments from the Dashboard that are clearly IT. Because we have continued to identify inappropriate reclassifications of IT investments, we continue to believe this recommendation has merit. For example, as part of the most recent budget process, Energy officials reported several of Energy's supercomputer investments as facilities rather than IT, thus removing those investments from the Dashboard and accounting for a portion of the recent decrease in investments. Energy officials also stated that OMB approved this change. These investments account for $368 million, or almost 25 percent, of Energy's fiscal year 2012 IT spending, and include the National Nuclear Security Administration's Sequoia system, which Energy reported as the most powerful computing system in the world as of June 2012. According to Energy officials, these investments were recategorized because they include both supercomputers and laboratory facilities (which are not IT). As another example, the Deputy Secretary of Commerce issued a directive in September 2012 which resulted in the removal of Commerce's satellite investments from the Dashboard. As we found in 2011, Commerce only reported the ground systems of a spacecraft as IT investments, and not the technology components on the spacecraft itself.[Footnote 22] With this directive, Commerce removed the ground- based portions from IT investment reporting as well, accounting for $447 million, or 26 percent, of the department's fiscal year 2012 IT spending. In making this decision, Commerce determined that it needed to refocus oversight efforts to a more appropriate level and consequently minimized the role of the CIO and others in the oversight of satellites. A Commerce official stated that Commerce plans to exclude all such investments from the department's fiscal year 2015 IT budget submission. However, these recategorizations run contrary to the Clinger-Cohen Act of 1996, which specifies requirements for the management of IT and defines IT as "any equipment or interconnected system or subsystem of equipment, used in the automatic acquisition, storage, analysis, evaluation, manipulation, management, movement, control, display, switching, interchange, transmission, or reception of data or information by the executive agency."[Footnote 23] Supercomputers are indisputably IT and satellite ground systems also meet the above definition as they receive and transmit satellite data. A staff member from the Office of E-Government stated that OMB could not stop agencies from making such recategorizations and that OMB had no control over such agency decisions. By gathering incomplete information on IT investments, OMB increases the risk of not fulfilling its oversight responsibilities, of agencies making inefficient and ineffective investment decisions, and of Congress and the public being misinformed as to the performance of federal IT investments. As part of the budget process, OMB is required to analyze, track, and evaluate the risks and results of all major IT investments.[Footnote 24] The Dashboard gives the public access to the same tools and analysis that the government uses in conducting this performance oversight. According to OMB, Dashboard data are intended to provide a near-real-time perspective on the performance of major investments. Although intended to be a mechanism to enhance transparency, OMB does not update the public version of the Dashboard as it and the agencies work to assist in the formulation of the President's budget request. Nevertheless, agencies continue to provide updates during this time. For the fiscal year 2013 presidential budget request, the static period during which the public saw no updates lasted for 7 months (September 2011 to March 2012) and for fiscal year 2014 the period lasted for 8 months (September 2012 to April 2013). In total, OMB did not update the public Dashboard for 15 of the past 24 months. An Office of E-Government staff member stated that OMB considers the Dashboard information submitted during the budget process to be predecisional and will only release it to the public in tandem with the President's budget. However, the CIO rating does not contain any of the information that the Dashboard's Web page for frequently asked questions[Footnote 25] identifies as predecisional, namely: "new" investments, planned IT spending levels in the budget year, costs of projects and activities that were completed during or after the budget year, and eliminated or downgraded investments. Given that the CIO rating does not include such data, the rating could be updated independent of the budget process without disclosing information that OMB views as predecisional. However, by withholding current investment performance information for much of the year, OMB decreases the utility of the Dashboard as a tool for greater IT investment oversight and transparency. Most CIO Ratings Are Consistent with Reported Investment Risk: Of the 80 selected investments at the eight agencies we reviewed, 53 of the CIO ratings were consistent with the risks portrayed in the supporting investment performance documents, 20 were partially consistent, and 7 VA investments were inconsistent. Those that were partially consistent had few instances of discrepancies during the 12- month period. For example, both of Commerce's inconsistent investments had a discrepancy during 1 of the 12 months which we reviewed. As previously mentioned, a CIO rating should reflect the level of risk facing an investment on a scale from 1 (high risk) to 5 (low risk) relative to that investment's ability to accomplish its goals. However, seven of the eight agencies we reviewed had at least one instance wherein the Dashboard's CIO ratings did not consistently reflect the risks identified in agency documents. Table 4 summarizes our assessment of the 10 investments at each of the selected agencies during the 12-month period from January 2012 through December 2012 and a discussion of the analysis of each agency's documentation follows the table. Table 4: Assessment of the Consistency of Selected Agencies' CIO Ratings with Supporting Documentation from January 2012 through December 2012: Agency: Department of Agriculture; Rating consistency of each agency’s 10 selected investments: The investment’s ratings were consistent with supporting documents: 2; The investment’s ratings were partially consistent with supporting documents: 8. Agency: Department of Commerce; Rating consistency of each agency’s 10 selected investments: The investment’s ratings were consistent with supporting documents: 8; The investment’s ratings were partially consistent with supporting documents: 2. Agency: Department of Energy[A]; Rating consistency of each agency’s 10 selected investments: The investment’s ratings were consistent with supporting documents: 5; The investment’s ratings were partially consistent with supporting documents: 5. Agency: Department of Justice[A]; Rating consistency of each agency’s 10 selected investments: The investment’s ratings were consistent with supporting documents: 9; The investment’s ratings were partially consistent with supporting documents: 1. Agency: Department of Transportation; Rating consistency of each agency’s 10 selected investments: The investment’s ratings were consistent with supporting documents: 10. Agency: Department of the Treasury; Rating consistency of each agency’s 10 selected investments: The investment’s ratings were consistent with supporting documents: 10. Agency: Department of Veterans Affairs; Rating consistency of each agency’s 10 selected investments: The investment’s ratings were partially consistent with supporting documents: 3; The investment’s ratings were inconsistent with supporting documents: 7. Agency: Social Security Administration; Rating consistency of each agency’s 10 selected investments: The investment’s ratings were consistent with supporting documents: 9; The investment’s ratings were partially consistent with supporting documents: 1. Source: GAO analysis of OMB’s Dashboard and agency data. [A] These inconsistencies were the result of OMB's budget process, as discussed below. [End of table] * Agriculture: Eight of the 10 investments at Agriculture had some inconsistencies with reported risks. In particular, Agriculture experienced technical issues uploading its ratings to the Dashboard in early 2012, which impacted the ratings of six investments. Agriculture officials identified an issue with these investments after observing that submitted changes were not reflected on the Dashboard. They addressed this issue by working with their contractor to determine the cause of the issue, modifying their process to prevent the issue from recurring, and establishing procedures to identify future technical issues. Additionally, Agriculture's Financial Management Modernization Initiative--one of the six investments that experienced the technical issue described above--and Human Resources Line of Business's October ratings should have been lower (higher risk) in October 2012. Officials stated that these CIO ratings were updated at the beginning of November 2012 and attributed the delays in the rating submissions to the annual budget process. Additionally, Agriculture's CIO rated the Farm Program Modernization investment as moderately low risk until November 2012, despite the investment showing indications of significantly increased risk as early as May 2012. For example, the number of high and very high risks increased every month in a series of briefings from May to July 2012, at which point there was 1 high-impact risk and 2 very high-impact risks each with an estimated 70 percent probability of occurring. Further, in September 2012 a senior management oversight committee determined the investment to be off-track and unable to resolve the issues. However, the investment remained rated on the Dashboard as moderately low risk until November 2012, when it was updated to medium risk. As a result, the CIO rating did not reflect the results of the oversight committee's review for 3 months. The investment was ultimately given a high-risk rating in December 2012. * Commerce: Two of the 10 selected Commerce investments differed from risk levels identified in underlying documentation for 1 month each. In particular, Commerce's Census - Economic Census and Surveys investment should have been green rather than yellow in March 2012, and its National Oceanic and Atmospheric Administration IT Infrastructure should have been yellow rather than green in June 2012. In both cases Commerce officials recognized that there were problems and took steps to correct them the following month. * Energy: Five of the 10 selected Energy investments remained listed on the Dashboard after the department had recategorized them as investment types that would have no longer been displayed. Specifically, in October 2012, as part of its annual budget process, Energy downgraded its SR Mission Support Systems from major to non- major and, as discussed previously, changed four of the selected supercomputer investments from IT to non-IT. An SR Mission Support Systems official explained that Energy downgraded the investment after realizing that significant portions of it could be interpreted as non- IT under the Federal Acquisition Regulation, and the remainder did not meet the department's threshold for being a major IT investment. Further, although Energy made these decisions in October 2012, OMB officials explained that they do not publish budgetary data such as these until the release of the President's budget, which did not happen until April 2013 at which time the Dashboard reflected Energy's changes. * Justice: One of the 10 selected Justice investments remained on the Dashboard after the department recategorized it as an investment type that would have no longer been displayed, similar to Energy's situation. Specifically, Justice downgraded its Law Enforcement Wireless Communications investment from major to non-major as part of its annual budget process in July 2012, but the Dashboard did not reflect the change until April 2013. As noted above, OMB officials explained that they do not publish budgetary data until the release of the President's budget, which occurred in April 2013. * Transportation: There were no inconsistencies between CIO ratings and reported investments' risks. Continuing to report consistent and timely data should help ensure that Transportation's Dashboard's CIO ratings are accurate. * Treasury: We did not identify inconsistencies between CIO ratings and reported investments' risks. Such attention to reporting consistent and timely data should help ensure the accuracy of the Dashboard's CIO ratings. * VA: Seven of the 10 selected investments were never updated during our evaluation period, and 3 were updated once. VA did not update its ratings because it did not have the ability to automatically submit ratings from September 2011 to March 2013, a period of 19 months. Instead, VA elected to build the capability to submit ratings to the Dashboard, rather than purchase this capability. VA completed development of this tool in March 2013, at which point it resumed making the required updates to CIO ratings on the Dashboard. For the 3 that VA updated during this period, VA used a manual budget process to update their ratings in September 2012. There were also several discrepancies within the ratings depicted on VA's internal Dashboard, known as the Project Management and Accountability System. Specifically, we found 19 instances where negative performance management review results did not change the system's ratings and 5 instances where missed deliverables were not recorded.[Footnote 26] For example, we identified 8 separate VA performance management reviews of the Medical Legacy investment in 2012 which did not impact the associated ratings, and 2 missed deliverables for the same investment which were not recorded. Neglecting to include such performance deviations in the system upon which VA bases its CIO ratings could have resulted in a lower (increased) risk rating. Further, VA's Office of the Inspector General also identified issues with this system in August 2011,[Footnote 27] including a lack of well-established key management controls to ensure data reliability, verify project compliance, and track project costs. VA agreed with those findings and provided steps the department would take to address the identified weaknesses. To that end, VA's February 2013 guidance for this system requires the inclusion of negative performance management reviews and missed deliverables when considering investment risk. * SSA: The CIO rating on the Dashboard differed from the risks portrayed in underlying documentation for 1 of the 10 selected SSA investments. In particular, SSA's Citizen Access Routing Enterprise through 2020 investment was described as both having zero percent schedule variance in project management reports, but also being more than a year behind schedule and unable to determine a milestone date in a CIO review. According to SSA officials from the Office of Systems, this is because of the new milestones and corresponding baseline that all SSA investments create at the start of every fiscal year based on annual funding amounts received from SSA's Strategic Information Technology Assessment and Review process. SSA describes this process further in its Information Resources Management Strategic Plan,[Footnote 28] which states that SSA makes adjustments each year based on current goals, objectives, and measures. Such changes, known as rebaselines,[Footnote 29] erase past cost and schedule variances, prevent the agency from monitoring year-to-year investment performance, and make it difficult to estimate and understand life- cycle costs. While this only affected one investment we reviewed, it has the potential to impact SSA's entire IT investment portfolio. As described in our Cost Estimating and Assessment Guide, the purpose of such baseline changes should be to restore management's control of the remaining effort by providing a meaningful basis for performance management. Further, these changes should be infrequent, and recurrent changes may indicate that the scope is not well understood or simply that program management is unable to develop realistic estimates. [Footnote 30] Regarding SSA, frequent rebaselining increases the risk that its investments have undetected cost or schedule variances that will impact the success of the associated investment. While agencies experienced several issues with reporting the risk of their investments, such as technical problems and delayed updates to the Dashboard, the CIO ratings were mostly or completely consistent with investment risk at seven of the eight selected agencies. Additionally, the agencies had already addressed several of the discrepancies that we identified. The final agency, VA, did not update 7 of its 10 selected investments because it elected to build, rather than buy, the ability to automatically update the Dashboard, and has now resumed updating all investments. To their credit, agencies' continued attention to reporting the risk of their major investments supports the Dashboard's goal of providing transparency and oversight of federal IT investments. Nevertheless, the rating issues that we identified with performance reporting and annual baselining, some of which are now corrected, serve to highlight the need for agencies' continued attention to the timeliness and accuracy of submitted information, in order to allow the Dashboard to continue to fulfill its stated purpose. Highest-Risk Investments Were Appropriately Addressed by Selected Agencies: We have previously concluded that, consistent with government and industry best practices, including our own guidance on IT investment management, agencies' highest-risk investments should receive additional management attention.[Footnote 31] In particular, agency leaders should use data-driven reviews as a leadership strategy to drive performance improvement.[Footnote 32] Correspondingly, OMB requires reviews, known as TechStat sessions, to enable the federal government to intervene to turnaround, halt, or terminate IT projects that are failing or are not producing results. Of the 80 investments we reviewed, there were 8 investments at four of the eight selected agencies that received a red (high or moderately high risk) rating in 2012. See figure 6 for a list of those at-risk investments and the associated CIO ratings. Figure 6: Dashboard Ratings in 2012 for the Eight At-Risk Investments: [Refer to PDF for image: illustrated table] Agency: Agriculture; Human Resources Line of Business: Service Center: 2012: January: Moderately High Risk; February: Medium Risk; March: Medium Risk; April: Medium Risk; May: Medium Risk; June: Medium Risk; July: Medium Risk; August: Medium Risk; September: Medium Risk; October: Medium Risk; November: Medium Risk; December: Medium Risk. Farm Program Modernization: 2012: January: Low Risk; February: Low Risk; March: Low Risk; April: Low Risk; May: Low Risk; June: Low Risk; July: Low Risk; August: Low Risk; September: Low Risk; October: Low Risk; November: Medium Risk; December: High Risk. Financial Management Modernization Initiative: 2012: January: Low Risk; February: Low Risk; March: Low Risk; April: Low Risk; May: Low Risk; June: Medium Risk; July: Medium Risk; August: Medium Risk; September: Medium Risk; October: Medium Risk; November: Moderately High Risk; December: Medium Risk. Agency: Commerce; Joint Polar Satellite System Ground System: 2012: January: Moderately High Risk; February: Moderately High Risk; March: Moderately High Risk; April: Moderately High Risk; May: Moderately High Risk; June: Moderately High Risk; July: Moderately High Risk; August: Moderately High Risk; September: Moderately High Risk; October: Moderately High Risk; November: Moderately High Risk; December: Moderately High Risk. Census IT Infrastructure: 2012: January: Medium Risk; February: Medium Risk; March: Medium Risk; April: Medium Risk; May: Medium Risk; June: Medium Risk; July: Medium Risk; August: Moderately High Risk; September: Moderately High Risk; October: Low Risk; November: Low Risk; December: Low Risk. Patents End-to-End: Software Engineering: 2012: January: Moderately High Risk; February: Medium Risk; March: Medium Risk; April: Medium Risk; May: Medium Risk; June: Medium Risk; July: Medium Risk; August: Medium Risk; September: Medium Risk; October: Medium Risk; November: Medium Risk; December: Medium Risk. Agency: Justice; Sentinel: 2012: January: Moderately High Risk; February: Moderately High Risk; March: Moderately High Risk; April: Moderately High Risk; May: Moderately High Risk; June: Medium Risk; July: Medium Risk; August: Low Risk; September: Low Risk; October: Low Risk; November: Low Risk; December: Low Risk. Agency: Transportation; En Route Automation Modernization: 2012: January: High Risk; February: Moderately High Risk; March: Moderately High Risk; April: Moderately High Risk; May: Medium Risk; June: Medium Risk; July: Medium Risk; August: Medium Risk; September: Medium Risk; October: Medium Risk; November: Medium Risk; December: Medium Risk. Source: GAO analysis of data from OMB’s Dashboard. [End of figure] Of the eight investments receiving a red rating in 2012, the agencies reviewed seven using tools such as TechStat sessions,[Footnote 33] department investment review boards, and other high-level reviews. Each of these resulted in action items intended to improve performance. The final investment, Agriculture's Human Resources Line of Business: Service Center was scheduled to be reviewed using a TechStat session, but in August 2013 officials from Agriculture's Office of the CIO stated that this investment was in the process of going through a rebaseline. As noted earlier, such changes should be infrequent and reinforce the need for agencies to review the highest- risk investments to ensure that root causes of baseline changes are effectively addressed. Each of these agencies we reviewed had similar approaches to identifying and conducting these reviews. The agencies with red-rated investments in 2012--Agriculture, Commerce, Justice, and Transportation--monitored investment performance, identified troubled investments using a variety of criteria, reviewed investments in need of attention, and assigned and tracked action items. For example, both Agriculture and Commerce identified investments for review using the Dashboard's CIO ratings. All four agencies further ensured that investments implemented corrective actions resulting from management reviews. For example, as a result of its November 2012 TechStat, Agriculture leaders assigned 17 action items to the Farm Program Modernization investment, which were to be completed by January 2013. Additionally, the four remaining agencies without red-rated investments--Energy, Treasury, VA, and SSA--have documented processes which provide comparable monitoring and oversight capabilities. For example, VA's process relies on missed deliverable dates as the key requirement to hold a review, after which senior management makes a determination as to the future of the investment. Alternatively, Energy reviews all major IT investments on a quarterly basis and requires those which fall outside expected thresholds to complete corrective actions. As such, all of the agencies have focused management attention on troubled investments and once identified, have established and followed through on clear action items to turn around or terminate such investments. Conclusions: Since its inception in 2009, the IT Dashboard has increased the transparency of the performance of major federal IT investments. Its CIO ratings, in particular, have improved visibility into changes in the risk level of investments over time, as well as agencies' ability to accomplish their goals. To that end, over the past 4 years, the agencies we reviewed have reported lower risk for more than one quarter of all their investments. However, the effectiveness of the Dashboard depends on the quality of the information that agencies submit. We previously recommended that OMB clarify guidance on whether certain types of systems should be included in IT reporting, but OMB did not agree. Agencies' subsequent decisions to remove investments from the Dashboard by changing investments' categorizations represent a troubling trend toward decreased transparency and accountability. Thus, we continue to believe that our prior recommendation remains valid and should be implemented. Additionally, OMB's annual freeze of the Dashboard for as long as 8 months negates one of the primary purposes of this valuable tool--to facilitate transparency and oversight of the government's billions of dollars in IT investments. Beyond the transparency they promote, CIO ratings present an opportunity to improve the data and processes agencies use to assess investment risk. Each of the agencies we examined had established such processes, and most of the resulting Dashboard ratings were consistent with underlying documentation. While two agencies, Transportation and Treasury, had ratings that accurately reflected their investments' supporting documentation, other agencies' Dashboard ratings were inconsistent with the actual risk of the associated investment. These inconsistencies--such as inaccurate reflection of negative performance, questionable decisions about the recategorization of investments, and too-frequent changes to performance baselines-- highlight the continued need for agencies to populate the Dashboard with an accurate representation of the full breadth and health of their investments. In providing this information, agencies will help the Dashboard accomplish its goal of providing transparency and oversight of millions of dollars in federal IT investments. Recommendations for Executive Action: To better ensure that the Dashboard provides meaningful ratings and reliable investment data, we are recommending that the Director of OMB direct the Federal CIO to make accessible regularly updated portions of the public version of the Dashboard (such as CIO ratings) independent of the annual budget process. In addition, to better ensure that the Dashboard provides accurate ratings, we are making three recommendations to the heads of three of the selected agencies. Specifically, we are recommending that: * The Secretary of Commerce direct the department CIO to ensure that the department's investments are appropriately categorized in accordance with existing statutes and that major IT investments are included on the Dashboard. * The Secretary of Energy direct the department CIO to ensure that the department's investments are appropriately categorized in accordance with existing statutes and that major IT investments are included on the Dashboard. * The Commissioner of the SSA direct the CIO to revise the agency's investment management processes to ensure that they are consistent with the baselining best practices identified in our published guidance on cost estimating and assessment. Agency Comments and Our Evaluation: We received comments on a draft of this report from OMB and all eight departments and agencies in our review. OMB neither agreed nor disagreed with our recommendation; Agriculture agreed with the report's findings; Commerce disagreed with our recommendation; Energy concurred with our recommendation with one exception related to supercomputers; Justice, Treasury, and Transportation neither agreed nor disagreed with the report's findings; VA agreed with the report's findings; and SSA agreed with our recommendation. Each agency's comments are discussed in more detail below. * In written comments, the Federal CIO neither agreed nor disagreed with our recommendation for OMB to make accessible regularly updated portions of the public version of the Dashboard (such as CIO ratings) independent of the annual budget process. However, OMB also noted that the manner in which agencies submit Dashboard information to OMB makes it difficult to separate materials it believes are predecisional or deliberative. Despite this, OMB agreed to explore whether it would be practical to make the Dashboard more accessible, and to consider how it could separate predecisional or deliberative materials. We support OMB in its efforts to increase public transparency and accountability. OMB continued to disagree with the 2011 recommendation that we believe continues to have merit, related to the clarity of its guidance on whether certain types of systems should be included in IT reporting. OMB believes that the existing guidance is appropriate and does not have plans to review it. However, as noted earlier in this report, we believe that the recommendation is appropriate because the existing guidance does not address key categories of IT investments where we continued to find inconsistencies. OMB also disputed our characterization of two issues. First, OMB noted that up-to-date Dashboard information is available to the agencies and OMB during the budget development period, who use it as a tool for investment oversight and decision making. While we acknowledge that OMB and federal agencies continue to have access to the Dashboard during the budget process, the public display of these data is intended to allow other oversight bodies, including Congress and the general public, to hold government agencies accountable for progress and results. Secondly, OMB stated that it began using the IT Dashboard to help identify at-risk investments starting with its launch in June 2009, rather than 2010. We modified the report to reflect OMB's statement. OMB also provided technical comments, which we have incorporated in the report as appropriate. OMB's written comments are provided in appendix III. * In written comments, Agriculture's CIO stated that the department concurred with the content of the report and had no comments. Agriculture's written comments are provided in appendix IV. * In written comments, Commerce agreed with most of the report's findings but disagreed with our recommendation to ensure that the department's investments are appropriately categorized in accordance with existing statutes and that major IT investments are included on the Dashboard. Specifically, Commerce stated that although it is no longer reporting publicly on its satellite ground system investments through the Dashboard, neither the department CIO nor OMB has relinquished its oversight role. Moreover, Commerce stated that it is reviewing its satellites more frequently and in more detail; as an example, the department noted that Commerce's CIO conducts monthly Dashboard-like assessments that cover the status of the satellite investments. However, regardless of these additional efforts, the removal of the satellite investments from the Dashboard prevents the public display of these data intended to allow OMB and other oversight bodies, including Congress, to hold the government agencies accountable for progress and results. Additionally, as discussed in this report, Commerce's recategorization of its satellite ground system investments is contrary to the definition of IT as set forth in the Clinger-Cohen Act. Based on these facts, we continue to believe that our recommendation to Commerce that its CIO ensure that the department's investments are appropriately categorized in accordance with existing statutes and that major IT investments are included on the Dashboard has merit and should be implemented. Commerce's written comments are provided in appendix V. The department also provided technical comments related to our characterization of the department's rating process, which we have incorporated in the report as appropriate. * In written comments, Energy concurred, but with a comment regarding our recommendation to ensure that the department's investments are appropriately categorized in accordance with existing statutes and that major IT investments are included on the Dashboard. Specifically, while agreeing that Energy should ensure that its IT investments are appropriately categorized, the department stated that it would continue to report investments that it believes should be categorized and managed on the Dashboard, noting that supercomputers would be an exception to this policy. However, this exception runs contrary to the finding on which this recommendation is based, namely that Energy removed $368 million in investments from the Dashboard, including a supercomputer reported in 2012 as the most powerful in the world. While Energy agreed to partner with OMB to develop an alternate mechanism to track and report supercomputer investments to OMB, it is not clear whether this information will be publicly available. Additionally, Energy's recategorization of its supercomputer investments is contrary to the definition of IT as set forth in the Clinger-Cohen Act, as discussed in this report. Therefore, we disagree that the department's planned actions adequately address existing requirements for open and transparent reporting of major IT investments, and we stand by our assessment that Energy's CIO should ensure that the department's investments are appropriately categorized in accordance with existing statutes and that major IT investments are included on the Dashboard. Energy's written comments are provided in appendix VI. * In comments provided via e-mail on November 6, 2013, a representative of Justice's Justice Management Division stated that the department had no comments. * In comments provided via e-mail on November 1, 2013, Transportation's Deputy Director of Audit Relations stated that the department had no comments. The department also provided technical comments, which we have incorporated in the report as appropriate. * In written comments, the Treasury's CIO stated that the department had no comments. Treasury's written comments are provided in appendix VII. * In written comments, VA's Chief of Staff stated that the department generally agreed with the findings of the report and provided general comments in which it confirmed that issues identified in the report had been addressed. Specifically, VA confirmed that the department has implemented the capability to automatically submit CIO ratings to the Dashboard and now uses more specific data to generate the CIO rating, such as the number of "red flags" associated with an investment or the number of TechStat reviews held. Finally, while stating that the department will continue its efforts to improve timely and accurate reporting to the Dashboard, VA also noted that it is important to recognize that the department manages investment delivery at the project level in comparison to the investment level found in the Dashboard. VA's written comments are provided in appendix VIII. * In written comments, SSA's Deputy Chief of Staff agreed with our recommendation to revise the agency's investment management processes to ensure that they are consistent with the baselining practices, and discussed planned actions to address it. SSA's written comments are provided in appendix IX. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to interested congressional committees; the Secretaries of Agriculture, Commerce, Energy, Transportation, Treasury, Veterans Affairs, Attorney General of the United States, Acting Commissioner of Social Security, the Director of the Office of Management and Budget; and other interested parties. In addition, the report will be available at no charge on the GAO website at [hyperlink, http://www.gao.gov]. If you or your staffs have any questions on matters discussed in this report, please contact me at (202) 512-9286 or pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix X. Signed by: David A. Powner: Director, Information Technology: Management Issues: [End of section] Appendix I: Objectives, Scope, and Methodology: Our objectives for this engagement were to (1) characterize the Chief Information Officer (CIO) ratings for selected federal agencies' information technology (IT) investments as reported on the Dashboard over time, (2) determine the extent to which selected agencies' CIO ratings are consistent with reported investment risk, and (3) determine the extent to which selected agencies are addressing at-risk investments. To address our first objective, we selected the eight agencies with the most reported major IT spending in fiscal year 2012, after excluding agencies reviewed in our most recent Dashboard report. [Footnote 34] The eight agencies are the Departments of Agriculture (Agriculture), Commerce (Commerce), Energy (Energy), Justice (Justice), Transportation (Transportation), the Treasury (Treasury), and Veterans Affairs (VA), and the Social Security Administration (SSA). The results in this report represent only these agencies. We downloaded and examined the Dashboard's CIO ratings for all major investments at the eight agencies (a total of 383 investments reported by these agencies).[Footnote 35] To characterize the numbers and percentages of major IT investments at each risk level at each of our subject agencies, we analyzed, summarized, and--where appropriate-- graphically depicted average CIO ratings for investments by agencies over time during the period from June 2009 to August 2013. Specifically, we compared each investment's first and last CIO ratings (including those investments that were not on the Dashboard at its inception, and those that were downgraded or eliminated) and summarized the data by agency. To describe whether CIO ratings indicated higher or lower investment risk over time, we calculated the numbers and percentages of investments (by agency and collectively for all the agencies) that maintained a constant rating over the entire performance period, and those that experienced a change to their CIO rating in at least one rating period. Then we analyzed the subset of investments that experienced at least one changed rating and compared the first CIO rating with the latest CIO rating (no later than August 2013) to determine the numbers and percentages of investments (by agency and collectively for all the agencies) that experienced a net rating increase, a net rating decrease, or no net change. We also reviewed investments which had been removed from the Dashboard due to recategorization and compared their definitions to the Clinger-Cohen Act's[Footnote 36] definition of IT. We presented our results to each agency and the Office of Management and Budget (OMB) and solicited their input, explanations for the results, and additional corroborating documentation, where appropriate. To accomplish the second objective, we selected the top 10 major investments at each of the eight selected agencies (for a total of 80 investments), with the highest reported IT spending in fiscal year 2012. We reviewed investment documentation (such as executive-level briefings, operational analyses, and TechStat results) and interviewed officials from each of the agencies to understand how they rate and monitor investments.[Footnote 37] Within these artifacts, we identified representations of investment risk and performance (such as cost and schedule variances), which could impact the success of the associated investment. We then organized the results by month, and compared these results to the relevant investment Dashboard CIO ratings for calendar year 2012.[Footnote 38] We then evaluated whether each investment's monthly CIO rating was consistent with reported investment risks, and categorized each investment by the number of months which were inconsistent. Investments which were consistent with underlying documentation for every month in 2012 were categorized as "consistent," those which were inconsistent for 1 or more months were categorized as "partially inconsistent," and those with inconsistencies in each month were "inconsistent." Finally, for our third objective, we reviewed and described each of the eight selected agencies' processes used to address the highest- risk major investments (such as TechStat guides and investment review board results). Among the 80 investments selected for the second objective, we identified investments which had received a "red" (high or moderately high) rating on the Dashboard during 2012. We reviewed the processes used to address these investments and interviewed relevant officials. We also examined the results of performance reviews and interviewed officials to determine whether the agencies implemented the appropriate risk management processes to oversee and review the selected investments. We conducted this performance audit from February to December 2013 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of section] Appendix II: Selected Investments: Below is the list of the investments that are included in this review, as well as the reported fiscal year 2012 IT spending. Table 5: Investments Selected for Review and Associated Fiscal Year 2012 Spending: Agency: Agriculture; Investment title: Forest Service Computer Base; IT spending: $256.5 million. Investment title: Human Resources Line of Business: Service Center; IT spending: $138.3 million. Investment title: Farm Program Modernization; IT spending: $112.4 million. Investment title: Comprehensive Loan Program; IT spending: $64.0 million. Investment title: Financial Management Modernization Initiative; IT spending: $63.1 million. Investment title: Public Health Data Communications Infrastructure System; IT spending: $53.3 million. Investment title: Public Safety Land Mobile Radio System; IT spending: $43.8 million. Investment title: Animal and Plant Health Inspection Service Enterprise Infrastructure; IT spending: $34.0 million. Investment title: Optimized Computing Environment; IT spending: $30.6 million. Investment title: Natural Resource Manager; IT spending: $30.4 million. Agency: Commerce; Investment title: Geostationary Operational Environmental Satellite R Series Ground Segment; IT spending: $257.0 million. Investment title: United States Patent and Trademark Office Network and Security Infrastructure; IT spending: $221.4 million. Investment title: Joint Polar Satellite System Ground System; IT spending: $153.6 million. Investment title: National Oceanic and Atmospheric Administration IT Infrastructure; IT spending: $132.6 million. Investment title: Census IT Infrastructure; IT spending: $130.0 million. Investment title: Census Economic Census and Surveys; IT spending: $74.9 million. Investment title: Advanced Weather Interactive Processing System; IT spending: $63.3 million. Investment title: Consolidated Herbert C. Hoover Building IT Infrastructure; IT spending: $56.5 million. Investment title: United States Patent and Trademark Office Patents End-to-End: Software Engineering; IT spending: $46.6 million. Investment title: National Climatic Data Center; IT spending: $42.6 million. Agency: Energy; Investment title: Consolidated Infrastructure, Office Automation, and Telecommunications Program; IT spending: $967.9 million. Investment title: Sequoia Platform; IT spending: $101.9 million. Investment title: Oak Ridge National Laboratory, Leadership Computing Facility - Direct Mission; IT spending: $94.0 million. Investment title: Argonne National Lab, Leadership Computing Facility - Direct mission; IT spending: $62.2 million. Investment title: Lawrence Berkeley National Lab, National Energy Research Scientific Computing Center -Direct mission; IT spending: $57.8 million. Investment title: SR Mission Support Systems; IT spending: $44.6 million. Investment title: Lawrence Berkeley National Lab, Energy Sciences Network - Direct Mission; IT spending: $34.5 million. Investment title: Identity, Credential, and Access Management; IT spending: $33.3 million. Investment title: Office of the Chief Financial Officer iManage; IT spending: $29.1 million. Investment title: Simulation-Based Engineering User Center; IT spending: $17.5 million. Agency: Justice; Investment title: Federal Bureau of Investigation (FBI) Next Generation Identification; IT spending: $220.5 million. Investment title: Justice Management Division Law Enforcement Wireless Communications; IT spending: $87.0 million. Investment title: Justice Management Division Unified Financial Management System; IT spending: $83.8 million. Investment title: Drug Enforcement Administration Firebird; IT spending: $82.9 million. Investment title: FBI Data Integration and Visualization System; IT spending: $59.0 million. Investment title: FBI Data Centers; IT spending: $57.1 million. Investment title: FBI Sentinel; IT spending: $46.1 million. Investment title: FBI National Crime Information Center; IT spending: $45.1 million. Investment title: FBI Terrorist Screening System; IT spending: $43.7 million. Investment title: FBI Digital Collection; IT spending: $43.2 million. Agency: Transportation; Investment title: Federal Aviation Administration (FAA) Automatic Dependent Surveillance-Broadcast; IT spending: $301.5 million. Investment title: FAA En Route Automation Modernization; IT spending: $271.3 million. Investment title: FAA Data Communications NextGen Support; IT spending: $146.2 million. Investment title: FAA Telecommunications Infrastructure; IT spending: $143.0 million. Investment title: FAA Terminal Automation Modernization and Replacement Phase III; IT spending: $136.7 million. Investment title: FAA Wide Area Augmentation System; IT spending: $123.4 million. Investment title: FAA Oceanic Automation System: Advanced Technologies and Oceanic Procedures; IT spending: $102.7 million. Investment title: FAA Terminal Automation Replacement System; IT spending: $88.6 million. Investment title: FAA System Wide Information Management; IT spending: $74.3 million. Investment title: FAA Regulation and Certification Infrastructure for System Safety; IT spending: $60.0 million. Agency: Treasury; Investment title: Internal Revenue Service (IRS) Main Frames and Servers Services and Support; IT spending: $449.3 million. Investment title: IRS Telecommunications Systems and Support; IT spending: $218.6 million. Investment title: IRS Customer Account Data Engine 2; IT spending: $210.3 million. Investment title: IRS End User Systems and Services; IT spending: $202.0 million. Investment title: Electronic Federal Tax Payment System; IT spending: $70.9 million. Investment title: Mainframe and Servers Support and Services; IT spending: $69.9 million. Investment title: Treasury Enterprise Identity, Credential, and Access Management; IT spending: $69.3 million. Investment title: Financial Management Service IT Infrastructure Mainframe and Servers Services and Support; IT spending: $59.7 million. Investment title: IRS Modernized e-File; IT spending: $49.1 million. Investment title: Retail Securities Services; IT spending: $45.9 million. Agency: VA; Investment title: Medical IT Support; IT spending: $1.070 billion. Investment title: Enterprise IT Support; IT spending: $563.5 million. Investment title: Benefits IT Support; IT spending: $199.8 million. Investment title: Corporate IT Support Enterprise Cyber Security and Privacy; IT spending: $172.9 million. Investment title: Medical Legacy; IT spending: $155.6 million. Investment title: Benefits 21st Century Paperless Delivery of Veterans Benefits; IT spending: $140.4 million. Investment title: InterAgency 21st Century-One Vet; IT spending: $136.9 million. Investment title: Medical 21st Century Development Core; IT spending: $85.5 million. Investment title: Benefits 21st Century Education; IT spending: $65.2 million. Investment title: InterAgency 21st Century Veterans Interoperability; IT spending: $52.0 million. Agency: SSA; Investment title: Infrastructure - Data Center; IT spending: $407.4 million. Investment title: Infrastructure - Office Automation; IT spending: $221.9 million. Investment title: Infrastructure - Telecommunications; IT spending: $204.6 million. Investment title: Disability Case Processing System; IT spending: $68.5 million. Investment title: Telephone Systems Replacement Project; IT spending: $51.9 million. Investment title: Intelligent Disability; IT spending: $50.5 million. Investment title: Disability Determination Services Automation; IT spending: $31.1 million. Investment title: Citizen Access Routing Enterprise through 2020; IT spending: $25.1 million. Investment title: National Support Center; IT spending: $23.0 million. Investment title: Earnings Redesign; IT spending: $17.8 million. Source: GAO analysis of data from OMB's IT Dashboard. [End of table] [End of section] Appendix III: Comments from the Office of Management and Budget: Executive Office of The President: Office of Management and Budget: Washington, D.C. 20503: November 25, 2013: Mr. David Powner: Director: Information Technology Management Issues: Government Accountability Office: 441 G St. NW: Washington D.C. 20548: Dear Mr. Powner: Thank you for providing the Office of Management and Budget (OMB) the opportunity to review the draft of GAO's report on "IT Dashboard: Agencies are Managing Investment Risk, but Improvements Needed in Reporting "(GAO-14-64) and to provide you with OMB's comments on the draft report. As initial matter, we appreciate the time and energy that GAO has devoted to the review of the IT Dashboard, which was launched in June 2009. As the draft report notes, GAO has previously issued several reports on the IT Dashboard over the years. In the draft report, GAO outlines the results of its review of the investment risk information that, as of March 2013, eight agencies had submitted to the IT Dashboard for 157 major information technology investments. Those eight agencies are the Departments of Agriculture, Commerce, Energy, Justice, Transportation, Treasury, and Veterans Affairs, and the Social Security Administration (SSA). In addition, the draft report includes three recommendations that are addressed to specific agencies (one recommendation each to Commerce, Energy, and SSA) and one recommendation that is addressed to OMB. It is our understanding that GAO is also providing the eight agencies an opportunity to comment on the draft report. As each agency is in the best position to provide comments on the findings and recommendations regarding that agency, OMB has focused its review of the draft on its more general aspects as well as its draft recommendation. The draft report includes the following recommendation to OMB: "To better ensure that the Dashboard provides meaningful ratings and reliable investment data, we are recommending that the Director of OMB direct the Federal CIO to make accessible regularly-updated portions of the public version of the Dashboard (such as CIO ratings) independent of the annual budget process." This draft recommendation is related to the statement in the "Conclusions" section that "OMB's annual freeze of the Dashboard for as long as 8 months negates one of the primary purposes of this valuable tool — to facilitate transparency and oversight of the government's billions of dollars in IT investments." And, this recommendation is related to the following discussion in the main body of the draft report: "Although intended to be a mechanism to enhance transparency, OMB does not update the public version of the Dashboard as it and the agencies work to assist in the formulation of the President's budget request. Nevertheless, agencies continue to provide updates during this time. For the fiscal year 2013 presidential budget request, the static period during which the public saw no updates lasted for 7 months (September 2011 to March 2012) and for fiscal year 2014 the period lasted for 8 months (September 2012 to April 2013). In total, OMB did not update the public Dashboard for 15 of the past 24 months. An Office of E-Government staff member stated that OMB considers the Dashboard information submitted during the budget process to be pre- decisional and will only release it to the public in tandem with the President's budget. However, the CIO rating does not contain any of the information that the Dashboard's webpage for frequently asked questions identifies as predecisional, namely: "new" investments, planned IT spending levels in the budget year, costs of projects and activities that complete during or after the budget year, and eliminated or downgraded investments. Given that the CIO rating does not include such data, the rating could be updated independent of the budget process without disclosing information that OMB views as pre-decisional. However, by withholding current investment performance information for much of the year, OMB decreases the utility of the Dashboard as a tool for greater IT investment oversight and transparency." (footnote omitted) In evaluating possible changes, OMB will need to take into account several considerations. As an initial matter, OMB believes strongly in the effectiveness of public transparency, through tools like the IT Dashboard, in increasing accountability and improving performance. It is for this reason that the IT Dashboard was launched in June 2009, as the draft report notes at page 10 ("We have previously reported that OMB has taken significant steps to enhance the oversight, transparency, and accountability of federal IT investments by creating its IT Dashboard, and by improving the accuracy of investment ratings."). At that same time, it is also important to preserve the confidentiality of the deliberations that are at the core of the budget-development process, which involves the identification, evaluation and consideration of budgetary alternatives. Preserving the confidentiality of this process has been a longstanding Executive Branch interest (and is discussed in Section 22 of OMB Circular A-11). With respect to the IT Dashboard, the agencies' submissions to OMB during the budget-development period include both factual information as well as predecisional deliberative materials. Moreover, the manner in which they are submitted to OMB makes it difficult, as a practical matter, to separate the factual information from the predecisional deliberative materials. As a result, given the existing data model and application design, OMB is currently not in a position to release the CIO ratings and other "regularly-updated portions" during the budget- development period without, at the same time, also releasing predecisional deliberative materials. In evaluating whether changes could be made that would make it practicable to post additional information on the IT Dashboard during the budget-development period we will consider the extent to which factual information can be more easily separated from the predecisional deliberative materials through the adoption of such approaches as revising the data model, application design, and agency submission systems; providing more extensive training and guidance to agencies regarding the preparation of their submissions; and undertaking an additional manual review of the submissions. OMB will explore whether it would be practicable to make changes, such as to the IT Dashboard data model and application design, that would enable the release during the budget-development period of "regularly- updated portions" such as CIO rating information. (The practice up to now has been that the CIO ratings and other regularly updatable fields are publicly released after the Budget is submitted to Congress, and they continue to be released until the beginning of the next budget- development process.) In addition, in its "Conclusions" section, the draft report reiterates a recommendation that GSA had made in a prior report: "... the effectiveness of the Dashboard depends on the quality of the information that agencies submit. We previously recommended that OMB clarify guidance on whether certain types of systems should be included in IT reporting, but OMB did not agree. Agencies' subsequent decisions to remove investments from the Dashboard by changing investments' categorizations represent a troubling trend toward decreased transparency and accountability. Thus, we continue to believe that our prior recommendation remains valid and should be implemented." As the draft report also explains, this recommendation had been included in GAO's report of September 2011 on "Information Technology: OMB Needs to Improve its Guidance on IT Investments" (GA0-11-826). That report describes OMB's comments, on the draft of this recommendation, as follows (page 24): "agency officials noted that existing guidance (including OMB circular A-11 and OMB memo 11-29) already discusses how to identify IT investments." In response to the current draft report's reiteration of the 2011 recommendation, we continue to believe that the existing guidance is appropriate, and do not have plans to revise it at this time (as Circular A-11 is updated annually, we review its guidance each year). Finally, we would like to clarify three factual inaccuracies in the draft report. First, in the "What GAO Found" section, the draft states that, because the public version of the IT Dashboard was not updated during the budget-development periods, it "was not available as a tool for investment oversight and decision-making" during those periods. While the public version was not updated during those periods (for the reasons discussed above), Federal agencies and OMB have continued to have access — throughout the year, including during those periods —to IT Dashboard information that is up-to-date. Thus, up-to-date Dashboard information is available to the agencies and OMB as a tool for investment oversight and decision-making. Second, in the "Why GAO did this Study" section and at page 3, the draft states that — "In January 2010, OMB began using the Dashboard as one of several tools to identify at-risk investments." While it is true that OMB began to hold TechStat reviews in 2010, OMB began using the IT Dashboard to help identify at-risk investments starting with its launch in June 2009. Third, as is noted above, the draft states on page 20 that — "Given that the CIO rating does not include [pre-decisional] data, the rating could be updated independent of the budget process without disclosing information that OMB views as pre-decisional." As is explained above, it is currently not practicable to separate the factual information from the pre-decisional deliberative materials during the budget- development period. However, as we have further stated, we will explore whether it would be practicable to make changes, such as to the IT Dashboard data model and application design, that would enable the release during the budget-development period of "regularly-updated portions" such as CIO rating information. Thank you again for providing OMB the opportunity to submit comments on the draft. Sincerely, Signed by: Steve VanRoekel: U.S. Chief Information Officer: Office of Management and Budget: [End of section] Appendix IV: Comments from the Department of Agriculture: USDA: United States Department of Agriculture: Departmental Management: Office of the Chief Information Officer: 1400 Independence Avenue S.W. Washington, DC 20250: David Powner: Director: Information Technology Management Issues: U.S. Government Accountability Office: 441 G Street, N.W. Washington, DC 20548: Dear Mr. Powner: The U.S. Department of Agriculture has reviewed the draft report GAO Draft Report GAO-14-64, "IT Dashboard: Agencies are Managing Investment Risk, but Related Ratings Need to Be More Accurate and Available" (job code 311285). Thank you for the opportunity to respond to the GAO draft report. We concur with the content of the report and have no comments. For additional information, please contact Christopher Wren, Office of the Chief Information Officer's audit liaison, at 202-260-0771. Sincerely, Signed by: Cheryl L. Cook: Chief Information Officer: [End of section] Appendix V: Comments from the Department of Commerce: The Deputy Secretary of Commerce: Washington, D.C. 20230: November 14, 2013: Mr. David A. Powner: Director, Information Technology Management Issues: U.S. Government Accountability Office: Washington, DC 20548: Dear Mr. Powner: Thank you for the opportunity to comment on the draft report from the U.S. Government Accountability Office (GAO) entitled, IT DASHBOARD: Agencies are Managing Investment Risk but Related Ratings Need to Be More Accurate and Available (GAO-14-64). We agree with most of the findings as they apply to the Department of Commerce IT Dashboard ratings, but we do not agree with the executive recommendation regarding satellite oversight, for the reasons stated in the attached comments_ We also believe the GAO's characterization of certain issues raised about the Department's rating processes and results can be improved to better illuminate the context within which certain decisions were made. If you have any questions about our response, please contact Stuart Simon, Office of IT Policy and Planning, at (202) 482-0275. Sincerely, Signed by: Patrick Gallagher: Acting Deputy Secretary of Commerce: U.S. Department of Commerce Comments on the United States Government Accountability Office Draft Report, IT Dashboard: Agencies are Managing investment Rick, but Related Ratings Need to Be More Accurate and Available GAO-14-64, November 2013. 1) On Page 17: Add the words "clarity of after the word "factor" in the third sentence to read: "Both of these agencies factor clarity of investment reporting..." 2) On Page 19: At the end of the first paragraph regarding the Commerce Department's decision to not classify satellites systems as IT, we recommend adding a sentence providing significant context, namely, "This change in satellite systems IT reporting makes Commerce practice consistent with the practice of other federal agencies, including NASA, which is now the Department of Commerce's prime contractor for developing its next generation polar and geostationary satellites." 3) On Page 22: Replace the last sentence at the end of the paragraph about Commerce with this one: "In both cases, Commerce officials made appropriate correction the next month, based on information received late in the Department's previous month's internal reporting cycle." 4) The Department does not concur with GAO's conclusions and recommendation regarding Departmental oversight of satellite systems as presented on pages 19 and 29 of the draft report. Although satellite ground system investments are no longer being publicly reported through the Federal IT Dashboard, neither the Department CIO nor the OMB has relinquished their oversight roles. Following the issuance of the September 2013 memorandum cited below, the satellites began being reviewed by the Department more holistically, more frequently, and in more detail than ever before. Specifically: * A decision memorandum issued by Acting Secretary Rebecca Blank in September 2012 called for reviews of satellite programs at the Department level to cover "all oversight elements (e.g., financial, IT related, etc.)." * A second memo on the subject of follow-on activities for oversight of Commerce satellite programs was issued shortly thereafter by Justin Ehrenwerth (then Chief of Staff to the Deputy Secretary). This memo assigned to the Commerce Department CIO responsibility for "Development and execution of IT oversight policies and procedures" and "Oversight and assessment of IT investment performance." * Abby Harper, NOAA Risk Management Officer, meets regularly with key staff from the Commerce Department's Office of Budget, Office of Acquisition Management, Office of the CIO, and Office of Program Evaluation and Risk Management to address any concerns regarding the satellite investments' status, risks, or other issues. * Although the satellite investments are no longer publicly reported on the Federal IT Dashboard, when the CIO does his monthly IT Dashboard assessment review with his staff, they cover the status of the major satellite IT investments, namely GOES-R Series Ground Segment, JPSS Ground System, JASON-3, and DSCOVR. While Commerce is no longer publicly reporting a CIO rating for these investments, IT Dashboard-like assessments continue to take place monthly. [End of section] Appendix VI: Comments from the Department of Energy: Department of Energy: Washington, DC 20585: November 8, 2013: Mr. David A. Powner: Director, Information Technology Management Issues: U.S. Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Dear Mr. Powner: The Department of Energy's (DOE) Office of the Chief Information Officer (OCIO) appreciates the opportunity to provide comments to the Government Accountability Office's (GAO) Draft Information Technology (IT) Dashboard Report, Agencies are Managing Investment Risk, but Related Ratings Need to Be More Accurate and Available (GA0-14-64). This audit was conducted to review the consistency of investment information submitted by federal agencies to the Office of Management and Budget's (OMB) IT Dashboard. We offer the following management response to the Energy recommendation: Recommendation: Ensure that the Department's investments are appropriately categorized in accordance with existing legislation and that IT investments are included on the Dashboard. DOE concurs with comment: 1) DOE will ensure its IT investments are categorized in accordance with existing legislation and OMB guidance and direction. We will also continue to report all major IT investments that we believe should be categorized and managed as IT investments on the Dashboard, with the exception of the supercomputers referenced in the report. 2) Senior officials from DOE and OMB have discussed proposals for reporting supercomputer investment data over the past year. As part of the 2013 PortfolioStat session, OMB agreed that DOE does not have to report the supercomputers on the IT Dashboard. Instead, DOE agreed to partner with OMB to develop an alternate mechanism to track and report supercomputer investments to OMB in order to provide increased transparency and oversight. Thank you for the opportunity to review this report. If you have any questions related to this letter, please feel free to contact me at (202) 586-0166. Sincerely, Signed by: Robert F. Brese: Chief Information Officer: [End of section] Appendix VII: Comments from the Department of the Treasury: Department of The Treasury: Washington, D.C. 20220: November 7, 2013: Mr. David A. Powner: Director: Information Technology Management Issues: U.S. Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Dear Mr. Powner, Thank you for the opportunity to provide comments on GAO's Draft Report, "IT Dashboard: Agencies are Managing Investment Risk but Related Ratings Need to Be More Accurate and Available (GAO-I4-64)." The Department of the Treasury has no comments on the Report and appreciates GAO's efforts in its development. Please contact me at 202-622-1200 if you need anything further. Sincerely, Signed by: Robyn East: Deputy Assistant Secretary for Information Systems and Chief Information Officer: [End of section] Appendix VIII: Comments from the Department of Veterans Affairs: Department of Veterans Affairs: Washington DC 20420 November 8, 2013 Mr. David A. Powner: Director, Information Technology Management Issues: U.S. Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Dear Mr. Powner: The Department of Veterans Affairs (VA) has reviewed the Government Accountability Office's (GAO) draft report, "IT Dashboard: Agencies are Managing Investment Risk, but Related Ratings Need to Be More Accurate and Available" (GAO-14-64). VA generally agrees with GAO's findings. The enclosure contains general comments related to the draft report. VA appreciates the opportunity to comment on your draft report. Sincerely, Signed by: Jose D. Riojas: Chief of Staff: Enclosure: Department of Veterans Affairs (VA) Response to Government Accountability Office (GAO) Draft Report "IT Dashboard: Agencies are Managing Investment Risk, but Related Ratings Need to Be More Accurate and Available" (GAO-14-64): General Comment: As stated in the GAO draft report, for a period of time, VA did not update the Chief Information Officer (CIO) rating on the Federal Information Technology (IT) Dashboard and the deficiency has been corrected. The draft report questions VA's ability to automatically submit CIO ratings and also the data used to determine the CIO rating. The CIO rating is a subjective rating at the investment level and the investments contain multiple Project Management Accountability System (PMAS) projects. The IT investments consist of multiple projects with a wide-variety of complexity and costs and, as a result, the ability to assess performance at the investment level is secondary and typically a by-product of individual project performance. CIO ratings, made at the investment level, were based on percentage of projects and did not initially consider the scope, size, or complexity of projects and as such our documentation did not match the overall investment rating in a particular month. To put this in perspective, the Benefits 21st Century Education investment showed a 40 percent (2 out of 5 projects) schedule variance; unfortunately, the 2 projects were Education Application Development — WSMS and Veterans Retraining Assistance Program, which are very small compared to Chapter 33 Automate GI Bill projects. These two projects represented only 1.1 percent ($1.0343M of $94.2026M) of the total planned cost of projects within this investment. Automatic submission of the CIO rating: When VA purchased the eCPIC tool to use for 300A submissions, it did not also purchase the module required to make CIO rating submissions. Within eCPIC, the functionality to update the CIO rating is contained within the 300B module and because the determination had been made that the 300B would be populated from the existing PMAS Dashboard, tie cost of acquiring the 300B module for the sole purpose of transmitting the CIO rating was not cost effective. For a period of time VA then had no automated tool by which to make CIO rating submissions. During that time however, VA did submit CI 3 ratings manually as required by the Office of Management and Budget (OMB) schema when submitting the annual budget request. As the PMAS Dashboard was being developed, it was recognized that this tool could be used to make automated CIO rating submissions, as it is the primary management tool for monitoring IT development projects within VA, and basic PMAS Dashboard project data is also transmitted to the Federal IT Dashboard. With slight modifications it could also be used as the foundation for automating the CIO rating. This capability was implemented this year. Data used to determine the CIO rating: Initially, different PMAS data was used to contribute to the CIO rating. Cost and schedule variance data was used to generate the recommended CIO rating, rather than PMAS specific data like Red Flags thrown and TechStats held. Once the capability to submit the CIO rating was implemented from within the PMAS Dashboard, PMAS specific data like Red Flags and TechStats could be used. The PMAS data, which is used as an input to the CIO rating, has been corrected to reflect VA's successful methodology to manage at the increment or lowest level of delivery. VA will continue its efforts to improve timely and accurate reporting to the Federal IT Dashboard. It is, however, important to recognize that VA manages investment delivery at the micro level in comparison to the macro level in the OMB CIO Dashboard. [End of section] Appendix IX: Comments from the Social Security Administration: Social Security: Office of the Commissioner: Social Security Administration: Baltimore, MD 21235-0001: Mr. Dan Bertoni, Director: Education, Workforce, and Income Security Issues: United States Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Dear Mr. Bertoni: Thank you for the opportunity to review the draft report, "Information Technology Dashboard: Agencies are Managing Investment Risk, but Related Ratings Need to Be More Accurate and Available" (GAO-14-64). Our response to the audit report recommendation is enclosed. If you have any questions, please contact me at (410) 966-9014. Your staff may contact Gary S. Hatcher, Senior Advisor for Records Management and Audit Liaison Staff, at (410) 965-0680. Sincerely, Signed by: Katherine Thornton: Deputy Chief of Staff: Enclosure: Comments On The Government Accountability Office Draft Report, "Information Technology Dashboard: Agencies Are Managing Investment Risk, But Related Ratings Need To Be More Accurate And Available" (GAO- 14-64): Recommendation 1: Revise the Agency's investment management processes to ensure that they are consistent with the baselining practices identified in GAO's published guidance on cost estimating and assessment. Response: We agree. We will limit our re-baselining practices to situations where it is apparent that the previous baseline no longer provides a meaningful basis for performance management. We will continue to review and update our baselining policy and procedures as needed. [End of section] Appendix X: GAO Contact and Staff Acknowledgments: GAO Contact: David A. Powner, (202) 512-9286 or pownerd@gao.gov: Staff Acknowledgments: In addition to the contact named above, individuals making contributions to this report included Dave Hinchman (Assistant Director), Rebecca Eyler, Mike Mithani, Kevin Walsh, and Shawn Ward. [End of section] Footnotes: [1] For example, the Paperwork Reduction Act (44 U.S.C. § 3501 et seq.) specifies OMB and agency responsibilities for managing IT and the Clinger-Cohen Act of 1996 (40 U.S.C. § 11302(c)) charges OMB with responsibility for analyzing, tracking, and evaluating the risks and results of all major IT investments as part of the federal budget process, and reporting to Congress on the performance benefits achieved by these investments. [2] "Major IT investment" means a system or an acquisition requiring special management attention because it has significant importance to the mission or function of the government; significant program or policy implications; high executive visibility; high development, operating, or maintenance costs; an unusual funding mechanism; or is defined as major by the agency's capital planning and investment control process. [3] The eight selected agencies are the Departments of Agriculture (Agriculture), Commerce (Commerce), Energy (Energy), Justice (Justice), Transportation (Transportation), the Treasury (Treasury), and Veterans Affairs (VA), as well as the Social Security Administration (SSA). See appendix II for a list of the selected investments. [4] GAO, Information Technology Dashboard: Opportunities Exist to Improve Transparency and Oversight of Investment Risk at Select Agencies, [hyperlink, http://www.gao.gov/products/GAO-13-98] (Washington, D.C.: Oct. 16, 2012). [5] GAO, Information Technology: OMB and Agencies Need to More Effectively Implement Major Initiatives to Save Billions of Dollars, [hyperlink, http://www.gao.gov/products/GAO-13-796T] (Washington, D.C.: July 25, 2013). [6] The 27 federal agencies are the Departments of Agriculture, Commerce, Defense, Education, Energy, Health and Human Services, Homeland Security, Housing and Urban Development, Justice, Labor, State, the Interior, the Treasury, Transportation, Veterans Affairs; the Environmental Protection Agency; the General Services Administration; the National Aeronautics and Space Administration; National Archives and Records Administration; National Science Foundation; Nuclear Regulatory Commission; Office of Personnel Management; Small Business Administration; Smithsonian Institution; Social Security Administration; U.S. Agency for International Development; and the U.S. Army Corps of Engineers. [7] [hyperlink, http://www.itdashboard.gov]. [8] Each agency's budget submissions include an exhibit 53 and an exhibit 300 for each major IT investment. The exhibit 53 lists all of an agency's IT projects and associated costs. Exhibit 300s (also called the Capital Asset Plan and Business Case) are prepared for every major IT investment, are used to justify resource requests, and are intended to enable an agency to demonstrate to its own management, as well as to OMB, that a major project is well-planned. [9] Chief Information Officer Authorities, Memorandum for Heads of Executive Departments and Agencies, M-11-29 (Washington, D.C.: Aug. 8, 2011). [10] GAO, Information Technology: OMB's Dashboard Has Increased Transparency and Oversight, but Improvements Needed, [hyperlink, http://www.gao.gov/products/GAO-10-701] (Washington, D.C.: July 16, 2010). [11] GAO, Information Technology: OMB Has Made Improvements to Its Dashboard, but Further Work Is Needed by Agencies and OMB to Ensure Data Accuracy, [hyperlink, http://www.gao.gov/products/GAO-11-262] (Washington, D.C.: Mar. 15, 2011). [12] GAO, Information Technology: OMB Needs to Improve Its Guidance on IT Investments, [hyperlink, http://www.gao.gov/products/GAO-11-826] (Washington, D.C.: Sept. 29, 2011). [13] GAO, IT Dashboard: Accuracy Has Improved, and Additional Efforts Are Under Way to Better Inform Decision Making, [hyperlink, http://www.gao.gov/products/GAO-12-210] (Washington, D.C.: Nov. 7, 2011). [14] [hyperlink, http://www.gao.gov/products/GAO-13-98]. [15] GAO, Information Technology: Additional Executive Review Sessions Needed to Address Troubled Projects, [hyperlink, http://www.gao.gov/products/GAO-13-524] (Washington, D.C.: June 13, 2013). [16] TechStat sessions are face-to-face meetings to terminate, halt, or turnaround IT investments that are failing or are not producing results. [17] U.S. Chief Information Officer, 25 Point Implementation Plan to Reform Federal Information Technology Management, The White House (Washington, D.C.: Dec. 9, 2010). [18] As mentioned earlier, CIOs are responsible for updating these rating as soon as new information becomes available which will affect an investment's assessment. [19] This number varies from the 244 described earlier because it takes into account investments that the selected agencies are no longer reporting on the Dashboard. The following section provides further details. [20] CIO's assign a rating of 1 (high risk) to 5 (low risk) based on the level of risk facing the investment. As such, a reduction in risk level is indicated by a higher CIO rating and an increase in risk level is indicated by a lower CIO rating. [21] [hyperlink, http://www.gao.gov/products/GAO-11-826]. [22] [hyperlink, http://www.gao.gov/products/GAO-11-826]. [23] Clinger-Cohen Act of 1996 (40 U.S.C. § 11101(6)). [24] Clinger-Cohen Act of 1996 (40 U.S.C. § 11302(c)). [25] [hyperlink, http://www.itdashboard.gov/faq]. [26] VA tracks missed increment deliverable dates by recording a "strike" against a project. VA may pause or close projects that receive three strikes. [27] VA, Audit of the Project Management Accountability System Implementation, 10-03162-262 (Aug. 29, 2011). [28] SSA, Information Resources Management Strategic Plan: FY 2012- 2016 (May 2012). [29] At times, a project's cost, schedule, and performance goals-- known as its baseline--are modified to reflect changed development circumstances. These changes--called a rebaseline--can be done for valid reasons, but can also be used to mask cost overruns and schedule delays. [30] GAO, Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs, [hyperlink, http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: Mar. 2, 2009). [31] GAO, Information Technology Investment Management: A Framework for Assessing and Improving Process Maturity, [hyperlink, http://www.gao.gov/products/GAO-04-394G] (Washington, D.C.: March 2004). [32] GAO, Managing for Results: Data-Driven Performance Reviews Show Promise But Agencies Should Explore How to Involve Other Relevant Agencies, [hyperlink, http://www.gao.gov/products/GAO-13-228] (Washington, D.C.: Feb. 27, 2013). [33] A TechStat is a face-to-face, evidence-based accountability review of an IT investment that enables the federal government to intervene to turnaround, halt, or terminate projects that are failing or are not producing results. OMB began leading TechStat sessions on agency IT projects in 2010, and subsequently required federal agencies and bureaus to hold them. [34] GAO, Information Technology Dashboard: Opportunities Exist to Improve Transparency and Oversight of Investment Risk at Select Agencies, [hyperlink, http://www.gao.gov/products/GAO-13-98] (Washington, D.C.: Oct. 16, 2012). [35] We did not independently evaluate the ratings of every investment reported by agencies. However, we determined that they were sufficiently reliable for the purposes of our objectives by confirming with each agency that the ratings that we downloaded from the IT Dashboard were complete, accurate, and reflected the data they had reported to OMB. [36] Clinger-Cohen Act of 1996 (40 U.S.C. § 11101(6)). [37] Because we were only evaluating the consistency of Dashboard ratings with reported risk, we did not evaluate the accuracy of the investment documentation. [38] We elected to review the investments' ratings during calendar year 2012 so that our assessment would span multiple fiscal years. [End of section] GAO’s Mission: The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to [hyperlink, http://www.gao.gov] and select “E-mail Updates.” Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO: Connect with GAO on facebook, flickr, twitter, and YouTube. Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts. Visit GAO on the web at [hyperlink, http://www.gao.gov]. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; E-mail: fraudnet@gao.gov; Automated answering system: (800) 424-5454 or (202) 512-7470. Congressional Relations: Katherine Siggerud, Managing Director, siggerudk@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, DC 20548. Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, DC 20548. [End of document]