IT Dashboard:

Agencies Need to Fully Consider Risks When Rating Their Major Investments

GAO-16-494: Published: Jun 2, 2016. Publicly Released: Jun 2, 2016.

Additional Materials:

Contact:

David A. Powner
(202) 512-9286
pownerd@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

What GAO Found

Agencies determined investments' Chief Information Officer (CIO) ratings using a variety of processes, which included the Office of Management and Budget's (OMB) six suggested factors (including risk management, requirements management, and historical performance). Specifically, all 17 selected agencies incorporated at least two of OMB's factors into their risk rating processes and 9 used all of the factors. However, agencies' interpretations of these factors varied. For example, most agencies considered active risks, such as funding cuts or staffing changes, when rating investments, but others only evaluated compliance with the agency's risk management processes. Further, 13 agencies required monthly updates to CIO ratings as does OMB (as of June 2015), 1 agency scheduled its reviews based on risk, and 3 agencies required updates less often than on a monthly basis.

GAO's assessments generally showed more risk than the associated CIO ratings. In particular, of the 95 investments assessed, GAO's assessments matched the CIO ratings 22 times, showed more risk 60 times, and showed less risk 13 times (see graphic).

Comparison of Selected Investments' Chief Information Officer Ratings to GAO Assessments

Comparison of Selected Investments' Chief Information Officer Ratings to GAO Assessments img  data-cke-saved-src=

Aside from the inherent judgmental nature of risk ratings, three issues contributed to these differences:

Forty of the 95 CIO ratings were not updated during the month GAO reviewed, which led to more differences between GAO's assessments and the CIOs' ratings. This underscores the importance of frequent rating updates, which help to ensure that the information on the Dashboard is timely and accurately reflects recent changes to investment status.

Three agencies' rating processes span longer than 1 month. Longer processes mean that CIO ratings are based upon older data and may not reflect the current level of investment risk.

Seven agencies' rating processes did not focus on active risks. According to OMB's guidance, CIO ratings should reflect the CIO's assessment of the risk and the investment's ability to accomplish its goals. CIO ratings that do not incorporate active risks increase the chance that ratings overstate the likelihood of investment success.

 

Why GAO Did This Study

Although the government spends more than $80 billion in information technology (IT) annually, many of the investments have failed or have been troubled. In December 2014, provisions commonly referred to as the Federal Information Technology Acquisition Reform Act (FITARA) were enacted. Among other things, FITARA states that OMB shall make available to the public a list of each major IT investment including data on cost, schedule, and performance. OMB does so via the Federal IT Dashboard—its public website that reports on major IT investments, including ratings from CIOs which should reflect the level of risk facing an investment.

GAO's objectives were to (1) describe agencies' processes for determining CIO risk ratings for major federal IT investments primarily in development and (2) assess the risk of federal IT investments and analyze any differences with the investments' CIO risk ratings. To do so, GAO selected major IT investments with at least 80 percent of their fiscal year 2015 budget allocated to development (resulting in 95 investments across 15 agencies) and compared CIO rating processes to OMB guidance. GAO also analyzed data on those investments to create its own risk assessments.

What GAO Recommends

GAO is making 25 recommendations to 15 agencies to improve the quality and frequency of CIO ratings. Twelve agencies generally agreed with or did not comment on the recommendations and three agencies disagreed, stating their CIO ratings were adequate. GAO continues to believe these recommendations are valid.

For more information, contact David A. Powner at (202) 512-9286 or pownerd@gao.gov.

Recommendations for Executive Action

  1. Status: Open

    Comments: When we confirm what actions have been taken, we will update the recommendation status.

    Recommendation: To better ensure that the Dashboard ratings more accurately reflect risk, the Secretaries of the Departments of Agriculture, Education, Energy, Health and Human Services, the Interior, State, and Veterans Affairs; and the Director of the Office of Personnel Management should direct their CIOs to factor active risks into their IT Dashboard CIO ratings.

    Agency Affected: Department of Agriculture

  2. Status: Open

    Comments: The Department agreed with the recommendation, but has not provided an update on its actions to address it. When we confirm what actions have been taken, we will update.

    Recommendation: To better ensure that the Dashboard ratings more accurately reflect risk, the Secretaries of the Departments of Agriculture, Education, Energy, Health and Human Services, the Interior, State, and Veterans Affairs; and the Director of the Office of Personnel Management should direct their CIOs to factor active risks into their IT Dashboard CIO ratings.

    Agency Affected: Department of Education

  3. Status: Open

    Comments: The Department agreed with the recommendation and, in a written response, stated that the Office of the CIO will update the CIO's OMB IT Dashboard Standard Operating Procedure to include the evaluation and assessment of active risks. This effort is to be completed by the end of December 2016. We will continue to monitor the implementation of this recommendation.

    Recommendation: To better ensure that the Dashboard ratings more accurately reflect risk, the Secretaries of the Departments of Agriculture, Education, Energy, Health and Human Services, the Interior, State, and Veterans Affairs; and the Director of the Office of Personnel Management should direct their CIOs to factor active risks into their IT Dashboard CIO ratings.

    Agency Affected: Department of Energy

  4. Status: Open

    Comments: The Department agreed with the recommendation and, in a written response, stated that it updated its CIO evaluation methodology to measure active risks in areas such as budget variance, performance, policy and governance compliance, risk management, and contract risk. When we confirm what actions have been taken, we will update.

    Recommendation: To better ensure that the Dashboard ratings more accurately reflect risk, the Secretaries of the Departments of Agriculture, Education, Energy, Health and Human Services, the Interior, State, and Veterans Affairs; and the Director of the Office of Personnel Management should direct their CIOs to factor active risks into their IT Dashboard CIO ratings.

    Agency Affected: Department of Health and Human Services

  5. Status: Open

    Comments: The Department agreed with this recommendation and, in a written response, stated that it plans to address this recommendation with the following actions: (1) developing a method to review and assign ratings for active risks that will be incorporated into CIO ratings and (2) integrating the risk rating methodology into a new process for all major investments' CIO ratings. We will continue to monitor the implementation of this recommendation.

    Recommendation: To better ensure that the Dashboard ratings more accurately reflect risk, the Secretaries of the Departments of Agriculture, Education, Energy, Health and Human Services, the Interior, State, and Veterans Affairs; and the Director of the Office of Personnel Management should direct their CIOs to factor active risks into their IT Dashboard CIO ratings.

    Agency Affected: Department of the Interior

  6. Status: Open

    Comments: The Department agreed with the recommendation and, in a written response, stated that it is amending its current monthly review process to ensure that risks are factored into its IT Dashboard CIO ratings. VA expects to complete this effort during the first quarter of 2017. We will continue to monitor the implementation of this recommendation.

    Recommendation: To better ensure that the Dashboard ratings more accurately reflect risk, the Secretaries of the Departments of Agriculture, Education, Energy, Health and Human Services, the Interior, State, and Veterans Affairs; and the Director of the Office of Personnel Management should direct their CIOs to factor active risks into their IT Dashboard CIO ratings.

    Agency Affected: Department of Veterans Affairs

  7. Status: Open

    Comments: The Department agreed with the recommendation, but has not provided an update on its actions to address the recommendation. When we confirm what actions have been taken, we will update.

    Recommendation: To better ensure that the Dashboard ratings more accurately reflect risk, the Secretaries of the Departments of Agriculture, Education, Energy, Health and Human Services, the Interior, State, and Veterans Affairs; and the Director of the Office of Personnel Management should direct their CIOs to factor active risks into their IT Dashboard CIO ratings.

    Agency Affected: Department of State

  8. Status: Open

    Comments: When we confirm what actions have been taken, we will update the recommendation status.

    Recommendation: To better ensure that the Dashboard ratings more accurately reflect risk, the Secretaries of the Departments of Agriculture, Education, Energy, Health and Human Services, the Interior, State, and Veterans Affairs; and the Director of the Office of Personnel Management should direct their CIOs to factor active risks into their IT Dashboard CIO ratings.

    Agency Affected: Office of Personnel Management

  9. Status: Closed - Implemented

    Comments: The Department of Homeland Security (DHS) disagreed with this recommendation. In its written responses, the department stated that its risk-based process complied with OMB's "Fiscal Year 2017 IT Budget-Capital Planning Guidance" with regard to the frequency of its CIO rating updates. However, as noted in the report, we maintained that the guidance required at least monthly updates, and DHS rated its investments either monthly, quarterly, or semi-annually, depending on investment risk. DHS also noted that its process would be supported by the release of subsequent OMB guidance. After the publication of our report in June 2016, OMB issued its "Fiscal Year 2018 IT Budget-Capital Planning Guidance." This guidance removed the mandatory reporting frequency, but stated that OMB expected that the CIOs would evaluate and rate their investments at specific times, including when the investment business cases were submitted to OMB in the agency budget request and when the business cases were prepared for the President's Budget release. In light of this new guidance, we analyzed the department's update frequency for its 97 investments (as listed on the IT Dashboard in June 2017). From June 2016 through May 2017, we found that 82 investments had updates posted during at least two separate months. For the other 15 investments, 11 were updated during one month, and 4 investments were not updated at all. This analysis shows that, for the majority of its investments, DHS is meeting OMB's expectations for at least semi-annual updates. By meeting these expectations, the department will help ensure that the information on the Dashboard is timely and accurately reflects recent changes.

    Recommendation: To better ensure that the Dashboard ratings more accurately reflect risk, the Secretaries of the Departments of Defense, Education, and Homeland Security; and the Commissioner of the Social Security Administration should direct their CIOs to update their CIO ratings at least as frequently as required in OMB's guidance.

    Agency Affected: Department of Homeland Security

  10. Status: Closed - Implemented

    Comments: The Department of Education (Education) initially partially concurred with the recommendation. In its comments published in the report, Education stated that OMB's "Fiscal Year 2017 IT Budget-Capital Planning Guidance" addressed the required frequency of updates in several places and that the section specific to CIO evaluations only required agencies to update their ratings as soon as new information became available. In response, we maintained that the requirement for monthly updates was explicitly stated in the guidance and was confirmed by OMB staff. In comments from July 2016, Education stated that it agreed with the recommendation to update its CIO ratings at least as frequently as required in OMB's guidance. It also noted that its process complied with current OMB guidance. Indeed, OMB issued its "Fiscal Year 2018 IT Budget-Capital Planning Guidance" after our report was issued in June 2016. This guidance removed the mandatory reporting frequency, but stated that OMB expected that the CIOs would evaluate and rate their investments at specific times, including when the investment business cases were submitted to OMB in the agency budget request and when the business cases were prepared for the President's Budget release. In light of this new guidance, we analyzed Education's update frequency for its 30 investments (as listed on the IT Dashboard in June 2017). From June 2016 through May 2017, we found that 29 investments had updates posted during at least two separate months. The remaining investment was updated during one month. This analysis shows that, for the majority of its investments, Education is meeting OMB's expectations for at least semi-annual updates. By meeting these expectations, the Department will help ensure that the information on the Dashboard is timely and accurately reflects recent changes.

    Recommendation: To better ensure that the Dashboard ratings more accurately reflect risk, the Secretaries of the Departments of Defense, Education, and Homeland Security; and the Commissioner of the Social Security Administration should direct their CIOs to update their CIO ratings at least as frequently as required in OMB's guidance.

    Agency Affected: Department of Education

  11. Status: Open

    Comments: The Department of Defense (DOD) disagreed with this recommendation. In its written response, the Department noted that its semi-annual reporting is consistent with FITARA requirements and is documented in its OMB-approved FITARA Implementation Plan. After the publication of our report in June 2016, OMB issued its "Fiscal Year 2018 IT Budget-Capital Planning Guidance." This guidance removes the mandatory reporting frequency, but states that OMB expects that the CIOs would evaluate and rate their investments at specific times, including when the investment business cases are submitted to OMB in the agency budget request and when the business cases are prepared for the President's Budget release. In light of this new guidance, we analyzed the Department's update frequency for its 34 major investments (as listed on the IT Dashboard in June 2017). From June 2016 through May 2017, we found that 26 of the investments' ratings were updated once: in May 2017. The other 8 investments were not updated during this timeframe. Prior to this, the last DOD rating updates were made in March 2016, over a year beforehand. This analysis shows that DOD is not adhering to either its own semi-annual reporting requirements or to OMB's expectations. As such, we are not closing the recommendation at this time. We will continue to monitor the IT Dashboard for changes to DOD's update frequency. We maintain that frequent rating updates help ensure that the information on the Dashboard is timely and accurately reflects recent changes. Without such updates, the CIO ratings on the IT Dashboard may not reflect the current level of investment risk.

    Recommendation: To better ensure that the Dashboard ratings more accurately reflect risk, the Secretaries of the Departments of Defense, Education, and Homeland Security; and the Commissioner of the Social Security Administration should direct their CIOs to update their CIO ratings at least as frequently as required in OMB's guidance.

    Agency Affected: Department of Defense

  12. Status: Closed - Implemented

    Comments: The Social Security Administration (SSA) agreed with the recommendation. In its written comments, SSA stated that its investment evaluations now occur on a more frequent basis, consistent with OMB's "Fiscal Year 2018 IT Budget-Capital Planning Guidance." OMB released this new version of their Capital Planning Guidance following the publication of our report in June 2016. This guidance removed the mandatory reporting frequency, but stated that OMB expected that the CIOs would evaluate and rate their investments at specific times, including when the investment business cases were submitted to OMB in the agency budget request and when the business cases were prepared for the President's Budget release. In light of this new guidance, we analyzed SSA's update frequency for its 11 investments (as listed on the IT Dashboard in June 2017). From June 2016 through May 2017, we found that all 11 investments had CIO rating updates posted during at least two separate months. This analysis shows that SSA is meeting OMB's expectations for at least semi-annual updates. By meeting these expectations, SSA will help ensure that the information on the Dashboard is timely and accurately reflects recent changes.

    Recommendation: To better ensure that the Dashboard ratings more accurately reflect risk, the Secretaries of the Departments of Defense, Education, and Homeland Security; and the Commissioner of the Social Security Administration should direct their CIOs to update their CIO ratings at least as frequently as required in OMB's guidance.

    Agency Affected: Social Security Administration

  13. Status: Open

    Comments: The Department agreed with the recommendation and, in a written response, stated that the Office of the CIO Enterprise Business Management Office is updating its program assessment guideline. The updated guideline will include risk-based scores as the basis for its investment ratings. The Department expects to release this new guideline by the end of December 2016. We will continue to monitor the implementation of this recommendation.

    Recommendation: To better ensure that the Dashboard ratings more accurately reflect risk, the Secretaries of the Departments of Agriculture, Commerce, Defense, Education, Energy, Health and Human Services, Homeland Security, State, Transportation, the Treasury, Veterans Affairs; the Administrator of the Environmental Protection Agency; and the Commissioner of the Social Security Administration should direct their CIOs to ensure that their CIO ratings reflect the level of risk facing an investment relative to that investment's ability to accomplish its goals.

    Agency Affected: Department of Homeland Security

  14. Status: Open

    Comments: When we confirm what actions have been taken, we will update the recommendation status.

    Recommendation: To better ensure that the Dashboard ratings more accurately reflect risk, the Secretaries of the Departments of Agriculture, Commerce, Defense, Education, Energy, Health and Human Services, Homeland Security, State, Transportation, the Treasury, Veterans Affairs; the Administrator of the Environmental Protection Agency; and the Commissioner of the Social Security Administration should direct their CIOs to ensure that their CIO ratings reflect the level of risk facing an investment relative to that investment's ability to accomplish its goals.

    Agency Affected: Department of Agriculture

  15. Status: Open

    Comments: The Department agreed with the recommendation, but has not provided an update on its actions to address it. When we confirm what actions have been taken, we will update.

    Recommendation: To better ensure that the Dashboard ratings more accurately reflect risk, the Secretaries of the Departments of Agriculture, Commerce, Defense, Education, Energy, Health and Human Services, Homeland Security, State, Transportation, the Treasury, Veterans Affairs; the Administrator of the Environmental Protection Agency; and the Commissioner of the Social Security Administration should direct their CIOs to ensure that their CIO ratings reflect the level of risk facing an investment relative to that investment's ability to accomplish its goals.

    Agency Affected: Department of Education

  16. Status: Open

    Comments: The Department agreed with our recommendation and, in a written response, stated that the CIO has revised the IT Dashboard assessment criteria to directly incorporate the degree of risk represented in the investments' Business Case documents. We will continue to monitor the implementation of this recommendation.

    Recommendation: To better ensure that the Dashboard ratings more accurately reflect risk, the Secretaries of the Departments of Agriculture, Commerce, Defense, Education, Energy, Health and Human Services, Homeland Security, State, Transportation, the Treasury, Veterans Affairs; the Administrator of the Environmental Protection Agency; and the Commissioner of the Social Security Administration should direct their CIOs to ensure that their CIO ratings reflect the level of risk facing an investment relative to that investment's ability to accomplish its goals.

    Agency Affected: Department of Commerce

  17. Status: Open

    Comments: When we confirm what actions have been taken, we will update the recommendation status.

    Recommendation: To better ensure that the Dashboard ratings more accurately reflect risk, the Secretaries of the Departments of Agriculture, Commerce, Defense, Education, Energy, Health and Human Services, Homeland Security, State, Transportation, the Treasury, Veterans Affairs; the Administrator of the Environmental Protection Agency; and the Commissioner of the Social Security Administration should direct their CIOs to ensure that their CIO ratings reflect the level of risk facing an investment relative to that investment's ability to accomplish its goals.

    Agency Affected: Department of Defense

  18. Status: Open

    Comments: The Department agreed with the recommendation and, in a written response, stated that the Office of the CIO will update its IT Dashboard Standard Operating Procedure to include an active risk sub-criteria comprised of probability and impact scores. This effort is to be completed by the end of December 2016. We will continue to monitor the implementation of this recommendation.

    Recommendation: To better ensure that the Dashboard ratings more accurately reflect risk, the Secretaries of the Departments of Agriculture, Commerce, Defense, Education, Energy, Health and Human Services, Homeland Security, State, Transportation, the Treasury, Veterans Affairs; the Administrator of the Environmental Protection Agency; and the Commissioner of the Social Security Administration should direct their CIOs to ensure that their CIO ratings reflect the level of risk facing an investment relative to that investment's ability to accomplish its goals.

    Agency Affected: Department of Energy

  19. Status: Open

    Comments: The Department agreed with the recommendation and, in a written response, stated that it updated its CIO evaluation methodology to measure active risks in areas such as budget variance, performance, policy and governance compliance, risk management, and contract risk. According to HHS, these risk areas reflect both internal and external risks that affect an investment's ability to accomplish its goals. When we confirm what actions have been taken, we will update.

    Recommendation: To better ensure that the Dashboard ratings more accurately reflect risk, the Secretaries of the Departments of Agriculture, Commerce, Defense, Education, Energy, Health and Human Services, Homeland Security, State, Transportation, the Treasury, Veterans Affairs; the Administrator of the Environmental Protection Agency; and the Commissioner of the Social Security Administration should direct their CIOs to ensure that their CIO ratings reflect the level of risk facing an investment relative to that investment's ability to accomplish its goals.

    Agency Affected: Department of Health and Human Services

  20. Status: Open

    Comments: The agency partially agreed with our recommendation and, in a written response, stated that its CIO rating criteria includes a review of the level of risk facing an investment relative to that investment's ability to accomplish its goals. The written statement also notes that the CIO receives regular updates from key stakeholders on investment risks and mitigation plans. When we confirm what actions have been taken, we will update.

    Recommendation: To better ensure that the Dashboard ratings more accurately reflect risk, the Secretaries of the Departments of Agriculture, Commerce, Defense, Education, Energy, Health and Human Services, Homeland Security, State, Transportation, the Treasury, Veterans Affairs; the Administrator of the Environmental Protection Agency; and the Commissioner of the Social Security Administration should direct their CIOs to ensure that their CIO ratings reflect the level of risk facing an investment relative to that investment's ability to accomplish its goals.

    Agency Affected: Social Security Administration

  21. Status: Open

    Comments: When we confirm what actions have been taken, we will update the recommendation status.

    Recommendation: To better ensure that the Dashboard ratings more accurately reflect risk, the Secretaries of the Departments of Agriculture, Commerce, Defense, Education, Energy, Health and Human Services, Homeland Security, State, Transportation, the Treasury, Veterans Affairs; the Administrator of the Environmental Protection Agency; and the Commissioner of the Social Security Administration should direct their CIOs to ensure that their CIO ratings reflect the level of risk facing an investment relative to that investment's ability to accomplish its goals.

    Agency Affected: Department of Transportation

  22. Status: Open

    Comments: When we confirm what actions have been taken, we will update the recommendation status.

    Recommendation: To better ensure that the Dashboard ratings more accurately reflect risk, the Secretaries of the Departments of Agriculture, Commerce, Defense, Education, Energy, Health and Human Services, Homeland Security, State, Transportation, the Treasury, Veterans Affairs; the Administrator of the Environmental Protection Agency; and the Commissioner of the Social Security Administration should direct their CIOs to ensure that their CIO ratings reflect the level of risk facing an investment relative to that investment's ability to accomplish its goals.

    Agency Affected: Department of the Treasury

  23. Status: Open

    Comments: The Department agreed with the recommendation and, in a written response, stated that it plans to require investment managers to assess operational risks detailing the probability and impact of pending threats to success. VA expects to complete this effort during the first quarter of 2017. We will continue to monitor the implementation of this recommendation.

    Recommendation: To better ensure that the Dashboard ratings more accurately reflect risk, the Secretaries of the Departments of Agriculture, Commerce, Defense, Education, Energy, Health and Human Services, Homeland Security, State, Transportation, the Treasury, Veterans Affairs; the Administrator of the Environmental Protection Agency; and the Commissioner of the Social Security Administration should direct their CIOs to ensure that their CIO ratings reflect the level of risk facing an investment relative to that investment's ability to accomplish its goals.

    Agency Affected: Department of Veterans Affairs

  24. Status: Open

    Comments: The Department agreed with the recommendation, but has not provided an update on its actions to address the recommendation. When we confirm what actions have been taken, we will update.

    Recommendation: To better ensure that the Dashboard ratings more accurately reflect risk, the Secretaries of the Departments of Agriculture, Commerce, Defense, Education, Energy, Health and Human Services, Homeland Security, State, Transportation, the Treasury, Veterans Affairs; the Administrator of the Environmental Protection Agency; and the Commissioner of the Social Security Administration should direct their CIOs to ensure that their CIO ratings reflect the level of risk facing an investment relative to that investment's ability to accomplish its goals.

    Agency Affected: Department of State

  25. Status: Open

    Comments: The agency disagreed with the recommendation and has not provided an update on its actions to address the recommendation. We will continue to monitor the implementation of this recommendation.

    Recommendation: To better ensure that the Dashboard ratings more accurately reflect risk, the Secretaries of the Departments of Agriculture, Commerce, Defense, Education, Energy, Health and Human Services, Homeland Security, State, Transportation, the Treasury, Veterans Affairs; the Administrator of the Environmental Protection Agency; and the Commissioner of the Social Security Administration should direct their CIOs to ensure that their CIO ratings reflect the level of risk facing an investment relative to that investment's ability to accomplish its goals.

    Agency Affected: Environmental Protection Agency

 

Explore the full database of GAO's Open Recommendations »

Oct 4, 2017

Sep 18, 2017

Sep 6, 2017

Jul 13, 2017

Jun 21, 2017

Jun 13, 2017

May 18, 2017

May 15, 2017

Apr 11, 2017

Looking for more? Browse all our products here