Foreign Assistance:

Agencies Can Improve the Quality and Dissemination of Program Evaluations

GAO-17-316: Published: Mar 3, 2017. Publicly Released: Mar 3, 2017.

Additional Materials:

Contact:

Jessica Farb
(202) 512-6991
farbj@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

The U.S. government plans to spend about $35 billion on foreign assistance programs in 2017, and program evaluations can help assess and improve the results of this spending.

However, our review of the six agencies providing the most on foreign aid (shown below) found that about a quarter of their program evaluations in 2015 lacked adequate information on results to inform future programs.

We recommended that each agency develop a plan to improve the quality of its evaluations. We also recommended that some agencies improve their procedures to disseminate their evaluation reports—which may help future programs benefit from lessons learned.

Estimated Percentages of Agency Evaluations Generally or Partially Meeting Applicable Quality Criteria or Not Meeting One or More Criteria

Summary Table of Foreign Assistance Evaluation Quality by Agency

Summary Table of Foreign Assistance Evaluation Quality by Agency

Additional Materials:

Contact:

Jessica Farb
(202) 512-6991
farbj@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

What GAO Found

An estimated 73 percent of evaluations completed in fiscal year 2015 by the six U.S. agencies GAO reviewed generally or partially addressed all of the quality criteria GAO identified for evaluation design, implementation, and conclusions (see fig.). Agencies met some elements of the criteria more often than others. For example, approximately 90 percent of all evaluations addressed questions that are generally aligned with program goals and were thus able to provide useful information about program results. About 40 percent of evaluations did not use generally appropriate sampling, data collection, or analysis methods. Although implementing evaluations overseas poses significant methodological challenges, GAO identified opportunities for each agency to improve evaluation quality and thereby strengthen its ability to manage aid funds more effectively based on results.

Estimated Percentage of Foreign Assistance Evaluations Meeting Evaluation Quality Criteria

Estimated Percentage of Foreign Assistance Evaluations Meeting Evaluation Quality Criteria

Note: The confidence intervals for our estimates of the quality of agency evaluations according to these categories did not exceed ±8 percent.

Evaluation costs ranged widely and were sometimes difficult to determine, but the majority of evaluations GAO examined cost less than $200,000. Millennium Challenge Corporation (MCC) evaluations had a median cost of about $269,000, while median costs for the U.S. Agency for International Development (USAID), the U.S. Department of Agriculture (USDA), and the Department of State (State) ranged from about $88,000 to about $178,000. GAO was unable to identify the specific costs for the Department of Defense (DOD) and Department of Health and Human Services (HHS) evaluations. High-quality evaluations tend to be more costly, but some well-designed lower-cost evaluations also met all quality criteria. Other factors related to evaluation costs include the evaluation's choice of methodology, its duration, and its location.

Agencies generally posted and distributed evaluations for the use of internal and external stakeholders. However, shortfalls in some agency efforts may limit the evaluations' usefulness.

  • Public posting. USDA has not developed procedures for reviewing and preparing its evaluations for public posting, but the other agencies posted nonsensitive reports on a public website.
  • Timeliness. Some HHS reports and more than half of MCC reports were posted a year or more after completion.
  • Dissemination planning. State does not currently have a policy requiring a plan that identifies potential users and the means of dissemination.

Why GAO Did This Study

The U.S. government plans to spend approximately $35 billion on foreign assistance in 2017. Evaluation is an essential tool for U.S. agencies to assess and improve the results of their programs. Government-wide guidance emphasizes the importance of evaluation, and the Foreign Aid Transparency and Accountability Act of 2016 requires the President to establish guidelines for conducting evaluations. However, evaluations can be challenging to conduct. GAO has previously reported on challenges in the design, implementation, and dissemination of the evaluations of individual foreign assistance programs.

GAO was asked to review foreign aid evaluations across multiple agencies. This report examines the (1) quality, (2) cost, and (3) dissemination of foreign aid program evaluations. GAO assessed a representative sample of 173 fiscal year 2015 evaluations for programs at the six agencies providing the largest amounts of U.S. foreign aid —USAID, State, MCC, HHS's Centers for Disease Control and Prevention under the President's Emergency Plan for AIDS Relief, USDA's Foreign Agricultural Service, and DOD's Global Train and Equip program—against leading evaluation quality criteria; analyzed cost and contract documents; and reviewed agency websites and dissemination procedures.

What GAO Recommends

GAO recommends that each of the six agencies develop a plan to improve the quality of its evaluations and that HHS, MCC, State, and USDA improve their procedures and planning for disseminating evaluation reports.

The agencies concurred with our recommendations.

For more information, contact Jessica Farb at (202) 512-6991 or farbj@gao.gov.

Recommendations for Executive Action

  1. Status: Open

    Comments: In a letter provided to GAO in May 2017, MCC stated that it had responded to our finding that MCC needed to more clearly document the independence of its evaluators and fully disclose any potential conflicts of interest in the published final evaluations. In response to this recommendation, MCC revised its standard evaluator contract language and now requires the evaluator's independence, and any potential conflicts of interest, be fully documented in the published evaluation report. GAO will follow up on documentation of this and any other steps responsive to our recommendation.

    Recommendation: To improve the reliability and usefulness of program evaluations for agency program and budget decisions, the Chief Executive Officer of MCC, the Administrator of USAID, the Secretary of Agriculture, the Secretary of Defense, the Secretary of State, and the Secretary of Health and Human Services (in cooperation with State's Office of the U.S. Global AIDS Coordinator and Health Diplomacy) should each develop a plan for improving the quality of evaluations for the programs included in our review, focusing on areas where our analysis has shown the largest areas for potential improvement.

    Agency Affected: Millennium Challenge Corporation

  2. Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.

    Recommendation: To improve the reliability and usefulness of program evaluations for agency program and budget decisions, the Chief Executive Officer of MCC, the Administrator of USAID, the Secretary of Agriculture, the Secretary of Defense, the Secretary of State, and the Secretary of Health and Human Services (in cooperation with State's Office of the U.S. Global AIDS Coordinator and Health Diplomacy) should each develop a plan for improving the quality of evaluations for the programs included in our review, focusing on areas where our analysis has shown the largest areas for potential improvement.

    Agency Affected: Department of Agriculture

  3. Status: Open

    Comments: In May 2017, DOD provided a response to our final report, dated March 30, 2017. The response noted that, in January 2017, the Department established policy on assessment, monitoring, and evaluation (AM&E) for security cooperation with the goal of improving the quality of program evaluation across the Department. DOD also stated that it would review best practices for AM&E to determine those that are best suited for security assistance, and will discuss our findings and recommendations with the independent evaluator who conducts security assistance evaluations and encourage them to consider the recommendations in future evaluations. GAO will review the January 2017 guidance and will follow-up on specific steps taken by the DOD to incorporate best practices.

    Recommendation: To improve the reliability and usefulness of program evaluations for agency program and budget decisions, the Chief Executive Officer of MCC, the Administrator of USAID, the Secretary of Agriculture, the Secretary of Defense, the Secretary of State, and the Secretary of Health and Human Services (in cooperation with State's Office of the U.S. Global AIDS Coordinator and Health Diplomacy) should each develop a plan for improving the quality of evaluations for the programs included in our review, focusing on areas where our analysis has shown the largest areas for potential improvement.

    Agency Affected: Department of Defense

  4. Status: Open

    Comments: In a response to GAO in May 2017, HHS stated that the CDC Operationalization of the Evaluation Standards of Practice (ESOP) was updated in January 2017 and provides guidance on evaluation planning, protocol development, implementation, reporting, dissemination, and use of evaluation results, as well as reporting requirements. HHS also stated that it now reviews all evaluation and performance monitoring plans and assess report quality at several stages prior to publication and has begun to provide webinars and templates for evaluators to ensure that standards are addressed and reflected in the report. GAO will review the updated documents and request documentation of the actions taken for responsiveness to our recommendation.

    Recommendation: To improve the reliability and usefulness of program evaluations for agency program and budget decisions, the Chief Executive Officer of MCC, the Administrator of USAID, the Secretary of Agriculture, the Secretary of Defense, the Secretary of State, and the Secretary of Health and Human Services (in cooperation with State's Office of the U.S. Global AIDS Coordinator and Health Diplomacy) should each develop a plan for improving the quality of evaluations for the programs included in our review, focusing on areas where our analysis has shown the largest areas for potential improvement.

    Agency Affected: Department of Health and Human Services

  5. Status: Open

    Comments: In a letter dated March 31, 2017, State reported that it would be expanding its evaluation policy into the new Program Design and Performance Management Policy for Programs, Projects, and Processes, expected to be in place by summer 2017. The policy and its implementation, along with the recently published Program Design and Performance Management toolkit, as well as updated policy guidance, constitute State's plan to improve evaluations. GAO will review the updated policy when it is released for its responsiveness to our recommendation.

    Recommendation: To improve the reliability and usefulness of program evaluations for agency program and budget decisions, the Chief Executive Officer of MCC, the Administrator of USAID, the Secretary of Agriculture, the Secretary of Defense, the Secretary of State, and the Secretary of Health and Human Services (in cooperation with State's Office of the U.S. Global AIDS Coordinator and Health Diplomacy) should each develop a plan for improving the quality of evaluations for the programs included in our review, focusing on areas where our analysis has shown the largest areas for potential improvement.

    Agency Affected: Department of State

  6. Status: Open

    Comments: In a letter provided to GAO in follow-up to our report, USAID stated that steps already taken include (1)recently updating and clarifying the requirements and quality standards for evaluations and (2) working to ensure that staff has the skills they need to manage evaluations through training and other capacity building actions. As of September 2017, GAO is reviewing the updated guidance and actions against the specific findings in our report to assess their responsiveness in addressing our findings.

    Recommendation: To improve the reliability and usefulness of program evaluations for agency program and budget decisions, the Chief Executive Officer of MCC, the Administrator of USAID, the Secretary of Agriculture, the Secretary of Defense, the Secretary of State, and the Secretary of Health and Human Services (in cooperation with State's Office of the U.S. Global AIDS Coordinator and Health Diplomacy) should each develop a plan for improving the quality of evaluations for the programs included in our review, focusing on areas where our analysis has shown the largest areas for potential improvement.

    Agency Affected: United States Agency for International Development

  7. Status: Closed - Implemented

    Comments: In comments on a draft of this report the Centers for Disease Control and Prevention (CDC) agreed with GAO's recommendation that it update its guidance and practices on the posting of evaluations to require the President?s Emergency Plan for AIDS Relief (PEPFAR) evaluations to be posted within the timeframe required by PEPFAR guidance. CDC noted that it was updating its guidance to require that all PEPFAR evaluations be posted online within 90 days, as required by PEPFAR. CDC took this action in January 2017, after our report was made available to them for agency comment and prior to the report's final publication on March 3, 2017.

    Recommendation: To better ensure that the evaluation findings reach their intended audiences and are available to facilitate incorporating lessons learned into future program design or budget decisions, the Secretary of Health and Human Services should direct the Centers for Disease Control and Prevention to update its guidance and practices on the posting of evaluations to require President's Emergency Plan for AIDS Relief (PEPFAR) evaluations to be posted within the timeframe required by PEPFAR guidance.

    Agency Affected: Department of Health and Human Services

  8. Status: Open

    Comments: In a letter provided to GAO in May 2017, MCC stated that it had initiated a re-design of its evaluation monitoring information system to provide MCC with detailed timelines of each component of the evaluation review and publication process. GAO will follow up on documentation of this and any other steps responsive to our recommendation.

    Recommendation: To better ensure that the evaluation findings reach their intended audiences and are available to facilitate incorporating lessons learned into future program design or budget decisions, the Chief Executive Officer of MCC should adjust MCC evaluation practices to make evaluation reports available within the timeframe required by MCC guidance.

    Agency Affected: Millennium Challenge Corporation

  9. Status: Open

    Comments: In a letter dated March 31, 2017, State reported that it would be expanding its evaluation policy into the new Program Design and Performance Management Policy for Programs, Projects, and Processes, expected to be in place by summer 2017. State reported that it would add a requirement for dissemination plans to the new policy. GAO will review the updated policy when it is released for its responsiveness to our recommendation.

    Recommendation: To better ensure that the evaluation findings reach their intended audiences and are available to facilitate incorporating lessons learned into future program design or budget decisions, the Secretary of State should amend State's evaluation policy to require the completion of dissemination plans for all agency evaluations

    Agency Affected: Department of State

  10. Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.

    Recommendation: To better ensure that the evaluation findings reach their intended audiences and are available to facilitate incorporating lessons learned into future program design or budget decisions, the Secretary of Agriculture should implement guidance and procedures for making FAS evaluations available online and searchable on a single website that can be accessed by the general public.

    Agency Affected: Department of Agriculture

 

Explore the full database of GAO's Open Recommendations »

Nov 21, 2017

Nov 13, 2017

Nov 2, 2017

Oct 31, 2017

Oct 26, 2017

Oct 19, 2017

Oct 12, 2017

Oct 4, 2017

Sep 28, 2017

Looking for more? Browse all our products here