This is the accessible text file for GAO report number GAO-09-980 entitled 'International Food Assistance: USAID Is Taking Actions to Improve Monitoring and Evaluation of Nonemergency Food Aid, but Weaknesses in Planning Could Impede Efforts' which was released on September 28, 2009. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Report to Congressional Committees: United States Government Accountability Office: GAO: September 2009: International Food Assistance: USAID Is Taking Actions to Improve Monitoring and Evaluation of Nonemergency Food Aid, but Weaknesses in Planning Could Impede Efforts: GAO-09-980: GAO Highlights: Highlights of GAO-09-980, a report to congressional committees. Why GAO Did This Study: In passing the Food for Peace Act in 2008, Congress authorized up to $22 million annually for fiscal years 2009 to 2012 to the U.S. Agency for International Development (USAID) to improve, monitor, and evaluate the effectiveness and efficiency of nonemergency food aid programs. Congress also required USAID to report on its oversight of these programs and the Comptroller General to review and report to Congress on USAID’s report. Through analysis of agency documents; interviews with agency officials, experts, and partners; and visits to Bangladesh and Haiti, this mandated report reviews (1) USAID’s plans and actions to improve its monitoring and evaluation of nonemergency food aid programs and (2) the extent to which USAID has integrated its monitoring and evaluation of nonemergency food aid with program management. What GAO Found: USAID’s actions to improve its monitoring and evaluation of these programs could be hindered by weak planning. Monitoring is essential to ensuring that USAID’s nonemergency food aid programs in developing countries are implemented as intended, and evaluation helps to assure that these programs achieve their goal of reducing global food insecurity. First, with funding from the Food for Peace Act, USAID’s Office of Food for Peace (FFP) plans to increase the number of field staff responsible for the monitoring of nonemergency food aid programs, has provided funding for the Famine Early Warning Systems Network, and has initiated an upgrade of its information technology system. However, FFP’s plans for the information technology upgrade lack a concept of operations document, which describes system characteristics for a proposed system from a user’s point of view and includes high-level descriptions of information systems, their interrelationships, and information flows. Second, with funding from other sources, USAID plans to carry out additional actions in an effort to improve its oversight of food aid, including the expansion of a computerized system for monitoring the implementation and management of nonemergency food aid programs. However, USAID has not determined a stable source of funding for these initiatives beyond the first year of operations due to legal restrictions that preclude the agency from using the newly authorized funding for grants and cooperative agreements. USAID’s monitoring and evaluation of its nonemergency food aid programs are consistent to varying degrees with some of the principles established by the American Evaluation Association’s Task Force on Evaluation Policy to integrate evaluation into program management. GAO found that (1) FFP’s actions were generally consistent with the principles for policies and procedures and for independence. For example, FFP has issued policies and procedures for monitoring and evaluating food aid programs and generally uses external evaluators to assess its multiyear assistance programs. (2) FFP’s actions were partially consistent with the principles for scope and coverage, dissemination of results, professional competence, and resources. For example, FFP relies on a range of staff to perform its monitoring and evaluation, but does not have reliable data on the numbers of field staff who have competencies in monitoring and evaluation, or their specific skills. (3) FFP’s actions were not consistent with the principles for monitoring and evaluation plans. While FFP plans some of its monitoring and evaluation activities—such as final evaluations for multiyear assistance programs—it lacks an integrated plan to ensure that monitoring and evaluation results will be used to improve program management. What GAO Recommends: GAO recommends that the USAID Administrator (1) develop a concept of operations document to help reduce the risks associated with upgrading FFP’s information technology system and (2) develop an integrated monitoring and evaluation plan to better oversee nonemergency food aid programs. USAID agreed with the first recommendation, but disagreed with the second recommendation, asserting that GAO did not fully consider its planning processes. GAO believes the recommendation remains valid in view of current findings. View [hyperlink, http://www.gao.gov/products/GAO-09-980] or key components. For more information, contact Thomas Melito at (202) 512- 9601 or melitot@gao.gov. [End of section] Contents: Letter: Results in Brief: Background: Weaknesses in Planning Could Hinder USAID's Actions to Improve Monitoring and Evaluation of Nonemergency Food Aid Programs: USAID's Monitoring and Evaluation of Nonemergency Food Aid Are Not Fully Integrated with Program Management: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Objectives, Scope, and Methodology: Appendix II: USAID Report to Congress: Appendix III: Examples of Activities Supported with Nonemergency Food Aid: Appendix IV: Comments from USAID: Appendix V: GAO Contact and Staff Acknowledgments: Related GAO Products: Table: Table 1: Types of Monitoring and Evaluation Activities Performed by FFP Staff, Implementing Partners, and External Evaluators: Figures: Figure 1: Countries That Received U.S. Nonemergency Food Aid in Fiscal Year 2008: Figure 2: A Framework for Program Management of Nonemergency Food Aid That Integrates Monitoring and Evaluation: Figure 3: USAID Allocations of the Food for Peace Act Funding for Monitoring and Evaluation in Fiscal Year 2009 and Implementation Timelines: Figure 4: FFP Priority Countries and Locations of New Positions for the Monitoring of Nonemergency Food Aid Programs: Figure 5: Examples of Activities Supported with Nonemergency Food Aid: Abbreviations: ADS: Automated Directives System: AEA: American Evaluation Association: AED: Academy for Educational Development: BEST: Bellmon Estimation for Title II: DA: Development Assistance: DAP: Development Assistance Program: DEC: Development Experience Clearinghouse: EFP: emergency food product: FANTA-2: Food and Nutrition Technical Assistance II Project: FAO: Food and Agriculture Organization: FEWS NET: Famine Early Warning Systems Network: FFP: Office of Food for Peace: FFPMS: Food for Peace Management System: IEEE: Institute of Electrical and Electronics Engineers: MYAP: Multiyear Assistance Program: NGO: nongovernmental organization: OFDA: Office of Foreign Disaster Assistance: PDA: personal data assistant: PM2A: Preventing Malnutrition in Children Under 2 Approach: SYAP: Single-Year Assistance Program: UN: United Nations: USAID: U.S. Agency for International Development: WFP: World Food Program: [End of section] United States Government Accountability Office: Washington, DC 20548: September 28, 2009: The Honorable Blanche Lambert Lincoln: Chair: The Honorable Saxby Chambliss: Ranking Minority Member: Committee on Agriculture, Nutrition, and Forestry: United States Senate: The Honorable Collin C. Peterson: Chair: The Honorable Frank D. Lucas: Ranking Member: Committee on Agriculture: House of Representatives: The Honorable Howard L. Berman: Chair: The Honorable Ileana Ros-Lehtinen: Ranking Member: Committee on Foreign Affairs: House of Representatives: The number of undernourished people worldwide has been growing and now exceeds 1 billion, according to the estimates of the United Nations (UN) Food and Agriculture Organization (FAO). This calls into question the feasibility of attaining the 1996 World Food Summit target of reducing the number of hungry people by half to no more than 420 million by 2015. The United States is the world's largest donor of food aid, and the U.S. Agency for International Development (USAID) Office for Food for Peace (FFP) administers the largest nonemergency food aid program funded under the Food for Peace Act.[Footnote 1] FFP has adopted as its strategic objective the reduction of food insecurity in vulnerable populations. Specifically, FFP's Strategic Plan for 2006- 2010 identifies two intermediate results: (1) enhancing FFP's global leadership in reducing global food insecurity and (2) increasing FFP's program impact in the field.[Footnote 2] Monitoring[Footnote 3] and evaluation[Footnote 4] of nonemergency food aid--one of the largest sources of U.S. funding focused on food security--are critical oversight and program management tools that would help to ensure that FFP meets its strategic objective. The size and scope of FFP's activities are substantial--in fiscal year 2008, FFP supported 20 grantees that implemented 92 activities in 28 countries-- but a number of recent reviews have found that monitoring and evaluation of these activities have been insufficient. In 2007, we found that the USAID Inspector General had previously reported that monitoring had not been regular and systematic and that, in some cases, intended recipients had not received food aid or the number of recipients could not be verified. In April 2007, we reported that USAID had only 23 Title II-funded staff assigned to missions and regional offices in just 10 countries to monitor programs that cost about $1.7 billion in 55 countries in the previous fiscal year. Our audit work also indicated that monitoring had been insufficient due to various factors including limited staff, competing priorities, and legal restrictions on the use of food aid resources. To ensure the effective use of food aid, we recommended that USAID improve its monitoring of food aid programs to strengthen proper management and implementation of these programs.[Footnote 5] USAID agreed, noting that monitoring of all its grant programs is required and that its implementing partners are legally required to monitor and report their findings to USAID. Like monitoring, evaluation is crucial to ensuring that best practices and lessons learned are considered in the management and implementation of existing programs and in designing new ones. However, for many years, USAID did not place a great deal of importance or invest many resources in evaluation. In a June 2009 speech, the Secretary of State, whose department coordinates U.S. development assistance, recognized that "… too often, program evaluation is treated as an afterthought, and sometimes is overlooked entirely." To rectify this situation, the Secretary announced that the Department of State (State) had for the first time developed a policy for integrating evaluation into its work worldwide.[Footnote 6] In passing the Food for Peace Act in 2008, Congress authorized up to $22 million annually in funding for fiscal years 2009 through 2012 to USAID for program oversight to improve, monitor, and evaluate the effectiveness and efficiency of nonemergency food aid programs--and, consistent with our 2007 recommendation, required the USAID Administrator to submit an implementation report on the agency's efforts in these areas.[Footnote 7] Congress also required the Comptroller General of the United States to review USAID's report and provide recommendations for improvement. To address this congressional mandate, we reviewed (1) USAID's plans and actions to improve its monitoring and evaluation of nonemergency food aid programs provided under the Food for Peace Act and (2) the extent to which USAID has integrated its monitoring and evaluation of nonemergency food aid with program management, particularly with regard to established principles. In this review, we focused on oversight practices of FFP, not of USAID as a whole. To address these objectives, we examined USAID's December 2008 report to Congress, which outlines FFP's plans to improve its monitoring and evaluation of nonemergency food aid programs, and actions FFP has taken to date. We interviewed USAID officials, including FFP field staff responsible for monitoring the implementation of nonemergency food aid programs, and implementing partners such as nongovernmental organizations (NGO) and the World Food Program (WFP). We considered established principles for monitoring and evaluation--especially the "Roadmap" principles of the American Evaluation Association (AEA),[Footnote 8] which have been developed to integrate monitoring and evaluation with program management; GAO standards and prior GAO work; USAID guidance; and guidelines that the Institute of Electrical and Electronics Engineers (IEEE) has established for information technology.[Footnote 9] We conducted fieldwork in Bangladesh and Haiti,[Footnote 10] where we met with U.S. mission officials, host government officials, bilateral and multilateral donor representatives, and NGO representatives. In Bangladesh and Haiti, we also visited various WFP and USAID project sites supported by nonemergency food aid and food aid logistics facilities. We conducted this performance audit from March 2009 to September 2009, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. (Appendix I provides a detailed discussion of our objectives, scope, and methodology.) Results in Brief: USAID's actions to improve its monitoring and evaluation of nonemergency food aid programs could be hindered by weak planning. First, with funding from the Food for Peace Act, USAID's FFP (1) plans to use $5 million to increase the number of field staff responsible for the monitoring of food aid programs, but it does not have a systematic process in place to determine appropriate staffing levels for effective monitoring and evaluation; (2) has provided $7.6 million in funding for the Famine Early Warning Systems Network (FEWS NET) operational budget in fiscal year 2009; and (3) has issued a contract for $637,000 to begin upgrading its information technology system, which is expected to fully capture food commodity management data and have the capability to interface with USAID's core financial system. However, FFP's plans for the information technology upgrade lack a concept of operations document, which describes characteristics for a proposed system from a user's point of view and includes high-level descriptions of information systems, their interrelationships, and information flows. Second, with funding from other sources, FFP (1) plans to expand the use of Layers--a computerized system for monitoring the implementation and management of nonemergency food aid programs--from 3 pilot countries to 20 countries by the end of 2012; (2) has commissioned several cross-cutting evaluations of nonemergency food aid projects through USAID's cooperative agreement with the Food and Nutrition Technical Assistance (FANTA-2) Project of the Academy for Educational Development (AED) and plans to improve the quality of assessments and targeting approaches for FFP programs; and (3) has launched a 3-year pilot project intended to improve the market analysis required before food aid programs are approved in recipient countries. In May 2009, however, USAID's General Counsel determined that the agency does not have the legal authority to use Food for Peace Act funding for the cooperative agreement required to implement Layers and the cross- cutting evaluations, which are essential to improving monitoring and evaluation of USAID's nonemergency food aid programs. USAID has not made provisions for alternative sources of funding for these initiatives beyond the cost of the start-up and first year of operations. USAID's monitoring and evaluation of its nonemergency food aid programs are consistent to varying degrees with some of the "Roadmap" principles established by AEA to integrate evaluation into program management. We found that USAID's actions were (1) generally consistent with the AEA principles for policies and procedures and for independence; (2) partially consistent with the principles for scope and coverage, dissemination of results, professional competence, and resources; and (3) not consistent with the principles for monitoring and evaluation plans, as discussed below: * Policies and procedures. In accordance with AEA's recommendation for established and published policies for monitoring and evaluation, USAID's guidance--which applies to FFP--details the agency's monitoring and evaluation requirements. In addition, FFP has issued several information bulletins for staff and implementing partners that explain policies and requirements regarding monitoring and evaluation, reporting, and performance indicators. * Independence. To help ensure independence as outlined in AEA's "Roadmap" principles, FFP requires that each multiyear assistance program's final evaluation be performed by external evaluators. In addition, it generally uses external entities to perform cross-cutting evaluations with funding from the Food for Peace Act and other sources. * Scope and coverage. In accordance with AEA's principle that monitoring should be conducted for most of an agency's programs and throughout the programs' life cycle, FFP requires the monitoring and evaluation of all of its multiyear assistance programs and has published policies and procedures to guide their oversight. However, in recent years, FFP has conducted only a few periodic, cross-cutting evaluations that examined the relative effectiveness of a range of its programs, projects, and activities. * Dissemination of results. AEA recommends that agencies disseminate the results of their monitoring and evaluation activities on easily accessible Web sites. However, FFP does not routinely and consistently disseminate evaluation results. Although an FFP requirement states that project evaluations should be posted in a central electronic clearinghouse, FFP has not ensured that implementing partners routinely comply with it. * Professional competence. AEA recommends that agencies should have competent and qualified staff to perform monitoring and evaluation. FFP relies on a range of staff to perform its monitoring and evaluation, including field staff, implementing partners' monitoring and evaluation staff, and FANTA-2 staff. However, it does not have reliable data on the numbers of field staff who have monitoring and evaluation competencies, or their specific skills. * Resources. AEA recommends that agencies should have sufficient resources for monitoring and evaluation. Although the 2008 Food for Peace Act grants USAID up to $22 million in new funding annually in fiscal years 2009 through 2012 to improve, monitor, and evaluate the effectiveness and efficiency of nonemergency food aid programs, it is difficult to determine whether FFP's level of funding for proper oversight of nonemergency food aid is sufficient because FFP does not know exactly how much it spends on monitoring and evaluation and has not done an evaluation of its resource needs. * Monitoring and evaluation plans. AEA recommends that agencies should have multiyear plans, updated annually, that take into account evaluation results for program budgeting, as well as ongoing program development and management. While FFP plans some of its monitoring and evaluation activities, such as final evaluations for multiyear assistance programs, it lacks an integrated plan for all of its oversight activities that takes into account the need for evaluation results as well as risk-based policy-making and management needs. In addition, FFP neither systematically prioritizes its monitoring and evaluation needs nor links resources to risk-based needs. To reduce the risks associated with upgrading FFP's information technology system, we are recommending that the Administrator of USAID develop a concept of operations document. Such a document should adhere to industry best practices and include key elements such as major system components, interfaces to external systems, and performance characteristics. It would also describe system characteristics for a proposed system from the user's point of view and include high-level descriptions of information systems, their interrelationships, and information flows. In view of USAID's new authority to direct up to $88 million over a 4- year period to improve, monitor, and evaluate the effectiveness and efficiency of nonemergency food aid programs, we also recommend that USAID develop an integrated monitoring and evaluation plan for its nonemergency food aid programs. Such a plan should, among other things, (1) link monitoring and evaluation to key USAID and FFP goals; (2) establish a systematic process for determining appropriate budget levels and staff resources for monitoring and evaluation based on an analysis of factors such as the nature, scope, and size of the programs; (3) examine all available options for funding monitoring and evaluation at headquarters and in recipient countries; and (4) establish time frames for implementing and evaluating the plan. USAID concurred with our first recommendation, but disagreed with the second recommendation, asserting that we did not adequately take into account documents that detail its planning processes. We considered all the documents that USAID cited and found that they did not constitute a plan, but only provided guidance for planning. Furthermore, the weaknesses we identified in FFP's monitoring and evaluation efforts, including in the areas of resources and professional competence, demonstrate the need for a comprehensive monitoring and evaluation plan. USAID provided written comments on a draft of this report. We believe this recommendation remains valid. We have reprinted USAID's comments in appendix IV, along with our responses. Additionally, USAID provided technical comments on a draft of this report, which we have addressed as appropriate. Background: USAID Plans to Administer Larger Nonemergency Food Aid Programs in Fewer Priority Countries: FFP is shifting to larger programs in fewer countries in order to leverage nonemergency food aid resources more effectively. FFP's Strategic Plan for 2006-2010 concentrates on a single objective of reducing food insecurity in vulnerable populations with an aim to fund programs that can be modified when circumstances change. In fiscal year 2008, under the Food for Peace Act, the United States provided $354 million in nonemergency food aid and FFP administered nonemergency food aid programs in 28 countries. Beginning in fiscal year 2010, FFP will limit its nonemergency food aid programs to 20 priority countries. [Footnote 11] Of these, 16 are in sub-Saharan Africa (Burkina Faso, Burundi, Chad, the Democratic Republic of Congo, Ethiopia, Liberia, Madagascar, Malawi, Mali, Mauritania, Mozambique, Niger, Sierra Leone, Sudan, Uganda, and Zambia), 2 are in Central and South Asia (Afghanistan and Bangladesh), and 2 are in Latin America and the Caribbean (Guatemala and Haiti). Figure 1 lists the countries that received nonemergency food aid in fiscal year 2008 and the most recent available statistics on the number of undernourished people and the prevalence of undernourishment in these countries, respectively. Figure 1: Countries That Received U.S. Nonemergency Food Aid in Fiscal Year 2008 (U.S. dollars in millions): [Refer to PDF for image: horizontal bar graph] Country; Mali; Nonemergency food aid: $2.0 million; Number of undernourished people, 2003-2005: 1.2 million; Percentage of population that was undernourished in 2003-2005: 11%. Country; Guinea; Nonemergency food aid: $3.0 million; Number of undernourished people, 2003-2005: 1.5 million; Percentage of population that was undernourished in 2003-2005: 17%. Country; Senegal; Nonemergency food aid: $4.1 million; Number of undernourished people, 2003-2005: 3.0 million; Percentage of population that was undernourished in 2003-2005: 26%. Country; Chad; Nonemergency food aid: $4.5 million; Number of undernourished people, 2003-2005: 3.8 million; Percentage of population that was undernourished in 2003-2005: 39%. Country; Burundi; Nonemergency food aid: $4.9 million; Number of undernourished people, 2003-2005: 4.8 million; Percentage of population that was undernourished in 2003-2005: 63%. Country; Mauritania; Nonemergency food aid: $5.0 million; Number of undernourished people, 2003-2005: 0.2 million; Percentage of population that was undernourished in 2003-2005: 8%. Country; Ghana; Nonemergency food aid: $6.9 million; Number of undernourished people, 2003-2005: 1.9 million; Percentage of population that was undernourished in 2003-2005: 9%. Country; Sierra Leone; Nonemergency food aid: $6.9 million; Number of undernourished people, 2003-2005: 2.5 million; Percentage of population that was undernourished in 2003-2005: 47%. Country; Liberia; Nonemergency food aid: $7.7 million; Number of undernourished people, 2003-2005: 1.3 million; Percentage of population that was undernourished in 2003-2005: 40%. Country; Bolivia; Nonemergency food aid: $9.0 million; Number of undernourished people, 2003-2005: 2.0 million; Percentage of population that was undernourished in 2003-2005: 22%. Country; Zambia; Nonemergency food aid: $9.1 million; Number of undernourished people, 2003-2005: 5.1 million; Percentage of population that was undernourished in 2003-2005: 45%. Country; Nicaragua; Nonemergency food aid: $9.3 million; Number of undernourished people, 2003-2005: 1.2 million; Percentage of population that was undernourished in 2003-2005: 22%. Country; Kenya; Nonemergency food aid: $9.4 million; Number of undernourished people, 2003-2005: 11.0 million; Percentage of population that was undernourished in 2003-2005: 32%. Country; Democratic Republic of Congo; Nonemergency food aid: $9.8 million; Number of undernourished people, 2003-2005: 43.0 million; Percentage of population that was undernourished in 2003-2005: 76%. Country; Afghanistan; Nonemergency food aid: $9.9 million; Number of undernourished people, 2003-2005: not available; Percentage of population that was undernourished in 2003-2005: not available. Country; Burkina Faso; Nonemergency food aid: $10.0 million; Number of undernourished people, 2003-2005: 1.3 million; Percentage of population that was undernourished in 2003-2005: 10%. Country; Honduras; Nonemergency food aid: $10.2 million; Number of undernourished people, 2003-2005: 0.8 million; Percentage of population that was undernourished in 2003-2005: 12%. Country; Rwanda; Nonemergency food aid: $11.4 million; Number of undernourished people, 2003-2005: 3.6 million; Percentage of population that was undernourished in 2003-2005: 40%. Country; Madagascar; Nonemergency food aid: $11.8 million; Number of undernourished people, 2003-2005: 6.6 million; Percentage of population that was undernourished in 2003-2005: 37%. Country; Niger; Nonemergency food aid: $12.8 million; Number of undernourished people, 2003-2005: 3.7 million; Percentage of population that was undernourished in 2003-2005: 29%. Country; India; Nonemergency food aid: $13.5 million; Number of undernourished people, 2003-2005: 230.5 million; Percentage of population that was undernourished in 2003-2005: 21%. Country; Guatemala; Nonemergency food aid: $13.9 million; Number of undernourished people, 2003-2005: 2.0 million; Percentage of population that was undernourished in 2003-2005: 16%. Country; Malawi; Nonemergency food aid: $17.9 million; Number of undernourished people, 2003-2005: 3.8 million; Percentage of population that was undernourished in 2003-2005: 29%. Country; Mozambique; Nonemergency food aid: $19.9 million; Number of undernourished people, 2003-2005: 7.5 million; Percentage of population that was undernourished in 2003-2005: 38%. Country; Ethiopia; Nonemergency food aid: $23.2 million; Number of undernourished people, 2003-2005: 35.2 million; Percentage of population that was undernourished in 2003-2005: 46%. Country; Uganda; Nonemergency food aid: $26.4 million; Number of undernourished people, 2003-2005: 4.1 million; Percentage of population that was undernourished in 2003-2005: 15%. Country; Haiti; Nonemergency food aid: $34.2v Number of undernourished people, 2003-2005: 5.3 million; Percentage of population that was undernourished in 2003-2005: 58%. Country; Bangladesh; Nonemergency food aid: $47.8 million; Number of undernourished people, 2003-2005: 40.1 million; Percentage of population that was undernourished in 2003-2005: 27%. Sources: GAO analysis of USAID and Food and Agriculture Organization of the United Nations data. [End of figure] Food for Peace Nonemergency Food Aid Programs Are Implemented through Grants for Single-Year and Multiyear Assistance Programs: FFP provides food aid funding under the Food for Peace Act using two primary funding mechanisms--Single-Year Assistance Programs (SYAP) and Multiyear Assistance Programs (MYAP). SYAPs address emergency needs and are used for direct distribution of food aid in emergency contexts.[Footnote 12] MYAPs, long-term programs with duration of 3 to 5 years, are used to implement development activities that target chronically food-insecure populations.[Footnote 13] In carrying out nonemergency programs through its implementing partners, FFP uses a combination of Food for Peace commodities for direct distribution, local currency generated through monetization,[Footnote 14] and cash resources to implement activities that target chronic food insecurity. (For illustrative examples of the types of activities supported by nonemergency food aid, such as project activities we observed in Bangladesh and Haiti, see appendix III.) Monitoring and Evaluation Are Essential to Effective Program Management: Monitoring and evaluation perform two separate but related functions. Monitoring is the collection of data to determine whether programs are being implemented as intended and the tracking of progress using preselected indicators throughout the life of the program. Evaluation consists of periodic or ad hoc studies to assess how and whether programs achieved their expected goals. It can also consider what outcomes can be attributed to the program, as well as program cost- effectiveness. Evaluations can rely on a range of quantitative and qualitative measures as well as preselected indicators, comprehensive research designs, and the use of appropriate statistical analysis of the data. Figure 2 outlines a framework based on our review of Food for Peace nonemergency programs, as well as AEA and WFP guidance that illustrates how monitoring and evaluation could be integrated with program management[Footnote 15] of nonemergency food aid. Figure 2: A Framework for Program Management of Nonemergency Food Aid That Integrates Monitoring and Evaluation: [Refer to PDF for image: illustration] Program Management: Assessment and analysis: * Assess the context, extent, and magnitude of needs arising from current or impending food insecurity. * Prioritize assistance for affected population groups and geographical areas. Planning: * Identify characteristics and needs of target population. * Develop objectives. * Select food distribution and/or development activity. Monitoring: * Determine if necessary inputs (equipment, supplies, and personnel) are in place. – Ensure that food is properly stored. – Check to see if health clinics are sufficiently stocked for intended beneficiaries. * Determine if programs are achieving their expected outputs by tracking performance indicators. * Ensure that target number of beneficiaries are receiving food and attending clinics. Implementation: * Early childhood development (accompanied by photograph); * Mother and child health and nutrition (accompanied by photograph); * Income generating activities (accompanied by two photographs). Evaluation: Focus on the extent to which program objectives were achieved and what outcomes can be directly attributed to programs. * Direct distribution programs may consider how the ration is affecting the health and nutritional status of beneficiaries. * Food-for-work programs may focus on how the activity is addressing temporary food insecurity and developing a sustainable community asset. Sources: GAO analysis, adapted from WFP information; GAO and USAID (photos). [End of figure] FFP Relies on a Range of Staff to Perform Various Types of Monitoring and Evaluation Activities: FFP relies on a range of staff to perform its monitoring and evaluation--including FFP officers in the field, implementing partners, contractors, and FANTA-2 staff.[Footnote 16] According to FFP, field staff visit project sites to ensure that programs are being implemented as expected. They also approve and oversee the implementing partners' selection of performance indicators and data collection. In addition to regularly collecting performance data, the implementing partners perform baseline studies and midterm evaluations for their multiyear programs and hire external evaluators to conduct the final evaluations for the multiyear programs. In addition, FFP relies on appropriately credentialed consultants and researchers for some cross-cutting evaluations, which, for example, consider the agency's exit strategies for nonemergency food aid programs in Bolivia, Honduras, and Kenya. Conducting evaluations can present many challenges, such as obtaining reliable data and employing qualified evaluators. These challenges can be compounded in developing countries, where local conditions may be difficult and local capacity is often limited. For example, a recent review of evaluations conducted by an international NGO cited numerous challenges in-country including climate, poor economic status, civil unrest, migration of beneficiaries, cultural biases, poor recordkeeping, and turnover in government and NGO staff.[Footnote 17] Table 1 summarizes the various types of monitoring and evaluation activities that FFP performs. The table also indicates which staff have primary responsibility for the activities, briefly describes the activities, and provides examples of them. Table 1: Types of Monitoring and Evaluation Activities Performed by FFP Staff, Implementing Partners, and External Evaluators: Types of monitoring and evaluation activities: Monitoring of project sites and activities; Staff with primary responsibility: USAID mission staff; Description of activities: Field staff visit project sites to check on inputs and program implementation; Examples: Field staff checks on, for example: * storage conditions of food aid warehouses, and; * whether clinics for mothers and children offer the required services. Types of monitoring and evaluation activities: Tracking of performance measures; Staff with primary responsibility: Implementing partners with oversight from FFP and the USAID mission; Description of activities: Implementing partners select performance measures used by FFP, State/F,[A] and the mission. USAID headquarters staff review and approve the indicators. Implementing partners collect data on the measures; Examples: USAID and implementing partners track, for example: * FFP indicators (e.g., the number of farmers who received extension services), and; * State/F indicators (e.g., the number of beneficiaries of agricultural assistance). Types of monitoring and evaluation activities: Evaluating program outcomes; Staff with primary responsibility: External evaluators; Description of activities: Implementing partners hire external evaluators to conduct the projects' final evaluations, and determine whether the goals were achieved; Examples: One final evaluation reported a reduction in acute malnutrition in children aged 6-59 months during the life of the project. The evaluations generally also provided reasons that goals were or were not met, and lessons learned. Types of monitoring and evaluation activities: Evaluating program impacts; Staff with primary responsibility: FANTA-2, its partner organizations, and contractors to FFP; Description of activities: FFP has commissioned several cross-cutting studies designed to evaluate what impacts can be attributed to its programs and to analyze the relative effectiveness of alternative delivery mechanisms; Examples: One evaluation examined the impact and cost-effectiveness of two approaches to prevent malnutrition in infants, using randomly selected comparison groups to judge the impacts and effectiveness of each approach. Types of monitoring and evaluation activities: Related studies; Staff with primary responsibility: FANTA-2, its partner organizations, and contractors to FFP; Description of activities: Studies that are intended to help understand and improve aspects of program management; Examples: FFP plans another full review of all its multiyear projects. The last such review recommended a series of improvements in, for example, program design and results reporting. Sources: GAO analysis of AED/FANTA-2 and USAID documents. [A] In June 2006, the Office of the Director of Foreign Assistance (State/F) was created at State to carry out the Director of Foreign Assistance's responsibilities and focus the use of foreign assistance on achieving the Secretary's transformational diplomacy goal: "to help build and sustain democratic, well-governed states that will respond to the needs of their people, reduce widespread poverty, and conduct themselves responsibly in the international system." State/F was given responsibility for developing, among other things (1) a coherent, coordinated U.S. government foreign assistance strategy; (2) multiyear country-specific assistance strategies and annual country specific operational plans; (3) consolidated policy, planning, budget, and implementation mechanisms and staff functions required to provide leadership to USAID and State foreign assistance; and (4) guidance for foreign assistance delivered through other U.S. government agencies. [End of table] Weaknesses in Planning Could Hinder USAID's Actions to Improve Monitoring and Evaluation of Nonemergency Food Aid Programs: USAID has begun taking a series of actions to improve its monitoring and evaluation of nonemergency food aid programs, as mandated by the Food for Peace Act passed in July 2008. These efforts include (1) increasing the number of field staff responsible for the monitoring of nonemergency food aid programs; (2) expanding FEWS NET; (3) upgrading FFP's information technology system; (4) expanding the use of Layers, a computerized monitoring system for quality assurance; (5) improving the quality of evaluations, assessments, and targeting approaches; and (6) launching a pilot project to improve market analysis in countries receiving U.S. nonemergency food aid. Weaknesses in planning could hinder USAID's ability to effectively implement these actions, but it is too early to assess impact because the agency's efforts are still in progress. As of August 2009, USAID estimated that, in fiscal year 2009, it will have spent about $13 million of the $22 million authorized for monitoring and evaluation of nonemergency food aid under the Food for Peace Act. Figure 3 summarizes the allocation of the funds and the implementation timelines. Figure 3: USAID Allocations of the Food for Peace Act Funding for Monitoring and Evaluation in Fiscal Year 2009 and Implementation Timelines: [Refer to PDF for image: pie-chart and timeline] Allocation of Food for Peace Act funding: New FFP field staff positions, $5,000,000[A]: 23%; FEWS NET, $7,600,000: 35%; Information technology upgrade, $637,000: 3%; Monetization study, $5,000: .02%; Unused funding, $8,758,000: 40%. Implementation timelines: New FFP field staff positions: Late FY 2008 through Mid-FY 2010; FEWS NET: Early FY2009 through Late FY2009; Information technology upgrade: Late FY2009 through Late FY2010; Monetization study: Late FY2009 through Early FY2010. Source: GAO analysis of USAID data. [A] As of August 2009, USAID had filled 9 of the 21 new positions and will therefore not spend the full $5 million it planned to spend within the fiscal year. [End of figure] USAID Has Allocated New Funding to Increase Staffing Levels and Upgrade Information Systems but Has Not Developed Key Processes and Plans in These Areas: Staffing Levels: USAID has announced plans to increase the number of field staff responsible for the monitoring of food aid programs, which would provide minimal FFP presence in all priority countries. The agency plans to use approximately $5 million of its new Food for Peace Act funding to add 21 full-time field staff whose responsibilities include the monitoring of nonemergency food aid programs. According to USAID, 1 of the new positions is located in USAID's Southern Africa Regional Office in Pretoria (South Africa) while the other 20 new positions are or will be located in designated priority countries for nonemergency food aid programs (including 19 in sub-Saharan Africa and 1 in Guatemala). As of August 2009, all 21 of the new positions had been established, of which 9 had been filled.[Footnote 18] Figure 4 shows the location of the new positions, including those filled, in FFP's 20 priority countries for nonemergency food aid. Figure 4: FFP Priority Countries and Locations of New Positions for the Monitoring of Nonemergency Food Aid Programs: [Refer to PDF for image: 3 maps] The map utilizes shading to depict the following: 20 priority countries for nonemergency food aid programs: Afghanistan; Bangladesh; Burkina Faso; Burundi; Chad; Democratic Republic of Congo; Ethiopia; Guatemala; Haiti; Liberia; Madagascar; Malawi; Mali; Mauritania; Mozambique; Niger; Sierra Leone; Sudan; Uganda; Zambia; New Food for Peace-funded positions, Filled: Burundi; Madagascar (3); Mail; Malawi; Niger; Southern Africa Regional Office; Zambia. New Food for Peace-funded positions, To be filled: Burkina Faso (2); Democratic Republic of Congo; Guatemala; Liberia; Madagascar; Niger; Sierra Leone (2); Sudan. Sources: GAO analysis of USAID data; Map Resources (map). [End of figure] We found that USAID does not have a systematic process in place to determine appropriate staffing levels for effective monitoring and evaluation. According to USAID officials, creating new positions in field missions where FFP has lacked staff was the primary consideration for their allocation. Hence, 17 of the new positions are in countries where USAID runs nonemergency food aid programs but has lacked FFP program management staff to date, namely: Burkina Faso, Burundi, Liberia, Madagascar, Malawi, Mali, Mauritania, Niger, Sierra Leone, and Zambia.[Footnote 19] The remaining 4 new positions augment existing FFP program management staff in the Democratic Republic of Congo, Guatemala, Sudan, and the Southern Africa Regional Office in Pretoria. Information Technology System: USAID plans to replace its current Food for Peace Information System with a new Food for Peace Management System (FFPMS) that it expects will address several of FFP's current information technology issues, including the collection, tracking, and interfacing of food aid program data. According to USAID's December 2008 report to Congress, the new FFPMS is expected to involve collaboration and testing with users in each business area in which the system is used. These areas include budgeting, proposal, document management, commodity purchase, commodity booking and shipping, tracking, monitoring and evaluation, reporting, closeout, and interface with internal and external information systems. In addition, USAID officials explained that, unlike FFPIS, FFPMS is intended to be an all-inclusive system that will fully capture the movement of U.S. food aid commodities, used for both emergency and nonemergency purposes, from the initial transfer authorization to the commodity's final distribution site. FFPMS is also expected to (1) capture implementing partners' data on Food for Peace resources, including revenues from monetization transactions as well as costs incurred; (2) allow FFP to aggregate information for its implementing partners' annual results reports; and (3) have the capability to interface with USAID internal systems--such as USAID's core financial system, known as Phoenix, which captures all USAID transactions--as well as with external agency information systems with which FFP conducts business to allow the collection and sharing of information among and between agencies. With systematic and electronic collection of information, USAID anticipates being able to monitor the relative efficiency of its nonemergency food aid programs and adequately monitor the degree to which monetization revenues cover costs. USAID is in the initial stages of this planned information systems upgrade. In June 2009, USAID awarded a contract for $637,000 to develop a management strategy and framework for implementing FFPMS by the end of fiscal year 2010. The contractor's responsibilities, as defined in the contract, are to revalidate assumptions made within the 2005 requirements and architecture diagram;[Footnote 20] revalidate FFP's technical environment; and update the requirements, documents, and architectural design. The contractor is also expected to make a recommendation to USAID by December 2009 for a commercial off-the-shelf software package that would best fit FFP's needs. A concept of operations document is a critical element in developing information system requirements and testing activities. However, USAID has not developed such a document for this initiative. This document is normally one of the first documents produced during a disciplined development effort because it describes system characteristics for a proposed system from a user's point of view.[Footnote 21] The key elements that should be included in the document are major system components, interfaces to external systems, and performance characteristics, such as speed and volume. A concept of operations document defines how an organization's day-to-day activities will be carried out to meet mission needs and includes high-level descriptions of information systems, their interrelationships, and information flows. It also describes the operations that must be performed, who must perform them, and where and how the operations will be carried out. Famine Early Warning Systems Network: To provide timely and rigorous early warning and vulnerability information on emerging food security issues, USAID is making a contribution to the FEWS NET fiscal year 2009 operational budget. FEWS NET monitors and analyzes relevant data, using multiple technologies such as satellites and field observations, to identify threats to food security and their impacts on livelihoods and markets. Chemonics, the private firm responsible for implementing FEWS NET along with several U.S. government agencies, currently maintains 20 national and 3 regional field offices in sub-Saharan Africa, Central America, Haiti, and Afghanistan. Anticipating an increase in food insecurity in countries where there is no FEWS NET presence, Chemonics plans to implement three remote monitoring pilot projects--in Burundi, Pakistan, and Yemen--that are expected to be completed by the end of fiscal year 2010.[Footnote 22] In fiscal year 2009, USAID is providing $7.6 million of the newly authorized Food for Peace Act funding to support FEWS NET. This funding is not designated to support the remote monitoring pilot projects but is a contribution to FEWS NET's operational budget of $20 million, with the remaining $12.4 million coming from the Development Assistance (DA), the Office of Foreign Disaster Assistance (OFDA), and other sources. Legal Restrictions on Usage of Food for Peace Act Funding Could Hamper USAID's Plans to Expand FFP's New Monitoring System and Improve Evaluations: Plans to Expand FFP's Computerized Monitoring System for Quality Assurance: Legal restrictions on the allowable uses of Food for Peace Act funding could hamper USAID's plans to expand FFP's new monitoring system for quality assurance. USAID plans to expand the use of Layers--a computerized system using personal data assistant (PDA) devices for monitoring the implementation and management of nonemergency food aid programs--from 3 pilot countries to 20 countries by the end of 2012, through its multiyear cooperative agreement with AED.[Footnote 23] The purpose of Layers is to help FFP staff responsible for the monitoring of food aid programs to collect data to assess the quality of implementing partner operations in the field. Layers can be used to monitor a wide range of programs and activities, including the conditions of storage in commodity warehouses and the quality of services delivered by FFP's implementing partners, such as food-for- work activities, provision of community health services, and maternal and child health and nutrition. Prior to 2009, Layers was piloted in Haiti, Ethiopia, and Madagascar. USAID estimates the one-time cost for the start-up to introduce Layers to a new country to be approximately $144,000. It began rolling out the system in Guatemala, Mali, and Uganda in 2009 and plans to do so in all the remaining priority countries by the end of 2012 as their multiyear assistance programs are approved. However, in May 2009, USAID's General Counsel determined that the Food for Peace Act precludes the agency from using the newly authorized funding for nonemergency food aid monitoring and evaluation for grants and cooperative agreements.[Footnote 24] Section 3012(f)(5) of the Food for Peace Act authorizes USAID to enter into personal services contracts to carry out monitoring and evaluation but does not refer to grants or cooperative agreements.[Footnote 25] As a result, FFP cannot move forward with its original plans to use Food for Peace Act funding to expand the implementation of Layers and will instead need to use other sources of funding that could be made available for this purpose.[Footnote 26] According to FFP officials, the current plan is to use other funding streams, such as the DA account, to fund the costs of the start-up of Layers and the first year of the system's operation. However, these officials noted that due to competing priorities for DA, it cannot be relied upon as a stable source of funding. Plans to Improve the Quality of Evaluations: USAID plans to improve the quality of evaluations through a number of ongoing and future studies. Using $5,000 of the newly authorized funding for monitoring and evaluation, the agency has signed a contract with Michigan State University for a study of monetization that is expected to be available for release in December 2009. It has also commissioned several cross-cutting evaluations and related studies that are expected to improve the quality of assessments, targeting approaches, design, and evaluation of nonemergency food aid programs. For example, through a cooperative agreement with AED, USAID is funding the FANTA-2 project to initiate the following multiyear studies in 2009: * a study of exit strategies in Bolivia, Honduras, and Kenya, which will produce recommendations for effective ways to sustain program benefits after U.S. nonemergency food aid programs have been phased out; * a study to identify key components of success in Preventing Malnutrition in Children Under 2 Approach (PM2A) programs in Guatemala and Burundi; * a study to improve the early warning and response capacity in FFP's multiyear assistance programs in Haiti, Niger, and Uganda; * an update of the 2002 Food Aid and Food Security Assessment, which will evaluate progress made under FFP's Strategic Plan for 2006-2010; and: * a study testing the efficacy and safety of emergency food products (EFP) that is co-funded by USAID's Bureau for Global Health and FFP. In addition to these cross-cutting evaluations and studies, FANTA-2 prepared four food security country frameworks--for Bangladesh, Burkina Faso, Liberia, and Sierra Leone--in 2009.[Footnote 27] Food security country frameworks are assessments that provide country-specific guidance to partners developing new MYAP proposals and describe how Food for Peace resources should be programmed and integrated with other U.S. government funds by defining objectives, approaches, regions, and institutional partnerships for effective use of U.S. government resources dedicated to reducing food insecurity. FFP estimates the cost of FANTA-2's work on the cross-cutting evaluations, related studies, and food security country frameworks in fiscal year 2009 to be $4.4 million. To improve the quality of assessments and targeting approaches, FFP is also initiating a new cooperative agreement to identify best practices that will increase the efficiency and effectiveness of Food for Peace food aid program management and capacity building. The aim is to promote the broader adoption of best practices and collaboration among FFP's implementing partners. As of August 2009, FFP had issued a Request for Applications seeking to identify potential partners' interests, priorities, and capacities and estimates the cost of the award at $3.5 million. However, USAID has determined that the Food for Peace Act precludes the agency from using the newly authorized funding for monitoring and evaluation of nonemergency food aid for grants and cooperative agreements. As a result, the completion of the multiyear cross-cutting evaluations, as well as future food security country frameworks and "best practices" assessments is not assured. FFP's current plan is to fund these studies from the DA account. USAID Is Using Other Funding to Improve Assessments Conducted Prior to Approving U.S. Food Aid Programs in Recipient Countries but Faces Implementation Challenges: In August 2008, USAID launched a 3-year Bellmon Estimation for Title II (BEST) pilot project intended to improve the market analysis required before food aid programs are approved in recipient countries--commonly referred to as the Bellmon analysis.[Footnote 28] The project is funded through resources other than the newly authorized Food for Peace Act funding for monitoring and evaluation of nonemergency food aid. It aims to improve how nonemergency food aid data are collected and analyzed, to determine the appropriate level of food aid, as well as the appropriate commodities that can be distributed and monetized in each country. The pilot project reduces the potential for conflict of interest because the organization that does the analysis is no longer the same organization whose interest is directly affected by the outcome of the analysis. In the past, USAID's implementing partners for nonemergency food aid distribution or monetization conducted and submitted the Bellmon analysis to USAID. However, the outcome of the analysis affected whether they could conduct monetization and, if so, how much food aid they could monetize--which, in turn, directly affected their operating budgets. Under the pilot project, USAID will have one organization, Fintrac, carry out the initial market analysis for all 20 of USAID's priority countries. According to Fintrac, having one organization do the analysis for all countries could help to improve consistency in analytical methodology, reporting structure, and format. BEST is still in its early stage of implementation. It encountered some difficulties in its first year of implementation and has yet to address some additional methodological challenges. One key challenge is ensuring the quality and consistency of the analysis, which can be impaired by a lack of reliable and up-to-date data. For example, Fintrac was able to use data from a recent livelihoods survey done by the World Food Program and FEWS NET in the Malawi assessment but had to conduct additional research and trips to Guatemala because it had no comparable updated data for that country. In addition, Fintrac officials told us that they have not developed quantifiable measures and thresholds agreed upon by USAID and its implementing partners to help guide USAID's programming decisions on levels of monetization. Without quantifiable measures and thresholds, implementing partners could challenge USAID's determination of monetization levels in a country. Finally, the law requires that the distribution of commodities will not result in a substantial disincentive or interference with domestic production or marketing in a recipient country--but the information Fintrac uses for its analysis, such as the historic levels of monetization relative to a country's imports or production, does not directly answer the question of whether U.S. food aid will cause such disincentives. USAID and Fintrac officials acknowledged that additional research is needed in this area. As of July 2009, Fintrac had completed analyses for 7 priority countries: Burundi, Ethiopia, Guatemala, Madagascar, Malawi, Sudan, and Uganda, with plans to complete analyses for the remaining 13 priority countries through 2011. USAID's Monitoring and Evaluation of Nonemergency Food Aid Are Not Fully Integrated with Program Management: USAID's monitoring and evaluation of its nonemergency food aid programs are consistent to varying degrees with some of the principles established by AEA to integrate evaluation into program management. We found that USAID's actions were (1) generally consistent with the AEA principles for policies and procedures and for independence; (2) partially consistent with the principles for scope and coverage, dissemination of results, professional competence, and resources; and (3) not consistent with the principles for monitoring and evaluation plans. USAID's Actions Were Generally Consistent with the AEA Principles for Policies and Procedures and for Independence: FFP Has Issued Policies and Procedures for Monitoring and Evaluation: [Side bar: Policies and Procedures: Principles: * Agency should publish policies and procedures and adopt quality standards to guide the conduct of evaluations. * Policies and procedures should identify the kinds of evaluations to be performed and the criteria and administrative steps for developing evaluation plans and setting priorities; selecting the appropriate evaluation approaches and methods to be used; consulting subject matter experts; ensuring the quality of the evaluation products; publishing evaluation reports; and ensuring the independence of the evaluation function. Source: AEA, An Evaluation Roadmap for a More Effective Government. End of sidebar] FFP has established policies and procedures for monitoring and evaluation that are contained primarily in several information bulletins for staff and implementing partners.[Footnote 29] Guidance can also be found in the FFP Process and Procedures Manual, which outlines field and headquarters staff's roles and responsibilities for monitoring and evaluation. In addition, FFP has issued a series of technical papers by FANTA-2 that discuss the methodological standards and techniques that can be used for many aspects of monitoring and evaluation, such as statistical sampling for surveys of beneficiaries and preparing survey questions for key indicators. For example, one technical paper discusses in detail a key FFP indicator, household dietary diversity, and explains how to collect data through a questionnaire, tabulate the responses, and set appropriate targets to track progress over time. FFP's policies and procedures, in conjunction with USAID's official guidance, cover the types of evaluations that can be performed, methodologies that can be employed, ways to plan evaluations, the means to disseminate studies, and the independence of the staff performing the evaluations. USAID's new central evaluation office collaborated with State's Office of the Director of Foreign Assistance (State/F) to develop evaluation guidelines and standards that were placed on the United States Foreign Assistance Web site in March 2009. These standards and guidance elaborate on the official USAID and FFP guidance, and they specifically address some elements that are not directly addressed in the official guidance, most notably the standards for quality. Collectively, the FFP- specific, USAID-wide, and Foreign Assistance policies and procedures are generally consistent with the AEA principles. FFP Requires External Evaluations to Help Ensure Independence: [Sidebar: Independence: Principle: Evaluation managers should be independent with respect to the design, conduct, and results of their evaluation studies. Source: AEA, An Evaluation Roadmap for a More Effective Government. [End of sidebar] To help ensure independence as outlined in the AEA principle, FFP requires that each multiyear assistance program's final evaluation be performed by external evaluators. It also relies on external entities to perform cross-cutting evaluations using the Food for Peace Act and other funding sources.[Footnote 30] Independence in the context of final evaluations for FFP multiyear assistance programs means that the key evaluation experts, the team leader, and sectoral leads, should be external consultants hired by the project. Our review of the 16 final evaluations that we found on USAID's Development Experience Clearinghouse (DEC) Web site determined that external consultants were generally listed as the prime authors.[Footnote 31] In some instances, the external evaluators were affiliated with a particular consulting firm and, in other instances, several external evaluators formed a team to perform the final evaluation. In addition, the cross-cutting evaluations and other FFP-funded studies are to be conducted by FANTA-2 and external consultants. FANTA-2 is part of AED, a nonprofit organization that cites integrity as one of its core values.[Footnote 32] Our review of selected task orders, signed and made available by FANTA-2, found them in line with the AEA principle for independence. For example, not only did its task order for a subrecipient of a new 3-year cross-cutting evaluation describe the project's phases, activities, and methodologies, but it also clearly specified that the subrecipient's staff would develop preliminary conclusions based on the collected data, draft interim reports, and submit a final report to USAID.[Footnote 33] USAID's Actions Were Partially Consistent with the AEA Principles for Scope and Coverage, Dissemination of Results, Professional Competence, and Resources: USAID Requires Few Cross-Cutting Evaluations: [Sidebar: Scope and Coverage: Principles: * Evaluation should span the life cycle of programs and policies, making it integral to planning, developing, and managing government programs at all stages of their development. * In general, federal programs and policies should be subject to evaluation, which is a responsible and necessary part of good management. Source: AEA, An Evaluation Roadmap for a More Effective Government. End of sidebar] In line with the AEA principles, FFP requires that all of its MYAPs be monitored and evaluated at all stages of their life cycle--its guidance specifies the activities that are required from proposal writing onward. The implementing partners must: * perform a baseline survey at the beginning of each MYAP; * submit annual reports, including annual results reports that present performance indicators and a narrative regarding progress made; * submit annual questionnaires that respond to questions about FFP's food security indicators; and: * submit midterm and final project evaluations. In addition, USAID project officers must review the evaluation plans and final reports. Finally, the final evaluation must include questionnaires that are comparable to the baseline surveys, along with information about sample design, universe, methods of analysis, and sampling frame. During our fieldwork in Bangladesh and Haiti, we saw several examples of FFP's implementing partners following these directives. For example, we visited a project in Bangladesh that had collected baseline data on a wide range of indicators that measured aspects of land ownership, height and weight-for-age measures, maternal and child health practices, access to water and sanitation, household food security, and disaster preparedness. In addition, we visited a project in Haiti that had produced a quarterly report including targets for more than 50 measures, along with data on whether or not those targets had been achieved, and reasons why some targets had not been met. FFP's required monitoring and evaluation of the multiyear programs focus on whether the programs are achieving their expected outcomes-- but they do not assess program impacts, or the effectiveness and efficiency, of particular approaches. To achieve this, FFP would need to conduct periodic, cross-cutting evaluations that compare what actually happened with what would have happened in the absence of the program or activity.[Footnote 34] While FFP has commissioned some of these studies, according to USAID officials, it has no requirements to do so[Footnote 35] and has conducted only a relatively small number over the last 5 years.[Footnote 36] In contrast, FFP required every one of more than 100 multiyear projects concluded over the last 5 years to be evaluated for outcomes. Implementing Partner Evaluation Results Are Not Routinely and Widely Disseminated: [Sidebar: Dissemination of Results: Principles: * The results of all evaluations related to public accountability should be made available publicly and in a timely manner. * Evaluations should be easily accessible through the Internet with user friendly search and retrieval technologies. * Evaluations of promising and effective program practices should be systematically and broadly disseminated to potential users. Source: AEA, An Evaluation Roadmap for a More Effective Government. End of sidebar] Although USAID policy requires final evaluations of its nonemergency food aid programs to be posted on a special Web site known as DEC, the implementing partners responsible for these evaluations often do not adhere to this requirement. The DEC Web site is accessible to the general public and contains both simple and more advanced search engines. For example, a user interested in Food for Peace nonemergency programs could perform a search for "agriculture," "maternal and child health," or "food security" final evaluations conducted in recent years. A user could also search for final evaluations sponsored by FFP. However, FFP officials told us that the MYAP implementing partners often do not submit their final evaluations to DEC, even though this is a program requirement. According to these officials, there is no penalty if the implementing partners do not submit their reports to DEC. We were unable to find in DEC a final evaluation of an integrated food security program that FFP staff in Bangladesh provided to us in hard copy. In addition, more than 100 FFP-sponsored final evaluations of multiyear programs should have been conducted from 2004 to 2009, according to FFP staff, but our search in DEC found only 16. Professional Competence Varies among the Different Groups of Staff Who Monitor and Evaluate FFP Programs: [Sidebar: Professional Competence: Principles: * Evaluations should be performed by professionals with evaluation training and experience appropriate to the evaluation activity in which they are engaged. * Federal agencies should be encouraged to recognize the multidisciplinary nature of evaluation and assure that the diversity of disciplines is appropriately represented in both internal and independent consulting teams. Source: AEA, An Evaluation Roadmap for a More Effective Government. End of sidebar] Although the entities that USAID has selected to perform cross-cutting evaluations have staff with appropriate professional training and experience, questions have been raised about the technical skills of some staff that perform monitoring and evaluation for the implementing partners. As FFP does not have reliable data on the number of its field staff that perform monitoring and evaluation functions, and the specific skills and competencies of those staff, it is not possible to determine their overall competence. In addition, as FFP's ability to hire monitoring and evaluation specialists in Washington, D.C., is limited, and as USAID has only recently reestablished its central evaluation unit, there has been an absence of skilled evaluators in headquarters that could assist FFP's officers in the field and implementing partners' staff. FFP's Partners Have Training and Experience Consistent with the AEA Principles: The partner organizations that FFP relies on to provide technical assistance and support and to conduct cross-cutting evaluations have training and expertise that are consistent with the AEA principles for professional competence. The AED FANTA-2 project, with which USAID has a cooperative agreement for various types of technical assistance, employs staff experienced in monitoring and evaluation, with backgrounds in economics, sociology, nutrition, anthropology, public health, and epidemiology. In addition, FANTA-2 has partnered with universities and research institutes, such as Tufts University and the International Food Policy Research Institute, for particular evaluations and other research projects. Concerns Have Been Raised about Certain Aspects of Implementing Partners' Monitoring and Evaluation Skills: FFP's implementing partners' staff perform important monitoring and evaluation functions, such as gathering data on performance indicators and tracking these indicators. Yet FFP officials in Washington, D.C., and in the field, as well as some external evaluators, have expressed concerns about the competency of these staff to perform certain aspects of monitoring and evaluation, as follows: * Staff can lack key evaluation skills. Implementing partners' staff can lack key evaluation skills, according to an AED employee who provided training and support to FFP's implementing partners and several external evaluators. According to the AED employee, implementing partners' staff could execute research designs adequately but often were not capable of designing the research themselves, or of taking responsibility for key aspects of data collection, such as developing surveys or selecting statistical samples. FFP officials stated that they recognize the need to build capacity among the implementing partners, and are providing monitoring and evaluation training in the field for these FFP and implementing partner staff. * Difficulties in hiring and retaining qualified staff. Adequacy of staffing levels and staff turnover have been cited as a concern in some countries. For example, the implementing partners in Haiti told us that it was difficult to attract and retain technically qualified staff. They stated that Haiti has only a limited supply of people with the appropriate skills and training for monitoring and evaluation. To address the shortage, one implementing partner recently hired a new monitoring and evaluation program officer from an African country who had appropriate methodological skills but did not speak either French or Creole, Haiti's official languages, at the time of his arrival in- country. FFP Lacks Data on the Monitoring and Evaluation Competencies of Its Field Staff: FFP does not have the data necessary to assure that it has enough staff with the right skills. According to FFP officials, USAID does not collect an inventory of its staff's skills and abilities,[Footnote 37] which would help the agency to determine whether it has staff with sufficient training and skills in monitoring and evaluation. Furthermore, neither USAID nor FFP have data that can indicate which staff spend the majority of their time on monitoring and evaluation, or how much staff time is spent on various monitoring and evaluation activities, such as checking on warehouses, advising implementing partners on performance indicators, or participating in evaluation designs. While FFP attempted to gather some data for us on its field staff who perform monitoring and evaluation, these data were not reliable. In Haiti, for example, FFP officials initially told us that nine staff acted as field monitors. However, we found that this number included staff members who did not spend much of their time involved in routine monitoring and evaluation, such as the FFP officer, his deputy, and two secretaries. Of the FFP staff in country, only four appeared to spend substantial parts of their time on specific monitoring activities, such as performing routine checks of warehouses or health clinics and giving implementing partners advice on indicators. One reason that it is difficult for FFP to determine which staff spend the majority of their time on monitoring is that definitions of this function vary. For example, while a recent FFP bulletin defined monitoring as the regular tracking of program implementation, FFP guidance to field staff defined monitoring more broadly, to include activities such as reviewing contracts and ensuring that financial audits are performed.[Footnote 38] If the broader definition of monitoring is used, more staff members can be classified as performing this function, but then it is not possible to determine which of those staff are spending the majority of their time on monitoring program inputs and program implementation. FFP Lacks Staff in Headquarters Who Are Skilled in Monitoring and Evaluation: AEA has recommended that to support evaluations agencies either establish centers of evaluation or appoint evaluation coordinators for individual programs that conduct their own evaluation. According to FFP officials, FFP has been unable to hire monitoring and evaluation staff at headquarters to assist and support its officers in the field. FFP officials attributed this to restrictions in the Food for Peace Act, which precludes them from hiring any staff in Washington, D.C., to monitor and evaluate nonemergency food aid programs.[Footnote 39] Moreover, as USAID has only recently reestablished its central evaluation unit,[Footnote 40] FFP has not been able to seek advice and guidance from professional, experienced evaluators at headquarters, or to ask them to perform evaluations of FFP's programs. Greater Resources Are Now Available for Monitoring and Evaluation, but Their Sufficiency Is Difficult to Determine Absent a Needs Assessment: [Sidebar: Resources: Principle: Sufficient resources should be made available for evaluation, including stable annually recurring sources of funds and special one-time funds for evaluation projects of interest to executive branch and congressional policymakers. Source: AEA, An Evaluation Roadmap for a More Effective Government. End of sidebar] The Food for Peace Act provides FFP with a stable source of new funding of up to $22 million annually for each of the fiscal years 2009 through 2012 to improve, monitor, and evaluate the effectiveness and efficiency of nonemergency food aid programs. In addition, FFP has access to other funding streams, such as DA funds, which have recently increased. As a result, FFP officials told us that they currently have more resources to devote to monitoring and evaluation than in the past. Although availability of the newly authorized funding for monitoring and evaluation as well as funding from other sources is generally consistent with the AEA principle on resources, it is not possible to determine if funding is sufficient because FFP has not systematically determined its research needs, prioritized those needs, or estimated the costs associated with them. In the course of our audit work, we identified several areas where the current resource levels might not be sufficient. For example, experts we interviewed provided examples of potential evaluations or studies that could help improve FFP's programs. However, according to USAID officials, FFP has not undertaken the formal exercise of compiling a list of these studies, or prioritizing its needs for them. In addition, FANTA-2 staff who review many of the implementing partners' multiyear plans for evaluation raised concerns that FFP does not sufficiently ensure that these evaluations are adequately funded. However, FFP has not reviewed or estimated the amounts its implementing partners are spending on monitoring and evaluating their multiyear projects because, according to FFP, that would be burdensome as it would require a review of each project's contract.[Footnote 41] Without such an assessment, it is not possible to determine whether sufficient resources are being spent on the required final evaluations. USAID's Actions Were Not Consistent with the AEA Principles for Monitoring and Evaluation Plans: [Side bar: Evaluation Plans: Principles: * Agency should require its major program components to prepare annual and multiyear evaluation plans of the studies and other evaluation activities that it will undertake. The plans should be updated annually. * The planning should take into account the needs of evaluation results for informing program budgeting, reauthorization, agency strategic plans, ongoing program development, and management and responses to critical issues. Source: AEA, An Evaluation Roadmap for a More Effective Government. End of sidebar] According to FFP officials, FFP plans some of its monitoring and evaluation activities,[Footnote 42] but lacks an overall plan that integrates its monitoring and evaluation with program management. FFP officials reported that they make decisions about monitoring and evaluation based on discussion and a thorough consideration of options but have not undertaken the rigorous planning process that is recommended by AEA. Therefore, FFP does not have an integrated plan for all of its oversight activities to facilitate strategic planning, budgeting, and program management, and to indicate how it will use monitoring and evaluation results to improve program management. Moreover, FFP does not have a mechanism to link resources to needs based on risk, according to USAID officials. As a result, when FFP allocates resources for monitoring and evaluation, it does not take into account factors such as the country context, vulnerability to fraud, waste, and abuse, program efficiency, and critical issues. As we noted earlier, we cannot determine whether FFP has sufficient resources because this type of planning has not been conducted, which means that FFP's monitoring and evaluation needs have not been prioritized and compared against available resources. A senior FFP official told us that he could see the potential value of such a plan but wondered whether FFP--which has a staff of about 60 people and which he characterized as being small in relation to other USAID units--could support such an effort. However, FFP appears to have the resources for such a planning effort, as the Food for Peace Act has provided it with up to $88 million to improve, monitor, and evaluate the effectiveness and efficiency of nonemergency food aid programs over a 4-year period, and has access to other resources that can be used for monitoring and evaluation, such as DA funds.[Footnote 43] Conclusions: Nonemergency food aid programs are an essential component of the U.S. strategy to reduce global food insecurity, and Congress has authorized new funding for USAID to improve its oversight of these programs. In response, USAID's FFP has initiated actions over the past year to strengthen its monitoring and evaluation of food assistance. However, weak planning could impede these efforts, including the upgrade of FFP's information technology system. In addition, USAID has yet to determine how it intends to assure stable resources, within existing authorities, to fully implement all the actions planned by FFP to improve monitoring and evaluation of nonemergency food aid. As a result, planned actions that are essential to effective program management, such as the expansion of a computerized monitoring system for food aid programs and cross-cutting evaluations, could be jeopardized. Finally, FFP lacks an integrated and comprehensive monitoring and evaluation plan that links resources to needs based on risk. In the absence of such a plan, it is difficult to determine whether USAID is allocating resources for monitoring and evaluation to its priority needs, such as program efficiency, critical issues, and programs vulnerable to fraud, waste, and abuse. Recommendations for Executive Action: To facilitate the implementation of ongoing efforts to improve monitoring and evaluation and ensure effective program management of nonemergency food aid, we recommend that the Administrator of USAID take the following two actions: * Develop a concept of operations document to help reduce the risks associated with upgrading FFP's information technology system. Such a document should adhere to industry best practices and include key elements such as major system components, interfaces to external systems, and performance characteristics. It would also describe system characteristics for a proposed system from a user's point of view and include high-level descriptions of information systems, their interrelationships, and information flows. * Develop an integrated monitoring and evaluation plan to ensure that the results of its oversight activities are used for effective management of nonemergency food aid programs. Such a plan should, among other things, (1) link monitoring and evaluation to key USAID and FFP goals; (2) establish a systematic process for determining appropriate budget levels and staff resources for monitoring and evaluation based on an analysis of the nature, scope, and size of the programs, and the unique conditions presented, in the recipient countries; (3) examine all available options for funding monitoring and evaluation at headquarters and in recipient countries; and (4) establish time frames for implementing and evaluating the plan. Agency Comments and Our Evaluation: USAID provided written comments on a draft of this report. We have reprinted these comments in appendix IV, along with our responses. Additionally, USAID provided technical comments on a draft of this report, which we have addressed as appropriate. USAID concurred with our first recommendation to develop a concept of operations document to help reduce the risks associated with upgrading FFP's information technology system. USAID disagreed with the second recommendation, asserting that we did not adequately take into account documents that detail its planning processes. We considered all the documents that USAID cited and found that they did not constitute plans, but only provided guidance for planning. Furthermore, the weaknesses we identified in FFP's monitoring and evaluation efforts, including in the areas of resources and professional competence, demonstrate the need for a comprehensive monitoring and evaluation plan. USAID acknowledged that it does not have an integrated monitoring and evaluation plan, but said that it does not see the benefits of developing it since the elements of such a plan exist in some form already. USAID noted that we did not mention FFP's Strategic Plan for 2006-2010 and a related document, which link monitoring and evaluation to agency goals and lay out performance indicators. We considered the strategic plan irrelevant to our analysis because it was prepared in 2005, prior to the passage of the 2008 Food for Peace Act, and covers only some aspects of monitoring. USAID also stated that its planned studies form part of an agency agenda to inform program management. We reported on FFP's planned studies but noted that, because funding for these studies is not assured due to legal restrictions, USAID may need to find alternative funding sources. Moreover, USAID officials reported that they did not use almost $9 million of last year's available funds authorized by the 2008 Food for Peace Act for oversight of nonemergency food aid. We consider these to be compelling reasons for USAID to develop an integrated monitoring and evaluation plan. In terms of staffing, FFP officials said that their first priority is to place staff in all the countries where they have nonemergency food aid programs. FFP acknowledged that it does not have data on field staff that perform monitoring and evaluation functions, but said that such data would be costly to collect, and that we had not made the case for collecting them. However, we found that FFP currently does not know what percentage of time its staff spend on monitoring and evaluation and what specific monitoring and evaluation functions they perform. We believe that because FFP had a substantial new funding authority for oversight activities, it should invest in a survey of staff skills and an assessment of monitoring and evaluation needs, which would provide FFP with essential data for future staffing decisions. We believe that this basis for our recommendation for an integrated monitoring and evaluation plan remains valid. We are sending copies of this report to interested Members of Congress and the Administrator of USAID. The report is also available at no charge on the GAO Web site at [hyperlink, http://www.gao.gov]. If you or your staffs have any questions about this report, please contact me at (202) 512-9601 or melitot@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix V. Signed by: Thomas Melito: Director, International Affairs and Trade: [End of section] Appendix I: Objectives, Scope, and Methodology: In passing the Food for Peace Act in 2008, Congress required the Comptroller General of the United States to review the U.S. Agency for International Development's (USAID) Report to Congress on the agency's efforts to strengthen the monitoring and evaluation of its nonemergency food aid programs and provide recommendations for improvement. To address this congressional mandate, we reviewed (1) USAID's plans and actions to improve its monitoring and evaluation of nonemergency food aid programs provided under the Food for Peace Act and (2) the extent to which USAID has integrated its monitoring and evaluation of nonemergency food aid with program management, particularly with regard to established principles. In this review, we focused on oversight practices of USAID's Office of Food for Peace (FFP), not of the agency as a whole. To address USAID's plans to improve the monitoring and evaluation of nonemergency food aid programs using the funds provided under the Food for Peace Act, we reviewed the list of activities that the agency reported to Congress in December 2008. This included plans to make improvements in: * the number of field staff monitoring food aid programs, * computerized systems for monitoring food aid programs, * the market analysis required before food aid programs are approved in recipient countries, * cross-cutting evaluations of nonemergency food aid projects, * FFP's information technology system, and: * the Famine Early Warning Systems Network (or FEWS NET). We interviewed knowledgeable USAID officials and partners responsible for implementing these plans and reviewed relevant documents, including contracts and plans. We also examined FFP data on the allocations of the Food for Peace Act funding for oversight of nonemergency food aid in fiscal year 2009. Based on interviews, and our analysis of the available documents, we determined that the data were sufficiently reliable to indicate the relative amounts allocated to each of the oversight activities, and to demonstrate that almost $9 million were not used in fiscal year 2009. In addition, we conducted fieldwork in Bangladesh and Haiti. We selected these two countries because they have the largest Food for Peace nonemergency food aid programs, but they differ in their approaches to monitoring and evaluation. Haiti has more staff working on these programs than does Bangladesh and uses the Layers computerized monitoring system, which Bangladesh does not. Haiti's staff spend relatively more time monitoring programs directly, while FFP in Bangladesh relies more on contract officers and implementing partners to perform its monitoring and evaluation. In addition, Haiti is in the Caribbean while Bangladesh is in South Asia, which provided a regional contrast. While we could not generalize from these two countries to all the countries that have Food for Peace programs, we gained useful insights into monitoring and evaluation from our fieldwork. To consider the extent to which USAID's monitoring and evaluation of nonemergency food aid are aligned with program management, we selected the American Evaluation Association's (AEA) principles for integrating evaluation into program management. We selected these principles as criteria because they are consistent with the goal set by the Secretary of State, who has ultimate responsibility for USAID, to ensure evidence- based decisions in development programs. In this review, we combined the AEA standards for scope and for coverage because we found overlap between them. In addition, we did not assess one of the AEA standards on methods because that would have involved a review of studies that was beyond the scope of this report, and because that principle was less specific than those for the other elements, essentially noting that a mix of methods should be used, but that the actual methods used should depend upon the topic to be evaluated. The AEA principles differ in length and detail, with some including several parts and clauses, while others are more succinct. For the more detailed principles, we identified the sentences that summarized them at the highest level and used those as our operating criteria. For the sake of transparency, we published the exact principles we used in our report next to our discussion of them. To gather evidence to apply against the principles, we interviewed cognizant FFP officials in Washington, D.C., and in Bangladesh and Haiti. We also interviewed the Chief of USAID's Evaluation Office, senior staff at the Food and Nutrition Technical Assistance (FANTA-2) project of the Academy for Educational Development (AED), with which USAID has ongoing cooperative agreement, as well as experts knowledgeable about USAID's evaluation and monitoring. We reviewed USAID and FFP policy guidance and bulletins that discuss and describe evaluation. We reviewed the evaluations that we found on USAID's Development Experience Clearinghouse (DEC) Web site and that were given to us by USAID and its implementing partners. We also examined related monitoring and evaluation documents, such as performance plans and annual project reports, and documentation of the training USAID provides on monitoring and evaluation. We reviewed this evidence and compared it against each of the AEA principles to make a general, high-level determination on the broad degree to which FFP was consistent or inconsistent with them. In the course of this review, we recognized that FFP has its own guidance and practices, but that it follows the overall guidance and practices set by USAID for the agency as a while, and considered each source as appropriate. In addition to the AEA principles, we also considered GAO standards and prior GAO work, particularly on workforce planning; USAID guidance; and guidelines that the Institute of Electrical and Electronics Engineers (IEEE) has established for information technology. As USAID performs a wide range of activities to monitor and evaluate its Food for Peace program, we defined evaluation broadly for the purposes of this report. This definition includes evaluating inputs and implementation,[Footnote 44] outcomes, and impacts. We also considered other research studies that FFP had conducted to help manage its programs, such as prospective studies on the impacts of providing particular food commodities to particular markets, and analyses of individual countries' food strategies. In addition, we included the selection and tracking of performance indicators in our review because FFP and USAID define monitoring to include the tracking of these indicators during the life cycle of their programs, and use them in the final evaluations. We conducted this performance audit from March 2009 to September 2009, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of section] Appendix II: USAID Report to Congress: Note: This is USAID's Report to Congress dated December 31, 2008, as required by the Food for Peace Act. USAID: From The American People: December 31, 2008: The Honorable Tom Harkin: Chairman: Committee on Agriculture, Nutrition, and Forestry: U.S. Senate: Washington, DC 20510: Dear Mr. Chairman: Pursuant to Section 3012(f)(3) of the Food for Peace Act (Public Law 480) dated June 18, 2008, we are pleased to submit the U.S. Agency for International Development's (USAID) report on efforts undertaken by the Administrator to conduct oversight of non-emergency food aid programs. If you have any questions concerning this report, please feel free to contact me. Sincerely, Signed by: [Illegible], for: Jeffrey J. Grieco: Assistant Administrator Bureau for Legislative and Public Affairs: Enclosure: a/s: Clearance Page For Letters to Congressmen Peterson, Harkin, Chambliss, Goodlatte, and Berman and Congresswoman Ros-Lehtinen, Regarding Report of Non-Emergency Food Aid Programs: Clearances: DCHA/FFP: Jeff Boms; cleared by email: Date 12/15/08. DCHA/FFP: Jonathan Dworken; cleared by email: Date 12/15/08. DCHA/PP Susan Bradley [Signed by]: Date 12/15/08. AA/DCHA: Rob Jenkins [Signed by]: Date 12/15/08. LPA: Rick Marshall [Signed by]: Date 12/15/08. DCHA/FFP: Angela Stephens: AS: x2-0414: 12/15/2008 P:\DCHA.SHARED\2009 Correspondence\FFP\Rpt to Congress re program oversight\FFP Program Oversight Report to Congress FINAL.doc. Report To Congress: Sec. 3012 of the Food, Conservation, and Energy Act of 2008 requires that not later than 180 days after the date of enactment of the Act, the Administrator of the U.S. Agency for International Development (USAID) submit to the appropriate committees of Congress a report on efforts undertaken by the Administrator to conduct oversight of non- emergency programs under Title II of P.L. 480, the Food for Peace Act. USAID is conducting oversight of its non-emergency food aid programs through: (1) a significant expansion of field monitors; (2) establishment of a three-year pilot project, the Bellmon Estimation for Title 11 (BEST) Project, which will conduct independent market analyses to ensure non-emergency program compliance with the Bellmon Amendment; (3) expansion of the Food and Nutrition Technical Assistance (FANTA) Project research to strengthen monitoring and evaluation for non- emergency programs; (4) improving the quality of assessments and targeting approaches; and (5) upgrading the Office of Food for Peace (FFP) information technology system. 1) Increase USAID Staff that Manage and Monitor Non-emergency Food Aid: Before the enactment of the 2008-2012 Farm Bill, FFP was precluded from using P.L. 480 Title II funds to hire personal services contractors (PSCs) primarily responsible for monitoring non-emergency programs. With this new authority, FFP has undertaken a thorough review of existing management and monitoring arrangements, and is expanding monitoring in approximately 10 countries with Title II non-emergency programs. Positions have already been advertised in three of the countries. 2) Bellmon Estimation for Title II (BEST) Project: Prior to the approval of proposed food aid levels for monetization and distribution programs, P.L. 480 Title II (Section 403(A) A-D of P.L. 480, as amended) requires that the United States Government (USG) make a positive Bellmon determination to ensure a) adequate storage faculties will be available in the recipient country at the time of the arrival of the commodities to prevent the spoilage or waste of the commodity, and b) the distribution and monetization of the commodity in the recipient country will not result in a substantial disincentive or interference with the domestic production or marketing in that country. In order to help USAID comply with the Bellmon Amendment and to make the Bellmon determination process more rigorous and consistent, FFP has established as of fiscal year 2009 a three-year pilot project, the Bellmon Estimation for Title II (BEST) Project, to conduct independent market analyses. Based on an established schedule, BEST will conduct an initial country situational study to develop a baseline market report (BMR). The BMR will include, but not be limited to, the following information: * Market and production system profile, summary; * Historical overview of the country's food aid program, covering USAID, USDA, WFP and other development, emergency and monetization food aid programs; * Distribution/logistics/storage profile; * Policy issues relevant to agriculture, markets and trade; and; * Product selection and analysis for key food security crops. In November 2008, USAID notified its Missions and FFP cooperating sponsors (CSs) about the BEST Project and CSs' continued obligation to contribute to market analyses and surveillance, and provided opportunity for them to comment on this new approach to compliance with the Bellmon Amendment. 3) Food and Nutrition Technical Assistance (FANTA) Project FFP research activities, through FANTA, will include: * Food Security Programming Strategies: A solid understanding of national food security conditions is a crucial prerequisite to sound programming. In response to demand, FFP, through the FANTA Project, will support Missions and governments to plan and implement in-depth food security assessments as they relate to the programming of Title II non-emergency resources. Working with Missions, governments, and other stakeholders, FFP, through the FANTA Project, will apply assessment results to prepare food security strategies that define objectives, approaches, and institutional partnerships for effective use of Title II non-emergency resources to reduce food insecurity. * Exit Strategies: Factors that lead to success in sustaining program benefits after Title II non-emergency food assistance has ended will be examined, with the goal of deriving recommendations for effective Title II non-emergency exit strategies. * Preventive Malnutrition for Children Under Two Approach (PM2A): This study will identify key components of success in programs that utilize the PM2A approach, in order to improve the cost effectiveness, design and content of such programs. This study will include training and supporting CSs to adapt, design and implement PM2A, supporting formative research and baseline surveys/analyses, and performing operations research. * Layers: FFP monitoring of Title II non-emergency programs is an important mechanism for ensuring quality. FFP will use Layers, a computerized system for monitoring food aid programs, to improve monitoring speed and efficiency and the use of information. The innovative Layers approach is a computerized food aid program monitoring system (hardware/software solution) using handheld computers (PDAs). Layers helps USAID missions collect program data to assess the quality of implementing partner operations in the field. A wide variety of issues can he evaluated, from the conditions of storage in commodity warehouses to the quality of services delivered by Title II CSs in their maternal child and health nutrition activities, food for work, food for education, or other interventions. * Monitoring and Evaluation (M&E) Training: FFP will help CSs harmonize impact and output indicators, implement joint baseline assessments and coordinate other components of M&E systems as relates to Title II nonemergency programs. For example, in November 2008, two regional workshops in Africa were held for CS staff overseeing Title II non- emergency programs. The workshops aimed to support CSs implementing new Title II programs by examining their results framework, harmonizing indicators, improving the usefulness and overall design of their M&E system, and reflecting on the design and planning of their baseline studies. Further M&E workshops arc planned. 4) Improve the Quality of Assessments and Targeting Approaches: For more than 20 years, FFP has provided separate cooperative agreements in support of capacity building and technical assistance to Title II CSs. These agreements have addressed various aspects of Title II program management through workshops, training, assessments and specialty tools, along with support to key personnel. These achievements have been captured throughout the years. Under the new program oversight authority in the Farm Bill, FFP is now focused on the next phase of support to increase the effectiveness and efficiency of Title II food aid programs through the broader establishment and adoption of standard best practices across CSs. The principal objectives of the proposed agreement will be to identify best practices in Title 11 food aid program management and capacity building; design and undertake training in best practices in program management and capacity building; communicate and disseminate information on best practices in program management and capacity building; and strengthen the coordination and collaboration among current and prospective Title II CSs. In this regard, FFP has already issued a Request for Information (pursuant to FAR 15.201) to seek information from potential grantees or participants as to their interest, priorities and capacities to participate in or undertake a Leader/Associates Award, i.e., cooperative agreement, to increase the effectiveness and efficiency of Title II food aid programs. 5) Food for Peace Information Technology System: The new Farm Bill provides USAID with the flexibility to use P.L. 480 Title II resources to upgrade FFP's information technology systems in order to improve tracking, reporting and evaluation capabilities, including non-emergency programs. The new system will involve collaboration and testing with users in each business area in which the system is used-budgeting, proposal, document management, commodity purchase, commodity booking and shipping, tracking, monitoring and evaluation, reporting, closeout and interface. FFP is in the process of reviewing the functional components of the architectural design to ensure that they align with these requirements. It is expected that an award for the implementation of the new system will he made in early 2009. [End of section] Appendix III Examples of Activities Supported with Nonemergency Food Aid: Figure 5 provides examples of activities we observed in Bangladesh and Haiti that are illustrative of the various activities supported with nonemergency food aid. Figure 5: Examples of Activities Supported with Nonemergency Food Aid: [Refer to PDF for image: 4 photographs] Direct distribution: Food provided in exchange for work; generally targeted seasonally and geographically to food insecure population groups. Photograph: Food-for-work: CRS Haiti provided temporary employment to 234 laborers for 6 months for soil conservation and reforestation activities. Photograph: Food-for-work: For several years, CARE Bangladesh has been helping the Tangail district prepare for natural disasters through infrastructure activities, including raising homesteads to mitigate the effects of severe flooding on households. Monetized food aid: Proceeds from U.S. food aid sold locally support development projects, such as the rehabilitation and construction of new water points and mother and child health and nutrition programs. Photograph: Safe water and sanitation: Save the Children Bangladesh has installed a water point in the Barisal district to ensure household access to safe water. Photograph: Mother and child health and nutrition: World Vision Haiti provides pregnant and lactating women and children under the age of 2 with monthly health checkups, immunizations, and counseling, at the Terre Casse clinic in Haiti’s Central Plateau region. Sources: GAO analysis; GAO and USAID (photos). [End of figure] [End of section] Appendix IV: Comments from USAID: Note: GAO comments supplementing those in the report text appear at the end of this appendix. USAID: From The American People: U.S. Agency for International Development: 1300 Pennsylvania Avenue, NW: Washington, DC 20523: [hyperlink, http://www.usaid.gov] September 12, 2009: Mr. Thomas Melito: Director: International Affairs and Trade: U.S. Government Accountability Office: Washington, DC 20548: Dear Mr. Melito: I am pleased to provide the formal response of the U.S. Agency for International Development (USAID) to the GAO draft report "International Food Assistance: USAID is Taking Actions to Improve Monitoring and Evaluation of Nonemergency Food Aid, but Weaknesses in Planning Could Impede Efforts" (GAO-09-980). The enclosed USAID comments are provided for incorporation with this letter as an appendix to the final report. Thank you for the opportunity to respond to the GAO final report and for the courtesies extended by your staff in the conduct of this audit review. Sincerely, Signed by: Drew W. Luten: Acting Assistant Administrator: Bureau for Management: Enclosure: a/s: [End of letter] USAID Comments On GAO Draft Report 09-980: USAID's Office of Food for Peace (FFP) takes seriously its obligation to the federal government and the American people to ensure that Title II resources are used effectively and efficiently to achieve the best possible food security outcomes and that food aid programs continually learn from past experiences and improve their implementation. The GAO recommends that the USAID Administrator (1) prepare a concept of operations document (CONOPS) to reduce the risks associated with upgrading FFP's information technology system and (2) develop an integrated monitoring and evaluation plan that is linked to program goals, establishes a systematic process for determining budget and staffing levels, examines funding options, and sets implementation time frames. Related to recommendation (1) that FFP develop a CONOPS to reduce the risks associated with upgrading FFP's information technology system, under a current contract, FFP will receive a CONOPS that will be part of the recommendations deliverable that will be produced at the completion of the IT functional requirements gathering process. Through the functional requirements gathering, the process currently entails reviewing all of the Title 1 day-to-day activities which are carried out to meet mission needs, and assessing the interrelationships of all applicable information systems, and developing high-level descriptions for all business processes and workflow. Through the understanding and update of the current state of operations, redundant and manual processes can be identified and eliminated to provide a more efficient and effective process which then can be implemented through the new information system solution. The CONOPS will be utilized for the implementation phase of the system's development lifecycle. Related to recommendation (2), the report does not adequately take into account existing and on-going M&E processes currently carried out by USAID and FFP. As recognized in published FFP guidance, and as outlined in USAID's Automated Directives System (ADS), USAID and FFP already describe and outline key monitoring and evaluation (M&E) responsibilities of Title II Multi-Year Assistance Program (MYAP) awardees and potential awardees, providing additional detail to the existing policies, including, but not limited to the following: [See comment 1] 1. ADS 200, Introduction to Program Policy, which provides an overview to the ADS Series 200. 2. ADS 201, Planning, which explains how USAID manages for long-term results and development impact, retaining rigorous internal management and documentation standards while collaborating with Department of State/Office of the Director of Foreign Assistance and other U.S. Government agencies on implementing a new foreign assistance policy agenda and reporting system; it also summarizes the Foreign Assistance Framework and its standardized program structure. 3. ADS 202, Achieving, which outlines the basic policy directives and required procedures used in achieving foreign assistance results with USAID-managed program and staff resources to help ensure that the development results that USAID seeks are obtained in a cost-effective and timely manner, consistent with applicable regulatory and accountability requirements. 4. ADS 203, Assessing and Learning, which describes USAID practices and standards used to determine how well assistance objectives are achieving their intended results. Potential awardees are requested to consult ADS 203 closely for important information regarding the design of M&E plans. 5. Evaluating Title II Development-oriented Multi-Year Assistance Projects at [hyperlink, http://www.fantaproject.org/downloads/pdfs/TN11_MYAP.pdf]. 6. Monitoring and Evaluation Framework for Title II Development- oriented Projects at [hyperlink, http://www.fantaproject.org/downloads/pdfs/TN10_MEFramework.pdf]. 7. Title 11 Evaluation Scopes of Work at [hyperlink, http://www.fantaproject.org/publications/tn2.shtml]. 8. (M&E) responsibilities of Title II Multi-Year Assistance Program awardees and potential awardees at [hyperlink, http://www.usaid.gov/our_work/humanitarian_assistance/ffp/ffpib.html]. In addition, in fiscal year (FY) 2008, FFP delivered 4-5 day M&E workshops to newly awarded MYAPs in order to improve the quality of Title II M&E data. The workshops help ensure that required indicators from FFP and USAID Missions are included in partner M&E plans and that data are collected in a consistent manner. In FY 2008, workshops were held in Mali and in Mozambique for 16 newly awarded MYAPs who improved their results frameworks, M&E plans, and the design of their baseline studies as a result of attending. USAID Mission staff who attended the workshops presented Mission indicators and had an opportunity to learn about the new Title II non-emergency programs in their country and about FFP M&E. Workshops have also been held in Ethiopia, Madagascar and Malawi Further M&E workshops are planned. The GAO recommends that USAID develop an integrated M&E plan that addresses four elements. All of those elements exist in some form already, and there is no evidence that there will be sufficient added value to integrating them all in a single, hybrid document that would contain a mix of a set of indicators, an evaluation plan, and a plan for the administration of monitoring and evaluation focused mostly on how to fund and staff M&E. For example, the existing FFP Strategic Plan and performance monitoring plan (PMP), which is not mentioned at all in the report, already links M&E to key USAID and FFP goals. The Strategic Plan discusses how FFP objectives fit into USAID goals and the PMP shows how FFP's indicators are linked to its objectives. The PMP lays out a set of indicators designed to measure the FFP objectives, explains the reasoning behind the selection of each indicator, and provides detailed information on indicator description, data collection methods, baselines and targets, and data quality assessment plans. [See comment 2] FFP's current evaluation plan contains a set of multi-year studies prioritized because they 1) build on previous investments (e.g. preventive malnutrition for under two[Footnote 45]), 2) are needed to operationalize strategic approaches under the Strategic Plan (e.g. FFP Trigger Indicators and Early Warning and Response Systems in Multi-Year Title II Assistance Programs Document[Footnote 464]), 3) take advantage of unique and time-bound opportunities (e.g. FFP is strengthening guidance for exit strategies working closely with Tufts University through FANTA-2), and 4) will form part of the analytic agenda to inform program management and the new strategy (e.g. FFP is now conducting a qualitative review of Title II non-emergency program activities). [See comment 3] In addition, referred to as Technical and Operational Performance Support (TOPS), FFP will soon provide a single award with the objective to increase the effectiveness and efficiency of the Title II food aid programs by improving the technical knowledge and operations and promoting food aid commodity program best practices through coordination and delivery of institutional strengthening and capacity building support for current Title II awardees and prospective awardees worldwide. Related to staffing, FFP's first priority is to ensure that there are in-country staff with Title II nonemergency program oversight responsibility in all countries that do not currently have them. This oversight responsibility includes M&E, but not exclusively. FFP does take into consideration the size and number of programs in the country and the difficulty of the conditions in determining where to place staff. [See comment 4] Field staff are hired based on an appropriate set of experience and skills to carry out their oversight duties, which include but are not restricted to M&E, and they are provided with training to gain and strengthen needed skills. While it is true that USAID does not have data on the numbers of field staff that have M&E competencies or an inventory of its staff's skills and abilities, the GAO has not made clear why this is necessary and what purpose it would serve, especially given the cost and effort to acquire this information and keep it updated. It would be nearly impossible to obtain accurate figures since M&E, especially monitoring, cannot and should not be disaggregated from other programmatic activities, especially as the practical use of this information is not evident. [See comment 5] USAID is and will continue to examine additional options for funding M&E at the headquarters and country level. The following are GAO's comments on the U.S. Agency for International Development letter dated September 17, 2009. GAO Comments: 1. We reviewed the eight documents listed and acknowledged several of them in our report's section on policies and procedures. However, none of these documents constitutes a plan. Instead, they are policy or technical documents that describe how USAID and its awardees should plan for aspects of monitoring and evaluation. 2. We considered FFP's Strategic Plan for 2006-2010 but determined it to be irrelevant to our analysis because it was prepared in 2005, prior to the passage of the 2008 Food for Peace Act, and covers only some aspects of monitoring. The Performance Management Plan is related to the strategic plan and provides a set of indicators to measure progress. Neither of these documents discusses funding, resources, staffing needs, or how results would be used to improve program management. 3. We reported on the planned studies earlier in this report. However, USAID provided us not with a plan, but rather with a list of selected studies, their implementers, costs, and a high-level description. In the list, USAID did not discuss alternatives, outline priorities, or explain how the selected studies would address FFP's most pressing needs or be used for program management. As we noted, several studies do not have assured funding due to the legal restrictions on FFP's funding, which makes the need for integrated planning even more critical. 4. We noted that FFP allocated the new staff positions in order to ensure coverage in all countries where it has programs. While this is one factor to consider in workforce planning, FFP did not consider other important factors, such as existing staff skills or country- specific needs for monitoring and evaluation because it did not collect data on them. Lacking a systematic process to collect such data, FFP is unable to determine if it has adequate staff with sufficient training and skills in monitoring and evaluation. 5. FFP officials reported that they had not used almost $9 million of the $22 million available in fiscal year 2009. Better planning could ensure that the available funds are more fully used in future years for activities such as surveying FFP staff on their monitoring and evaluation skills and conducting an assessment of FFP monitoring and evaluation needs. [End of section] Appendix V: GAO Contact and Staff Acknowledgments: GAO Contact: Thomas Melito, (202) 512-9601, or melitot@gao.gov: Staff Acknowledgments: In addition to the individual named above, Phillip Thomas, Martin De Alteriis, and Michael LaForge, Assistant Directors; and Sada Aksartova; Carol Bray; Ming Chen; Debbie Chung; Lynn Cothern; Mark Dowling; Brian Egger; Ian Ferguson; Etana Finkler; Kendall Helm; Joy Labez; Jeff Miller; Yvonne Moss; Ulyana Panchishin; Lisa Reijula; Julia Ann Roberts; Barbara Shields; Stephanie Shipman; and George Taylor made key contributions to this report. [End of section] Related GAO Products: International Food Assistance: Key Issues for Congressional Oversight. [hyperlink, http://www.gao.gov/products/GAO-09-977SP]. Washington, D.C.: September 30, 2009. International Food Assistance: Local and Regional Procurement Provides Opportunities to Enhance U.S. Food Aid, but Challenges May Constrain Its Implementation. [hyperlink, http://www.gao.gov/products/GAO-09-757T]. Washington, D.C.: June 4, 2009. International Food Assistance: Local and Regional Procurement Can Enhance the Efficiency of U.S. Food Aid, but Challenges May Constrain Its Implementation. [hyperlink, http://www.gao.gov/products/GAO-09-570]. Washington, D.C.: May 29, 2009. International Food Security: Insufficient Efforts by Host Governments and Donors Threaten Progress to Halve Hunger in Sub-Saharan Africa by 2015. [hyperlink, http://www.gao.gov/products/GAO-08-680]. Washington, D.C.: May 29, 2008. Foreign Assistance: Various Challenges Limit the Efficiency and Effectiveness of U.S. Food Aid. [hyperlink, http://www.gao.gov/products/GAO-07-905T]. Washington, D.C.: May 24, 2007. Foreign Assistance: Various Challenges Impede the Efficiency and Effectiveness of U.S. Food Aid. [hyperlink, http://www.gao.gov/products/GAO-07-560]. Washington, D.C.: April 13, 2007. Foreign Assistance: U.S. Agencies Face Challenges to Improving the Efficiency and Effectiveness of Food Aid. [hyperlink, http://www.gao.gov/products/GAO-07-616T]. Washington, D.C.: March 21, 2007. Darfur Crisis: Progress in Aid and Peace Monitoring Threatened by Ongoing Violence and Operational Challenges. [hyperlink, http://www.gao.gov/products/GAO-07-9]. Washington, D.C.: November 9, 2006. Maritime Security Fleet: Many Factors Determine Impact of Potential Limits of Food Aid Shipments. [hyperlink, http://www.gao.gov/products/GAO-04-1065]. Washington, D.C.: September 13, 2004. United Nations: Observations on the Oil for Food Program and Iraq's Food Security. [hyperlink, http://www.gao.gov/products/GAO-04-880T]. Washington, D.C.: June 16, 2004. Foreign Assistance: Lack of Strategic Focus and Obstacles to Agricultural Recovery Threaten Afghanistan's Stability. [hyperlink, http://www.gao.gov/products/GAO-03-607]. Washington, D.C.: June 30, 2003. Foreign Assistance: Sustained Efforts Needed to Help Southern Africa Recover from Food Crisis. [hyperlink, http://www.gao.gov/products/GAO-03-644]. Washington, D.C.: June 25, 2003. Food Aid: Experience of U.S. Programs Suggest Opportunities for Improvement. [hyperlink, http://www.gao.gov/products/GAO-02-801T]. Washington, D.C.: June 4, 2002. Foreign Assistance: Global Food for Education Initiative Faces Challenges for Successful Implementation. [hyperlink, http://www.gao.gov/products/GAO-02-328]. Washington, D.C.: Feb. 28, 2002. Foreign Assistance: U.S. Food Aid Program to Russia Had Weak Internal Controls. [hyperlink, http://www.gao.gov/products/GAO/NSIAD/AIMD-00-329]. Washington, D.C.: September 29, 2000. Foreign Assistance: U.S. Bilateral Food Assistance to North Korea Had Mixed Results. [hyperlink, http://www.gao.gov/products/GAO/NSIAD-00-175]. Washington, D.C.: June 15, 2000. Foreign Assistance: Donation of U.S. Planting Seed to Russia in 1999 Had Weaknesses. [hyperlink, http://www.gao.gov/products/GAO/NSIAD-00-91]. Washington, D.C.: March 9, 2000. Foreign Assistance: North Korea Restricts Food Aid Monitoring. [hyperlink, http://www.gao.gov/products/GAO/NSIAD-00-35]. Washington, D.C.: October 8, 1999. Food Security: Factors That Could Affect Progress toward Meeting World Food Summit Goals. [hyperlink, http://www.gao.gov/products/GAO/NSIAD-99-15]. Washington, D.C.: March 22, 1999. Food Security: Preparations for the 1996 World Food Summit. [hyperlink, http://www.gao.gov/products/GAO/NSIAD-97-44]. Washington, D.C.: November 7, 1996. [End of section] Footnotes: [1] Section 3001 of Pub. L. No. 110-246, the Food, Conservation, and Energy Act of 2008 changed the title of the underlying legislation from the Agricultural Trade Development Assistance Act of 1954, also known as Pub. L. 480, to the Food for Peace Act. Title II of the Food for Peace Act addresses donation of agricultural commodities for humanitarian purposes. [2] USAID FFP, Bureau for Democracy, Conflict, and Humanitarian Assistance, Strategic Plan for 2006-2010 (Washington, D.C.: May 2005). [3] For the purposes of this report, we consider whether agencies are monitoring (1) that the necessary inputs for programs (equipment, supplies, and personnel) are in place and that programs are being implemented as intended and (2) that programs are achieving their expected outputs and targets by regularly tracking performance indicators. See appendix I for more details. [4] For the purposes of this report, we consider whether agencies are evaluating on a periodic or ad hoc basis (1) the extent to which program objectives were achieved as well as the factors that influenced outcome achievement and (2) the degree to which outcomes and impacts can be attributed directly to programs, and the cost effectiveness of the programs. These studies are often considered cross-cutting because they look across projects, programs, and countries, among other things. See appendix I for more details. [5] This recommendation was also addressed to the U.S. Department of Agriculture. See GAO, Foreign Assistance: Various Challenges Impede the Efficiency and Effectiveness of U.S. Food Aid, [hyperlink, http://www.gao.gov/products/GAO-07-560] (Washington, D.C.: Apr. 13, 2007). [6] In the same speech, the Secretary of State also noted that U.S. diplomacy and development demand "our best efforts and deepest commitment to achieving sustainable results" and called for an evidence- based approach to foreign policy. [7] See appendix II for USAID's report to Congress issued on December 31, 2008. [8] AEA is an international professional association of evaluators devoted to the application and exploration of program evaluation, personnel evaluation, technology, and other forms of evaluation. In February 2009, AEA's Task Force on Evaluation Policy issued An Evaluation Roadmap for a More Effective Government, which recommended a framework for each federal agency to guide the development and implementation of its evaluation programs. The framework offers a set of guiding principles intended to facilitate the integration of monitoring and evaluation activities with program management. We used the AEA framework in considering the extent to which USAID's practices for monitoring and evaluation are consistent with established principles. Additionally, GAO has issued guidance on aspects of monitoring and evaluation over the years in reports including Performance Measurement and Evaluation: Definitions and Relationships, [hyperlink, http://www.gao.gov/products/GAO-05-739SP] (Washington, D.C.: May 2005); The Results Act: An Evaluator's Guide to Assessing Agency Annual Performance Plans, [hyperlink, http://www.gao.gov/products/GAO/GGD-10.1.20] (Washington, D.C.: April 1998); and Designing Evaluations, [hyperlink, http://www.gao.gov/products/GAO/PEMD-10.1.4] (Washington, D.C.: March 1991). [9] IEEE is a nonprofit, technical professional association that develops standards for a broad range of global industries including the information technology and information assurance industries and is a leading source for defining best practices. [10] We selected Bangladesh and Haiti based on the size and scope of USAID's nonemergency food aid programs, the presence of field monitoring staff, and the differences in how food aid programs are monitored in these countries (For example, Layers, a computerized monitoring system for quality assurance, is used in Haiti but not in Bangladesh). [11] According to FFP, the selection of the 20 priority countries for nonemergency food aid programs, completed in 2009, was based on a weighted average of the country's ranking under three food insecurity indicators: (1) percentage of children stunted (utilization), (2) percentage of population living under $1 a day (access), and (3) percentage of population undernourished (availability). The utilization indicator contributes 60 percent to the needs-based country rank, whereas the access indicator and the availability indicator contribute 30 and 10 percent, respectively. [12] On a case-by-case basis, SYAPs may be extended beyond the initial 12 months approved for the program. [13] These development projects are typically supported by implementing partners that sell food aid commodities as a means to generate cash to fund these projects. Food-insecure populations, as defined in FFP's Strategic Plan for 2006-2010, are people at risk of food insecurity because of physiological, socioeconomic, and political status, physical security, and limited or weak governance or a population whose ability to cope has been temporarily hindered by a shock. Beginning in 2006, multiyear assistance programs, MYAPs, replaced Development Assistance Programs (DAP). DAPs focused on increasing food security, whereas MYAPs--reflecting FFP's main strategic objective for 2006-2010--focus on reducing food insecurity. [14] Monetization is a practice that involves the sale in a recipient country of food aid commodities that have been purchased in and shipped from the United States. [15] Following the AEA guidance, we are defining program management as the management of a program's key stages, from planning and initial development through start up, ongoing implementation, and reauthorization. [16] The FANTA-2 project was established through a cooperative agreement between USAID and AED. Under subagreements, FANTA-2 staff collaborate with partner organizations such as Cornell University, Tufts University, Food Aid Management, the International Food Policy Research Institute, Technical Assistance to NGOs International, and the World Health Organization. [17] See Michael Bamberger, Jim Rugh, and Linda Mabry, Real World Evaluation: Working under Budget, Time, and Data and Political Constraints (Thousand Oaks, Calif.: Sage, 2006). The authors reviewed evaluations of CARE International programs and projects. [18] The 9 filled positions are as follows: 1 in Burundi, 3 in Madagascar, 1 in Malawi, 1 in Mali, 1 in Niger, 1 in Zambia, as well as 1 position in the Southern Africa Regional Office. [19] According to USAID, 4 new positions in Madagascar include 1 administrative staff member and 1 driver. [20] The term "architectural diagram" includes a description of an organization's current or "as is" information technology environment and its target or "to be" information technology environment. The architecture also includes a transition or sequencing plan, which is based on an analysis of the gaps between the "as is" and "to be" environments. [21] IEEE Guide for Information Technology--System Definition--Concept of Operations Document, Std. 1362-1998. [22] Remote monitoring will be based upon a regular assessment of (1) a livelihoods-based analytic food security framework of food security, (2) weather and crop monitoring and assessment, and (3) markets and trade monitoring and assessment. [23] Layers was developed as part of the FANTA-2 project. FANTA staff conduct initial training on the use of Layers for FFP field staff and provide continuing technical assistance to FFP staff after Layers is introduced. [24] Federal grants and cooperative agreements are used to accomplish a public purpose of support or stimulation authorized by the federal statute. The award mechanisms of grants and cooperative agreements are the largest components of awards made by USAID. [25] Personal services contracts are characterized by the employer- employee relationship that exists between the government and the contractor. The contractor is treated very much like the civil service staff, frequently performing the same or similar work. USAID hires personal services contractors to work in a variety of fields both in USAID/Washington and in USAID missions worldwide. [26] In addition, the viability of Layers will depend on USAID missions' willingness to use their own funds to provide ongoing support for Layers--estimated by FFP at $60,000 annually--after the first year. FFP and FANTA-2 are optimistic that missions will want to do so both because the cost is small relative to improvements in the monitoring of the implementing partners' programs and because the system can be easily adapted for nonfood aid programs. [27] Prior to 2009, food security country frameworks had been prepared for Madagascar, Malawi, and Mozambique. [28] In 1977, Congress passed the Bellmon Amendment to Pub. L. No. 480 (Pub. L. No. 480 was renamed the Food for Peace Act in 2008). As required by the Bellmon Amendment, before food aid can be supplied, the U.S. government must determine that (1) adequate storage facilities are available in the recipient country at the time of export of the commodity to prevent waste or spoilage and (2) the distribution of the commodities will not result in a substantial disincentive or interference with domestic production or marketing in that country. [29] These bulletins explain requirements regarding monitoring and evaluation, reporting, and performance indicators and reinforce the Automated Directives System (ADS), chapter 203, which details the agency's monitoring and evaluation requirements. [30] According to guidance issued by State's Office of the Director of Foreign Assistance in March 2009, independent evaluations tend to have a high degree of credibility because the evaluators conducting them are not responsible to managers in the projects that are being evaluated. However, USAID's capacity to perform independent evaluations in recent years has been limited because it lacked a central evaluation unit between 2006 and 2008. [31] In one instance, we were unable to determine whether the lead evaluator was independent due to a lack of information in the evaluation and, in another instance, it was not clear whether the external evaluators had worked as a team with the implementing partners. In addition to the 16 final evaluations we found in DEC, we looked at a final evaluation that the USAID mission in Bangladesh provided to us in hard copy and found that it had been conducted by an external evaluator. [32] AED defines "integrity" as striving for the highest ethical standards, transparency, and accountability in all aspects of its work as an independent organization free of political influence. [33] Tufts University, an approved subrecipient under the FANTA-2 cooperative agreement, recently received a 3-year task order to assess the success of exit strategies for FFP's food aid programs in Bolivia. [34] According to USAID officials, there is an example of a cross- cutting evaluation that was funded in part by FFP and compared two approaches to prevent malnutrition in infants in Haiti. The first approach targeted the interventions on all children aged between 6 and 23 months (the "preventive" model) and the second approach targeted underweight children aged between 6 and 60 months (the "recuperative" model). By the use of carefully selected comparison groups, the study concluded that the preventive early intervention model yielded the greatest benefits. However, the only other example FFP provided of a recent cross-cutting study was a FANTA Report on Food Aid and Food Security Assessment: A Review of the Title II Development Food Aid Program issued in 2002. [35] Guidance recently issued by FFP describes evaluation as answering the "why," "why not" and the "what else" of performance and states that it is used on a periodic basis to assess effects and impacts, as well as efficiency and effectiveness. However, the guidance only requires evaluations of the extent to which Food for Peace programs are meeting outcomes and objectives. [36] FFP plans for conducting future cross-cutting impact evaluations and other studies were discussed in this report. [37] According to USAID's most recent workforce plan, the agency has created competency models and conducts annual competency assessments for the human resources, information technology, and acquisitions occupations. It projects that it will complete competency models for the foreign service in fiscal year 2010, civil service in fiscal year 2011, and foreign service nationals and U.S. personal services contractors in fiscal year 2013. USAID, USAID Five-Year Workforce Plan FY 2009-FY 2013. [38] During our fieldwork in Bangladesh and Haiti, we found that the number of staff devoted to monitoring and evaluation, and the amount of time they dedicated to those activities, varied considerably between the two missions. FFP in Haiti dedicated more staff to monitoring and evaluation, and they reported that they spent more of their time on monitoring and evaluation than did FFP staff in Bangladesh. [39] The Food for Peace Act authorizes the hiring of personal services contractors only in recipient countries. [40] USAID's central evaluation office was reestablished in 2008 but, as of August 2009, had only one employee, who is its chief. However, according to the chief, the office has begun the process of hiring about seven new staff. [41] At our request, the two FFP implementing partners that operate in Bangladesh both estimated that about 3 percent of their total project budgets were spent on monitoring and evaluation. However, these estimates were for all their monitoring and evaluation, including staff salaries. We were not able to independently verify these estimates. [42] For example, FFP implementing partners are to plan final evaluations for multiyear assistance programs. [43] USAID's central evaluation unit is in the process of preparing an agencywide plan for evaluation, which might be useful to units like FFP as they develop plans for their own programs. The central evaluation unit had hoped to complete its agencywide plan by summer 2009 but recently reported that the plan was unlikely to be finalized until a new USAID Administrator was confirmed. [44] We considered the field monitoring of USAID programs a form of input and implementation evaluation. [45] see PM2A guidance and research at [hyperlink, http://www.fanta.org]. [46] [hyperlink, http://www.usaid.gov/our_work/humanitarian_assistance/ffp/FFPtrigger_ind icators.pdf]. [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s Web site, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: