This is the accessible text file for GAO report number GAO-07-909 
entitled 'Millennium Challenge Corporation: Vanuatu Compact Overstates 
Projected Program Impact' which was released on July 26, 2007. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to the Chairman, Committee on Foreign Affairs, House of 
Representatives: 

United States Government Accountability Office: 

GAO: 

July 2007: 

Millennium Challenge Corporation: 

Vanuatu Compact Overstates Projected Program Impact: 

GAO-07-909: 

GAO Highlights: 

Highlights of GAO-07-909, a report to the Chairman, Committee on 
Foreign Affairs, House of Representatives 

Why GAO Did This Study: 

In January 2004, Congress established the Millennium Challenge 
Corporation (MCC) for foreign assistance. Congress has appropriated 
almost $6 billion to MCC. As of March 2007, MCC had signed almost $3 
billion in compacts with 11 countries, including a 5-year, $65.7 
million compact with Vanuatu. MCC states that the Vanuatu compact will 
have a transformational effect on the country’s economy, increasing per 
capita income and GDP and benefiting 65,000 poor, rural people. GAO 
examined (1) MCC’s methods of projecting economic benefits, (2) MCC’s 
portrayal and analysis of the projected benefits, and (3) risks that 
may affect the compact’s impact. GAO reviewed MCC's analyses and met 
with officials and business owners in Vanuatu as well as with other 
donors. 

What GAO Found: 

MCC projects that the Vanuatu compact’s transportation infrastructure 
projects will provide direct benefits such as reduced transportation 
costs and induced benefits from growth in tourism and agriculture. MCC 
estimated the costs and benefits over 20 years, with benefits beginning 
in full in 2008 or 2009 and growing each year, and it counted poor, 
rural beneficiaries by defining the area where benefits were likely to 
accrue. Using projected benefits and costs, MCC calculated the 
compact’s economic rate of return (ERR) and its effects on Vanuatu’s 
gross domestic product (GDP) and per capita income. MCC’s portrayal of 
the projected impact does not reflect its underlying data. MCC states 
that per capita income will increase by approximately $200, or 15 
percent, by 2010 and by $488, or 37 percent, by 2015. However, MCC’s 
underlying data show that these figures represent the sum of individual 
years’ gains in per capita income relative to 2005 and that actual 
gains will be $51, or 3.9 percent, in 2010 and $61, or 4.6 percent, in 
2015. MCC also states that GDP will increase by an additional 3 percent 
a year, but its data show that after GDP growth of 6 percent in 2007, 
the economy’s growth will continue at about 3 percent, as it would 
without the compact. MCC states that the compact will benefit 
approximately 65,000 poor, rural inhabitants, but this statement does 
not identify the financial benefits that accrue to the rural poor or 
reflect its own analysis that 57 percent of benefits go to others. We 
identified five key risks that could affect the compact’s projected 
impacts. (1) Cost estimate contingencies may not be sufficient to cover 
project overruns. (2) Compact benefits will likely accrue more slowly 
than MCC projected. (3) Benefit estimates assume continued maintenance, 
but MCC’s ability to ensure maintenance will end in 2011, and Vanuatu’s 
maintenance record is poor. (4) Induced benefits depend on businesses’ 
and residents’ response to new opportunities. (5) Efficiency gains, 
such as time saved in transit, may not increase per capita income. Our 
analysis of these areas of risk illustrates the extent that MCC’s 
projections are dependent on assumptions of immediate realization of 
benefits, long-term maintenance, realization of induced benefits, and 
benefits from efficiency gains. 

Figure: Vanuatu Compact's Impact on Per Capita Income According to MCC 
Statement vs. MCC Data: 

[See PDF for Image] 

Source: GAO analysis of MCC data. 

[End of figure] 

What GAO Recommends: 

GAO recommends that the Chief Executive Officer of MCC (1) revise the 
public reporting of the Vanuatu compact’s projected impact, (2) assess 
whether similar reporting in other compacts accurately reflects 
underlying analyses, and (3) improve its economic analyses by more 
fully accounting for risks to project benefits. MCC did not directly 
address our recommendations but commented that it had not intended to 
make misleading statements and that its portrayal of projected results 
was factual and consistent with underlying data. 

[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-909]. 

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact David Gootnick, (202) 512-
3149, gootnickd@gao.gov. 

[End of section] 

[See PDF for image] 

[End of figure] 

Contents: 

Letter: 

Results in Brief: 

Background: 

MCC Projected Compact's Impact Using Estimates of Benefits, Costs, and 
Catchment Area: 

MCC's Data Do Not Support Its Portrayal of Compact Benefits: 

Several Risks May Lead to Reduced Project Benefits: 

Conclusions: 

Recommendations: 

Agency Comments and Our Evaluation: 

Appendix I: Objectives, Scope, and Methodology: 

Appendix II: Prior Development Assistance to Vanuatu: 

Appendix III: MCC Estimates of Per Capita Income Impact and Poor, Rural 
Beneficiaries: 

Appendix IV: MCC Cost Estimates for the Vanuatu Compact: 

Appendix V: Illustrative Alternative Calculations of Vanuatu Compact 
Impact: 

Appendix VI: Comments from the Millennium Challenge Corporation: 

GAO Comments: 

Appendix VII: GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: Average Annual Contributions of Top Five Donors to Vanuatu in 
2004 and 2005: 

Table 2: Number of Poor, Rural Beneficiaries of Vanuatu Compact, as 
Calculated by MCC: 

Table 3: Effect of Calculation and Data Errors on Beneficiary 
Population in Efate and Santo: 

Table 4: Effect of MCC Assumptions on Beneficiary Population in Efate 
and Santo: 

Table 5: Summary of Compact Impacts as Presented by MCC, with GAO 
Recalculations: 

Table 6: Summary of Compact ERR, GDP, and Per Capita Income impacts 
under Alternative Scenarios of Accounting for Risks to Benefits: 

Table 7: Summary of Selected Compact Project Impacts under Alternative 
Scenarios Accounting for Risks to Benefits: 

Figures: 

Figure 1: Location of Vanuatu: 

Figure 2: Development and Implementation of Vanuatu Compact: 

Figure 3: MCC Vanuatu Projects by Size and Location: 

Figure 4: MCC Statement of Impacts in March 2006 Congressional 
Notification: 

Figure 5: MCC's Logic Model for the Vanuatu Compact: 

Figure 6: MCC Benefits Calculations as Source of Statements about 
Impact of Vanuatu Compact: 

Figure 7: Vanuatu Compact's Projected Impact on Real Per Capita Income 
According to MCC Statement and MCC Data Relative to 2005 Per Capita 
Income: 

Figure 8: MCC Methodology for Projecting Vanuatu Compact's Impact on 
Real Per Capita Income: 

Figure 9: Vanuatu GDP Growth with and without MCC Compact: 

Figure 10: MCC Analysis of Distribution of Vanuatu Compact Benefits: 

Figure 11: MCC Compact's Projected Impact on Vanuatu Per Capita Income 
with and without Adjusted Population Estimates: 

Figure 12: MCC Compact's Projected Impact on Vanuatu Per Capita Income 
in Real Terms Relative to Per Capita Income without MCC: 

Figure 13: MCC Efate Ring Road Catchment Area: 

Abbreviations: 

ADB: Asian Development Bank: 
CEO: Chief Executive Officer: 
ERR: economic rate of return: 
GDP: gross domestic product: 
IMF: International Monetary Fund: 
MCA: Millennium Challenge Account: 
MCC: Millennium Challenge Corporation: 
PWD: Public Works Department: 

United States Government Accountability Office: 
Washington, DC 20548: 

July 11, 2007: 

The Honorable Tom Lantos: 
Chairman: 
Committee on Foreign Affairs: 
House of Representatives: 

Dear Mr. Lantos: 

In January 2004, Congress established the Millennium Challenge 
Corporation (MCC) to administer the Millennium Challenge Account (MCA) 
for foreign assistance. MCC's mission is to reduce poverty by 
supporting sustainable, transformative economic growth in developing 
countries that create and maintain sound policy environments. MCC 
carries out this mission by funding projects or activities in 
developing countries that have demonstrated a commitment to ruling 
justly and democratically, encouraging economic freedom, and investing 
in people. Congress appropriated almost $6 billion for fiscal years 
2004 to 2007 to MCC, and the President has requested an additional $3 
billion in MCC funding for fiscal year 2008. 

As of March 2007, MCC had signed compacts with 11 countries, totaling 
approximately $3 billion, including a 5-year, $65.7 million compact 
with the Pacific island nation of Vanuatu.[Footnote 1] Although MCC's 
compact with Vanuatu is its smallest compact monetarily, it provides by 
far the largest amount relative to the country's population and gross 
domestic product (GDP).[Footnote 2] In a process known as due 
diligence, MCC analyzed Vanuatu's proposal for compact assistance to 
determine the compact's expected economic rate of return (ERR)[Footnote 
3] and impact on poverty reduction and economic growth. After 
completing its due diligence analysis, MCC stated in its compact and in 
its notification to Congress that it expects the Vanuatu compact to 
have a transformational impact on Vanuatu's economic development--an 
effect MCC defines as "a dramatic and long-lasting impact on poverty 
reduction through sustainable economic growth."[Footnote 4] MCC states 
that its compacts will provide or contribute to a transformational 
impact in 5 of its 11 compacts.[Footnote 5] 

At your request, we reviewed MCC's economic analyses of the Vanuatu 
compact. Specifically, we examined: 

* MCC's methods of projecting the compact's economic benefits and 
methods of calculating the benefits, 

* MCC's portrayal and analysis of the projected benefits, and: 

* risks that could affect the compact's impact on poverty reduction and 
economic growth. 

We reviewed MCC's record of due diligence and supplemented this review 
with interviews with MCC officials to identify MCC's logic, data, 
methods, and assumptions for determining the compact's projected costs 
and benefits, and to identify the projected effects on GDP, per capita 
income, and poverty. We evaluated MCC's statements about the compact's 
impacts in its notification to Congress, in the Vanuatu compact, and in 
its "investment memo"[Footnote 6] by comparing these statements with 
the underlying analyses and data used to support them. We could not 
validate most of MCC's underlying data and assumptions because the data 
were not available or could not be checked within the time frames of 
our engagement. We examined MCC's methodologies and checked the 
analyses for calculation errors. Further, we identified risks to MCC's 
compact results based on our review of MCC's internal documentation, 
donor reporting, and academic development literature. We interviewed 
Vanuatu and MCC officials and contacted the contractor that assisted in 
MCC's analyses. We also interviewed interested parties such as tourism 
and agriculture business owners in Vanuatu. We focused our analysis and 
field work on MCC's three transportation infrastructure projects on 
Vanuatu's two most populous islands, Santo and Efate, which represent 
56 percent of compact cost. We modeled several areas of project risks 
to illustrate their maximum impact on the economic analyses of ERR, 
GDP, and per capita income. In modeling these risks, we used the data 
from MCC's economic analyses; we did not validate these data. We 
conducted our review from August 2006 through May 2007 in accordance 
with generally accepted government auditing standards. 

Results in Brief: 

MCC projected the impact of the Vanuatu compact by estimating the 
program's benefits, costs, and beneficiaries and calculating the 
compact's effect on per capita income, GDP, and poverty reduction. 
According to MCC, transportation infrastructure improvements will 
provide direct benefits from construction spending in the local 
economy, reduced transportation costs, and improved services. The 
improved infrastructure will also provide induced benefits from growth 
in Vanuatu's tourism and agriculture sectors. MCC estimated the value 
of these benefits over a 20-year period, beginning in full in 2008 or 
2009 and growing each year. MCC developed its project cost estimates 
based on existing cost estimates prepared for the government of Vanuatu 
and for another donor. To determine the number of poor, rural 
beneficiaries, MCC defined a catchment area--the geographic area in 
which benefits may be expected to accrue--using maps of Vanuatu and 
data from the most recent Vanuatu census. Using its projected benefit 
and cost data, MCC calculated (1) the compact's ERR, comparing 
projected benefits to projected costs; (2) per capita income, 
determining the total benefits and dividing the total value by 
Vanuatu's baseline population; and (3) the compact's effect on 
Vanuatu's GDP by computing the total benefits added to the economy. 

MCC's portrayal of the Vanuatu compact's impact does not reflect the 
data and analysis underlying its projections of the compact's benefits. 
MCC states that as a result of the compact, per capita income will 
increase by approximately $200, or 15 percent, by 2010 and $488, or 37 
percent, by 2015. This statement suggests that per capita income in 
2010 and 2015 will be, respectively, 15 percent and 37 percent higher 
than without the compact. However, MCC's underlying data show that 
these figures represent the sum of gains in per capita income for 
individual years relative to 2005 rather than actual, or net, annual 
gains as of 2010 or 2015. For example, MCC sums the per capita income 
gains relative to 2005 for 2006 to 2010, averaging 3 percent, as 15 
percent in 2010, without stating that these gains are cumulative. Our 
analysis of MCC's data shows that actual gains in per capita income, 
relative to income in 2005, would be $51, or 3.9 percent, in 2010 and 
$61, or 4.6 percent, in 2015. Likewise, MCC states that Vanuatu's GDP 
will increase by "an additional 3 percent a year." However, MCC's 
underlying data and calculations show that while the level of Vanuatu's 
GDP will grow by 6 percent in 2007, the economy's growth rate in 
subsequent years continues at approximately 3 percent, MCC's assumed 
rate without the compact. Regarding MCC's portrayal of the compact's 
impact on poverty reduction, MCC's publicly available documents state 
that the compact is expected to benefit approximately 65,000 poor, 
rural inhabitants "living nearby and using the roads to access markets 
and social services." However, MCC's underlying documentation shows 
that these 65,000 beneficiaries will not receive the majority of the 
benefits but will instead share the 43 percent of the compact's 
monetary benefits that MCC expects to go to the local population; the 
remaining 57 percent will go to beneficiaries such as tourism services 
providers, transport companies, and local businesses. Finally, although 
MCC's estimates of compact benefits and beneficiaries are generally 
reasonable, some of the calculations and assumptions it used are 
problematic. Correcting MCC's calculations of per capita income 
benefits slightly reduces the estimated benefit, and correcting MCC's 
calculations and fully discounting some of its assumptions regarding 
the count of beneficiaries would reduce estimated beneficiaries on 
Santo and Efate by about one-third. 

We identified five key risks that could affect the Vanuatu compact's 
projected impact on poverty reduction and economic growth. 

* The contingencies included in MCC's calculations of construction 
costs may not be sufficient to cover potential cost overruns. We 
received MCC cost estimate documentation for 5 of MCC's 11 construction 
projects showing that MCC's estimates for these 5 include design 
contingencies of 20 percent; however, a previous study found average 
cost overruns of more than 20 percent in transportation infrastructure 
projects in other countries. Further, the risk of excessive cost 
overruns is significant in a small country such as Vanuatu. Any 
construction cost overrun could cause MCC to reduce the compact's scope 
and therefore its benefits. 

* Although the compact's benefits are projected to begin shortly after 
completion of the projects, some of these benefits will likely accrue 
more slowly. For example, according to agricultural and timber 
producers, their businesses will likely respond gradually to any 
increased market opportunities. 

* Whereas the benefit projections assume continued maintenance of 
completed projects, MCC's ability to ensure such maintenance will end 
in 2011. Moreover, previous donors to Vanuatu have found Vanuatu's 
record of maintaining donor projects to be poor. Reduced maintenance 
would lead to reduced benefits from the project. 

* The projected induced benefits from expanded tourism and agriculture 
depend on businesses and rural inhabitants responding to opportunities 
created by improved infrastructure. 

* Efficiency gains that MCC counts as direct benefits, such as time 
saved in transit, may not be put to economic use and result in 
increased per capita income, as MCC projects. 

Our analysis of these areas of risk illustrates the extent to which 
MCC's benefit projections are dependent on assumptions of immediate 
realization of benefits, successful long-term maintenance, realization 
of induced benefits, and benefits from efficiency gains. Accounting for 
these risks can reduce overall compact ERR from 24.2 percent, as 
projected by MCC, to between 5.5 percent and 16.5 percent, and some 
projects may have a negative ERR.[Footnote 7] 

To help better express and determine the impact of its compacts, this 
report recommends that the Chief Executive Officer (CEO) of MCC (1) 
revise the public reporting of the projected impact of the Vanuatu 
compact, (2) assess whether similar statements in other compacts 
accurately reflect underlying data, and (3) improve MCC's economic 
analysis by phasing costs and benefits and more fully accounting for 
risks to project benefits. 

We provided a draft of this report to MCC. In commenting on the draft, 
MCC did not directly acknowledge our recommendations. MCC acknowledged 
that its use of projected cumulative compact impact on income and 
growth was misleading, but asserted that it had no intention to mislead 
and that its portrayal of projected compact benefits was factually 
correct. MCC questioned our finding that its underlying data and 
analysis do not support its portrayal of compact benefits and our 
characterization of the program's risks. In response to MCC's comments, 
we clarified the terms and presentation of our analysis of MCC's 
statements and data and provided additional information on the basis 
for our findings. We also provided technical corrections where 
appropriate. 

Background: 

Description of Vanuatu: 

Vanuatu consists of 83 islands spread over hundreds of miles of ocean 
in the South Pacific, 1,300 miles northeast of Sydney, Australia (see 
fig. 1). About 39 percent of the population is concentrated on the 
islands of Santo and Efate. Vanuatu's capital, Port Vila, is on Efate, 
and Vanuatu's only other urban center, Luganville, is on Santo. The 
country has three official languages--English, French, and Bislama--but 
more than 100 other dialects are also spoken. Traditional custom chiefs 
have a significant role in the state particularly in rural areas. Civil 
unrest has caused occasional disruptions; for example, riots erupted in 
Port Vila over the misappropriation of assets from the Vanuatu National 
Provident Fund in 1998, and an ethnic conflict on Efate led Vanuatu's 
parliamentary government to declare a state of emergency in March 2007. 

Figure 1: Location of Vanuatu: 

[See PDF for image] 

Source: GAO based on MCC data; Map Resources (map). 

[End of figure] 

Vanuatu's Economy: 

In the past decade, Vanuatu's real GDP growth averaged 2 percent, 
although more rapid population growth led to a decline in per capita 
GDP over the same period. Average growth of real GDP per capita was 
negative from 1993 to 2005. In its economic analyses, MCC used a 
baseline 2005 Vanuatu per capita income of $1,326. An estimated 40 
percent of Vanuatu's population of about 207,000 has an income below 
the international poverty line of $1 per day. Agriculture and tourism 
are the principal productive sectors of Vanuatu's economy, contributing 
approximately 15 percent and 19 percent to GDP, respectively. Although 
agriculture represents a relatively small share of Vanuatu's overall 
economy, approximately 80 percent of Vanuatu's residents live in rural 
areas and depend on subsistence agriculture for food and shelter, 
selling surplus commodities to generate cash for school fees, 
transportation, consumer goods, and services. Copra (dried coconut) is 
the main cash crop; kava and cacao are also grown. The tourism sector 
is dominated by expatriates of foreign countries living in Vanuatu, who 
also predominate in other formal sectors of the economy such as 
plantation agriculture and retail trade. According to the Asian 
Development Bank (ADB), tourism is one of the most promising sectors in 
Vanuatu in terms of its potential for earning foreign exchange and 
generating employment. According to a survey conducted by the Vanuatu 
National Statistics office, the largest markets for Vanuatu tourism are 
Australia and New Zealand. Despite prior donor efforts, private sector 
development in Vanuatu's tourism and agricultural sectors faces 
challenges, including political uncertainty, and high costs of doing 
business. (See app. II for a summary of donor efforts and Vanuatu's 
development challenges.) 

MCC's Compact with Vanuatu: 

On May 6, 2004, MCC determined that Vanuatu was eligible to submit a 
compact proposal for Millennium Challenge Account (MCA) 
funding.[Footnote 8] Vanuatu's proposal identified transportation 
infrastructure as a key constraint to private-sector development. Based 
on the analysis performed during approximately 5 months of due 
diligence,[Footnote 9] MCC signed a 5-year, $65.7 million compact with 
the government of Vanuatu on March 2, 2006, with entry into 
force[Footnote 10] on April 28, 2006.[Footnote 11] In keeping with its 
statutory requirements, MCC submitted a congressional notification of 
the compact signing on March 7, 2006. 

Figure 2 illustrates the chronology of the development and 
implementation of the Vanuatu proposal and compact. 

Figure 2: Development and Implementation of Vanuatu Compact: 

[See PDF for image] 

Source: GAO analysis of MCC data contained in Investment Memo. 

[End of figure] 

The $65.7 million Vanuatu compact includes $54.5 million for the 
rehabilitation or construction of 11 transportation infrastructure 
assets on 8 of Vanuatu's 83 islands, including roads, wharves, an 
airstrip, and warehouses (see fig. 3). The compact also includes $6.2 
million for an institutional strengthening program to increase the 
capacity of the Vanuatu Public Works Department (PWD) to maintain 
transportation infrastructure.[Footnote 12] The remaining $5 million is 
for program management and monitoring and evaluation. More than half of 
the compact, $37 million, is budgeted for three road projects on Santo 
and Efate islands. On both islands, the compact will upgrade existing 
roads, while on Santo the compact also includes five new bridges on an 
existing road. To oversee and manage the compact programs, the Vanuatu 
government has established MCA-Vanuatu, an independent entity housed 
within the Vanuatu Ministry of Finance and Economic Management. 

Figure 3: MCC Vanuatu Projects by Size and Location: 

[See PDF for image] 

Source: GAO based on MCC data;  Map Resources (map). 

[End of figure] 

The compact and the congressional notification state that the compact 
will have a transformational impact on Vanuatu's economic development, 
increasing average per capita income by approximately $200--15 percent-
-by 2010 and increasing total GDP by "an additional 3 percent a year." 
The investment memo further quantifies the per capita income increase 
as $488--37 percent--by 2015. The compact and the congressional 
notification further state that the compact will provide benefits to 
approximately 65,000 poor, rural inhabitants (see fig. 4). 

Figure 4: MCC Statement of Impacts in March 2006 Congressional 
Notification: 

[See PDF for image] 

Source: MCC Congressional Notification, March 2006. 

[End of figure] 

MCC Projected Compact's Impact Using Estimates of Benefits, Costs, and 
Catchment Area: 

To project the Vanuatu compact program's benefits, costs, and 
beneficiaries and calculate the compact's impact on per capita income, 
GDP, and poverty reduction, MCC made site visits to Vanuatu and 
reviewed available documents to gather needed data. MCC determined that 
investments in transportation infrastructure will lead to increases in 
incomes and used the available data and made assumptions to determine 
the compact's benefits and costs. MCC also estimated the number of 
beneficiaries within a defined catchment area--the geographic area in 
which benefits may be expected to accrue. MCC then used the benefits 
and costs to calculate summary statistics about the compact's ERR and 
effects on Vanuatu's GDP and per capita income. 

MCC Calculated Benefits, Costs, and Beneficiaries: 

To prepare its economic analyses of the compact program's benefits, 
costs, and number of beneficiaries, MCC sent a team of employees and 
contractors to Vanuatu. The contractor MCC retained had previously 
served as a consultant to the ADB in Vanuatu and had prepared a 
multiyear study of projects proposed to the ADB as part of its Vanuatu 
Outer Islands Infrastructure Development Project in 2000 to 2003. Seven 
of MCC's 11 transportation infrastructure projects were initially 
developed as part of the ADB study. While in Vanuatu, the MCC team met 
with officials representing other donors, Vanuatu's Department of 
Economic and Sectoral Planning, the National Statistics Office, and the 
Vanuatu PWD, among others. The MCC team also met with a number of 
interested and informed parties representing the transportation, 
tourism, and agriculture sectors and reviewed a number of documents to 
gather the information needed for its economic analyses.[Footnote 13] 

Benefits: 

MCC's compact aims to benefit poor, rural agricultural producers and 
providers of tourism-related goods and services by reducing 
transportation costs and improving the reliability of access to 
transportation services. MCC's logic model for the compact posits that 
improvements in infrastructure will lead to (1) direct benefits from 
increased transportation reliability, construction spending, and 
reduced transportation costs and (2) induced benefits from greater 
tourism and agricultural trade. The direct and induced benefits in turn 
lead to increases in per capita income and GDP and reduction in poverty 
(see fig. 5). 

Figure 5: MCC's Logic Model for the Vanuatu Compact: 

[See PDF for image] 

Source: MCC, with GAO analysis. 

[End of figure] 

MCC calculated several different direct and induced benefits for the 
compact's projects over a 20-year period. Direct benefits include the 
local value added of construction spending in the economy, reduced 
spoilage of agricultural goods, time saved in transit on the improved 
roads, and reduced user costs for operators of ships, aircraft, and 
vehicles.[Footnote 14] Induced benefits are those projected to result 
from Vanuatu's response to new economic opportunities in tourism and 
agriculture. For example, MCC assumes that the projects will cause 
Vanuatu tourism to grow at 7 percent per year instead of the recent 
rate of 5 percent; tourists will increase their daily spending; new 
hotel rooms will be constructed and hotel occupancy will increase; and 
crop, livestock, and fisheries production will increase. Benefits from 
the completed projects are counted as beginning in full in 2008 or 2009 
and are assumed to increase a minimum of 3 percent every year 
thereafter. MCC expects the benefits of the compact to flow from 
different sources, depending on the project and its location. In Efate, 
the Ring Road is expected to provide direct benefits from decreased 
road user costs and induced benefits through tourism and foreign 
resident spending. In Santo, MCC anticipates benefits from all of these 
efforts, as well as the induced benefit of increased agricultural 
production. On other islands, where tourism is not as developed, the 
benefits primarily derive from user cost savings and increased 
agriculture.[Footnote 15] 

Costs: 

To calculate compact construction and maintenance costs, MCC used 
existing data from previous contractor studies and data from the 
Vanuatu PWD. In MCC's economic model, construction costs are assumed to 
be incurred in the first year after compact signing. Although listed on 
the cost side of the model, MCC also counted the effect of the 
construction spending on the economy as a benefit in the year it takes 
place.[Footnote 16] According to MCC's contractor, it used two 
different primary sources in developing the MCC cost estimates: 

* The contractor used its own 2003 analysis for the ADB projects and 
updated its analysis to account for inflation and changes in project 
scope. 

* The contractor used a cost analysis prepared by a different 
contractor for the government of Vanuatu, dated approximately 2004, as 
a basis for MCC's cost estimates for the Santo and Efate projects. 
MCC's contractor then prepared a new estimate because MCC's projects 
had a different scope of work. We requested from MCC a copy of the 
original report to the government of Vanuatu. According to MCC 
officials, MCC did not have it and the government of Vanuatu was not 
willing to provide the report to MCC for our review. These Santo and 
Efate projects represent 56 percent of compact cost. 

MCC assumed a cost for continued annual maintenance based on past 
estimated maintenance costs provided by the PWD. MCC applied 
assumptions and estimated maintenance costs based on incomplete 
information, because the PWD did not track maintenance costs in its 
budget. For some projects, MCC assumed that maintenance costs would 
decrease once the construction project was complete. MCC also estimated 
and included the cost of a periodic major reinvestment for some 
projects. For example, on the Efate Ring Road, MCC assumed a cost for 
rehabilitation in 2017 and 2026. 

Beneficiaries: 

MCC assessed the number of poor, rural beneficiaries by determining the 
catchment area--the geographic area in which benefits may be expected 
to accrue. For example, MCA-Vanuatu officials told us that they defined 
the catchment area in Efate as consisting of villages within 3 miles of 
the rehabilitated Efate Ring Road. MCC used Vanuatu maps to identify 
villages in the catchment area and used the 1999 Vanuatu National 
Population and Housing Census to determine the number of persons living 
in those villages. In all, MCC calculated that approximately 65,000 
poor, rural people on the eight islands would benefit from MCC 
projects. 

MCC Used Projected Benefits and Costs to Determine ERR and Effects on 
GDP and Per Capita Income: 

MCC used its projection of costs and benefits over a 20-year period as 
the basis for calculating three summaries of the compact's impact: its 
ERR, effect on per capita income, and effect on GDP (see fig. 6). The 
compact's ERR reflects the ratio of the benefits of the compact in 
relation to its costs, expressed as a percentage. For the Vanuatu 
compact, MCC reported an overall compact ERR of 24.7 percent over 20 
years.[Footnote 17] 

Figure 6: MCC Benefits Calculations as Source of Statements about 
Impact of Vanuatu Compact: 

[See PDF for image] 

Source: GAO analysis of MCC data. 

[End of figure] 

MCC also prepared a sensitivity analysis to assess how a range of 
possible outcomes would affect compact results. MCC's tests included: 

* a 1-year delay of the start date for accrued benefits; 

* a 20 percent increase of all costs; 

* a 20 percent decrease of all benefits; and: 

* a "stress test," with a 20 percent increase of all costs and a 20 
percent decrease of all benefits. 

MCC tested a best-case scenario based on a 10 percent increase in 
benefits and a 10 percent decrease in costs. For the overall compact, 
MCC calculated a best-case ERR of 30.2 percent and worst-case of 13.9 
percent. MCC also used the cost and benefit data for the compact 
projects to determine the projects' expected impacts on Vanuatu's per 
capita income and GDP. 

MCC's Data Do Not Support Its Portrayal of Compact Benefits: 

The anticipated benefits of the Vanuatu compact that MCC stated in 
March 2006 in the compact and in its Congressional Notification do not 
accurately reflect MCC's supporting data. MCC projects an increase in 
per capita income of $200 by 2010, but this increase is cumulative; 
MCC's underlying analysis shows a projected increase of $51 in 2010. 
Similarly, while MCC's statement about the compact's transformational 
effect on GDP could be interpreted as increasing the growth rate, the 
projected effect on GDP represents an increase to the level, not the 
growth rate, of GDP. The supporting data show the growth rate remaining 
roughly the same as it would be without the compact. In addition, the 
65,000 rural poor cited by MCC may share less than half of the 
compact's benefits with others in Vanuatu who are not poor or not 
rural. In addition, although MCC's estimates of project benefits and 
beneficiaries are generally reasonable, MCC made some calculation 
errors and questionable assumptions in developing these estimates. 
Correcting MCC's calculations of the per capita income benefit slightly 
reduces the benefit to $49, or 3.7 percent, in 2010; correcting 
calculations and fully discounting some of MCC's assumptions regarding 
beneficiaries would reduce the estimate of beneficiaries on Santo and 
Efate by about one-third. 

Portrayal of Per Capita Income Benefit Suggests Larger Effect Than Data 
Support: 

MCC's portrayal of the compact's projected impact suggests a greater 
effect on Vanuatu's per capita income than its analysis supports. In 
its publicly available documents such as the Vanuatu compact, and the 
congressional notification, issued in March 2006, MCC states that the 
transportation infrastructure project "is expected to have a 
transformational impact . . . increasing average income per capita (in 
real terms) by approximately $200, or 15 percent of current income per 
capita, by 2010." In addition, MCC's investment memo states that the 
compact will cause per capita income to increase by $488, or 37 
percent, by 2015. 

MCC's statements suggest that as a result of the program, average 
incomes in Vanuatu will be 15 percent higher in 2010 and 37 percent 
higher in 2015 than they would be without the compact. However, MCC's 
underlying data show that these percentages represent the sum of 
increases from per capita income in 2005 that MCC projects for each 
year. For example, according to MCC's data, Vanuatu's per capita income 
in a given year between 2006 and 2010 will range from about 2 percent 
to almost 4 percent higher than in 2005; however, MCC sums these 
percentages as 15 percent, without stating that this percentage is a 
cumulative increase from 2005. Our analysis of MCC's data shows that 
actual gains in per capita income, relative to income in 2005, would be 
$51, or 3.9 percent, in 2010 and $61, or 4.6 percent, in 2015 (see fig. 
7). 

Figure 7: Vanuatu Compact's Projected Impact on Real Per Capita Income 
According to MCC Statement and MCC Data Relative to 2005 Per Capita 
Income: 

[See PDF for image] 

Source: GAO analysis of MCC data contained in Investment Memo. 

Note: MCC's statement: "Increasing average income per capita (in real 
terms) by approximately $200 or 15 percent of current income per capita 
by 2010" and by $488--37 percent--by 2015. 

[End of figure] 

Figure 8 further illustrates MCC's methodology in projecting the 
compact's impact on per capita income levels for 2010 and 2015. 

Figure 8: MCC Methodology for Projecting Vanuatu Compact's Impact on 
Real Per Capita Income: 

[See PDF for image] 

Source: GAO analysis data contained in Investment Memo. 

[End of figure] 

Portrayal of GDP Benefit Differs from Underlying Analysis: 

Like its portrayal of the per capita income benefit, MCC's portrayal of 
the program's GDP impact differs from that supported by the underlying 
data. In the compact and the 2006 Congressional Notification, MCC 
states that the compact will have a transformational effect on 
Vanuatu's economy, causing GDP to "increase by an additional 3 percent 
a year." Given the GDP growth rate of about 3 percent that MCC expects 
in Vanuatu without the compact, MCC's statement of a transformational 
effect suggests that the GDP growth rate will rise to about 6 percent. 
However, MCC's underlying data show that although Vanuatu's GDP growth 
rate will rise to about 6 percent in 2007, in subsequent years the GDP 
growth rate will revert to roughly the rate MCC assumes would occur 
without the compact, approximately 3 percent (see fig. 9). Although 
MCC's data show that the compact will result in a higher level (i.e., 
dollar value) of GDP, the data do not show a transformational increase 
to the GDP growth rate. 

Figure 9: Vanuatu GDP Growth with and without MCC Compact: 

[See PDF for image] 

Source; GAO analysis of MCC data. 

Notes: 

According to MCC, "GDP is expected to increase by an additional 3 
percent a year as a result of the MCA program." 

According to MCC data, the compact will have a small impact on GDP 
growth rate in later years. In 2010 to 2015, the GDP growth rate 
resulting from the compact will be 3.1 percent, compared with 3 percent 
without the compact. 

[End of figure] 

Portrayal of Poverty Reduction Does Not Quantify Benefits to Rural 
Poor: 

MCC's portrayal of the compact's projected impact on poverty does not 
identify the proportion of the financial benefits that will accrue to 
the rural poor. In the compact and the Congressional Notification, MCC 
states that the program is expected to benefit "approximately 65,000 
poor, rural inhabitants living nearby and using the roads to access 
markets and social services." However, in its underlying documentation, 
MCC specifies that 43 percent of the monetary benefits are expected to 
go to the local population. The remaining 57 percent of the benefits 
are expected to accrue to other beneficiaries, including expatriate 
tourism services providers, transport providers, government, and local 
businesses (see fig. 10). Given that many of MCC's benefits flow to the 
expatriate-dominated tourism sector, MCC makes an appropriate 
assumption that the local population will not receive all the benefits. 
However, the compact, congressional notifications, and other publicly 
available MCC documents do not provide this information. 

Figure 10: MCC Analysis of Distribution of Vanuatu Compact Benefits: 

[See PDF for image] 

Source: MCC analysis. 

Note: MCC defines "local population" as comprising local producers, 
local consumers, and inhabitants of rural communities. 

[End of figure] 

Although MCC expects that 43 percent of monetary benefits will go to 
the local population, MCC does not establish the proportion of local- 
population benefits that will go to the rural poor. Because MCC defines 
the local population as "local producers, local consumers and 
inhabitants of remote communities," the 65,000 poor, rural 
beneficiaries that MCC projects may share local-population benefits 
with those who are urban and are not poor. 

Estimates of Compact Benefits Are Generally Reasonable, but Some 
Calculations and Assumptions Are Problematic: 

MCC's documentation states that it used conservative assumptions in 
developing its benefit estimates. Although our fieldwork and meetings 
in Vanuatu generally affirmed MCC's assumptions about benefits, many 
participants in our Vanuatu discussion groups noted that the positive 
impacts of the MCC compact are contingent on other factors, such as the 
development of tourist facilities and activities. However, our review 
of MCC's analyses identified some calculation errors in its 
determination of the compact's impact on per capita income and 
estimation of the number of compact beneficiaries, as well as 
questionable assumptions in regard to the beneficiary population. (See 
app. III for a detailed discussion of these calculation errors and the 
effect of MCC's assumptions.) 

* Calculation errors. To determine per capita income effects, MCC 
incorrectly adjusted the value of a benefit stream for inflation that 
was already presented in real (i.e., inflation-adjusted) terms. In 
addition, MCC failed to account for population growth in projecting per 
capita income effects from the compact. Although these errors largely 
cancel each other, correcting them reduces MCC's projection of the per 
capita income benefit slightly, from $51 to $49 in 2010 and from $61 to 
$57 in 2015. Correcting calculation errors significantly increases the 
number of rural beneficiaries; MCC's estimate of compact beneficiaries 
in Santo and Efate did not take into account population growth since 
the 1999 census. 

* Assumptions. Our analysis of MCC's data for Santo and Efate indicates 
that MCC's count of beneficiaries may be overestimated. MCC's count of 
poor, rural beneficiaries includes all rural inhabitants in the 
catchment area, indicating that MCC assumes that all rural inhabitants 
are poor. However, according to ADB reporting, rural poverty in Vanuatu 
is widespread but not universal. The poverty level, defined as having 
an income of one U.S. dollar per day, is 51 percent in rural 
areas.[Footnote 18] Further, in defining the catchment areas in Efate 
and Santo, MCC assumed that residents of villages near existing paved 
portions of the Ring Road not improved by MCC, as well as those on off- 
shore islets, would benefit fully from the compact. 

Correcting calculation errors and fully discounting MCC's assumption 
including these residents in the catchment area would reduce the 
beneficiary count on Efate and Santo by 32 percent--from 26,553, as 
stated by MCC, to 18,070--indicating that MCC may overestimate the 
compact's beneficiaries. 

Several Risks May Lead to Reduced Project Benefits: 

We identified five key risks that may affect the Vanuatu compact's 
projected impact on poverty reduction and economic growth. First, the 
contingencies included in MCC's calculations of construction costs may 
not be sufficient to cover average transportation project overruns. 
Second, although the compact's benefits are projected to begin shortly 
after completion of the projects, some of these benefits are likely to 
accrue more slowly. Third, while the benefit projections assume 
continued maintenance of completed projects, MCC's ability to ensure 
such maintenance will end in 2011, and Vanuatu's record of road 
maintenance is poor. Fourth, the projected induced benefits from 
expanded tourism and agriculture depend on businesses and rural 
inhabitants responding to opportunities created by improved 
infrastructure. Fifth, efficiency gains, such as time saved in transit, 
may not result in increased per capita income, as MCC projects. Our 
analysis of these areas of risk illustrates the extent to which MCC's 
projections of benefits are dependent on assumptions of immediate 
realization of benefits, successful long-term maintenance, realization 
of induced benefits, and benefits from efficiency gains.[Footnote 19] 

Construction Costs May Exceed Contingencies: 

Although MCC considered the risk of construction cost increases, the 
contingencies used in its calculations may not be sufficient to cover 
actual construction costs. We received documentation from MCC's 
contractor of its MCC cost estimates for only 5 of MCC's 11 
construction projects. For these five, it used a design contingency of 
20 percent.[Footnote 20] In its due diligence book, MCC states that its 
cost estimates include physical contingencies, with an average value of 
15 percent, and price contingencies, with an average value of 12 
percent. However, cost overruns of more than 20 percent occur in many 
transportation projects.[Footnote 21] For example, a study of more than 
250 transportation projects in Europe, North America, and elsewhere 
found that costs for all projects were 28 percent higher, on average, 
than forecasted at the time of decision to build, while road projects 
averaged escalations of 20.4 percent.[Footnote 22] Further, as MCC's 
analysis notes, the risk of excessive cost overruns is significant in a 
small country such as Vanuatu, because (1) few comparable projects have 
been undertaken in Vanuatu and, consequently, little previous cost 
information is available and (2) no local contractors are capable of 
undertaking a project of this size, and foreign contractors may apply 
large margins to cover unknown factors.[Footnote 23] Any construction 
cost overrun must be made up within the Vanuatu compact budget by 
reducing the scope, and therefore the benefits, of the compact 
projects.[Footnote 24] Reduced project benefits would in turn reduce 
the compact's ERR and effects on per capita income and GDP. 

In addition, because MCC was unable to provide the data that it used to 
produce the estimates for the Santo and Efate projects, the project 
costs may be at further risk from unchecked assumptions and data (see 
app. IV). 

Compact Benefits Are Likely to Accrue More Slowly Than Projected: 

Although MCC's analysis assumes compact benefits from 2008 or 2009-- 
shortly after the end of project construction--we found that benefits 
are likely to accrue more slowly. Our document review and discussions 
with tourism services providers and agricultural and timber producers 
suggest that these businesses will likely react gradually to any 
increased market opportunities resulting from MCC's projects, in part 
because of constraints to expanding economic activity.[Footnote 25] For 
example: 

* According to tourism officials and business owners, several types of 
activities need to progress to enable industry growth. Factors needed 
to foster tourism growth in Vanuatu include marketing Vanuatu as a 
tourist destination; promoting different types of products to different 
markets; improving domestic and international air and sea access; 
developing or upgrading the electric power, water, and road 
infrastructures; and recruiting and training workers. Tourism service 
providers in Santo and Efate identified improving air capacity as an 
important need. 

* According to an official from the fisheries department in Santo, 
besides rough roads, the lack of ice-making equipment and lack of feed 
are important barriers to developing Santo's small-scale aquaculture. 

* Timber production expands slowly; for example, a timber company owner 
in Santo stated that newly planted trees could not be harvested for 
about 15 years. 

* In both Efate and Santo, feeder roads are in worse condition than the 
project roads and are critical to agriculture shipments.[Footnote 26] 

These constraints suggest that future benefits related to tourism and 
agriculture will be phased in once transportation infrastructure is 
improved. Moreover, MCC assumes all construction costs will be incurred 
in the first year, instead of phasing these costs over the multiyear 
construction schedule. Our analysis shows that if costs are phased over 
3 years and benefits are phased over 5 years, overall compact ERR 
declines from 24.2 percent, as projected by MCC, to 16.5 percent (see 
app. V). 

Project Maintenance after Compact Expiration Cannot Be Ensured: 

Uncertainty about the maintenance of completed transportation 
infrastructure projects after 2011 may affect the compact's projected 
benefits. According to World Bank and ADB officials, continuing donor 
involvement is needed to ensure the maintenance and sustainability of 
completed projects. In addition, during our visits to Efate and Santo, 
tourism and agriculture business representatives cited continued road 
maintenance as a critical concern. However, although MCC has budgeted 
$6.2 million for institutional strengthening of the Vanuatu PWD, MCC 
has no means of ensuring the maintenance of completed projects after 
the compact expires in 2011; the Millennium Challenge Act limits 
compacts to 5 years. 

Although the conditions precedent[Footnote 27] to the Vanuatu compact 
require the government's commitment to ongoing project maintenance, 
World Bank and ADB officials told us, based on their experience with 
Pacific countries, that covenants such as these are difficult to 
enforce after the project is completed and often are not effective in 
ensuring sustainability. According to Vanuatu government officials, 
funds from vehicle registration and related fees are available for 
maintenance of the completed projects; however, the government has not 
dedicated these funds for this purpose. In addition, according to donor 
reporting, the government has failed to sustain maintenance of previous 
donor projects, primarily because of a lack of funds and a shortage of 
resources, skills, and capabilities. Some of the department's equipment 
on Santo Island is more than 27 years old, and newer equipment remains 
idle for long periods because spare parts are scarce; overall, the 
department lacks adequate equipment for Santo's 1,000-kilometer road 
network. Although the compact provides funds for equipment and for 
technical assistance to increase capacity, MCC cannot ensure the 
maintenance of infrastructure after the compact ends. 

Poor maintenance performance will reduce the benefits projected in the 
MCC compact.[Footnote 28] Our analysis shows that with phasing of costs 
and benefits, no large periodic maintenance expenditures, and 
inadequate maintenance performance, overall compact ERR decreases from 
24.2 percent, as projected by MCC, to 16.5 percent with phasing and to 
13.8 percent without maintenance.[Footnote 29] 

Induced Benefits Require Tourism Providers and Agricultural Producers 
to Respond to Opportunities: 

The compact's induced benefits depend on the response of Vanuatu 
tourism providers and agricultural producers. However, constraints 
affecting these economic sectors--such as the air-passenger capacity 
limitations that affect Vanuatu's tourism--may prevent the sector from 
expanding as MCC projects, affecting the reliability of MCC's estimate 
of program benefits. 

Some development assistance organizations do not include induced 
benefits in calculations of ERR. In particular, World Bank officials 
told us they have not considered induced benefits in economic analyses 
of infrastructure for 10 years, because they deem such benefits too 
conjectural and subject to manipulation.[Footnote 30] For example, if a 
projected road is assumed to transform an area's agricultural 
production, it is easy to show large benefits and high ERR.[Footnote 
31] 

Limited response to the compact by tourism providers and agricultural 
producers would have a significant impact on compact benefits. For 
example, our analysis shows that the phasing of costs and benefits 
reduces the overall compact ERR from 24.2 percent to 16.5 percent, and 
the omission of induced benefits (i.e., no producer response) further 
reduces the compact ERR to 5.5 percent. Two projects would have a 
negative ERR. 

Efficiency Gains May Not Cause Measurable Change in Per Capita Income: 

MCC counts efficiency gains--such as time saved because of better 
roads--as compact benefits. However, although efficiency gains could 
improve social welfare, they may not lead to changes in per capita 
income or GDP or be directly measurable as net additions to the 
economy. 

Excluding efficiency gains from ERR calculations reduces the compact's 
overall ERR. For example, our analysis shows that the phasing of costs 
and benefits and the omission of efficiency gains from the Vanuatu ERR 
calculation causes the overall compact ERR to decline from 24.2 percent 
to 16.5 percent with phasing and to 11.8 percent without calculating 
efficiency gains. This analysis also results in a negative ERR for two 
projects (see app. V).[Footnote 32] 

Conclusions: 

MCC obtained the input of knowledgeable stakeholders in projecting the 
benefits, costs, and number of beneficiaries of its compact with 
Vanuatu, which addresses one of Vanuatu's primary constraints to 
economic growth. However, MCC's public statements about these benefits-
-particularly its projection of the compact's effect on per capita 
income--suggest greater impacts than MCC's underlying data and analysis 
support. MCC's statements can be understood only by reviewing 
supporting source documents and spreadsheets, which are not publicly 
available. These gaps could lead to unrealistic expectations about the 
compact's effect within Vanuatu; for example, by suggesting that per 
capita incomes will increase so quickly, MCC suggests that its compact 
will achieve sustainable growth in a way that other donors to Vanuatu 
have not been able to achieve. Further, these gaps between MCC's 
statements and underlying analysis raise questions about other 
compacts' projections of transformative impact on country economies or 
economic sectors. Without an accurate representation of the compacts' 
projected benefits, the extent to which the compacts further MCC's 
goals of poverty reduction, economic growth, and transformative 
development cannot be accurately evaluated. Further, MCC's economic 
analyses for the Vanuatu compact did not fully consider the phasing of 
costs and benefits, such as the time required to improve the 
infrastructure and for the economy to respond to the opportunities from 
the improved infrastructure. Additionally, MCC's analysis did not fully 
account for risks that could substantially reduce compact benefits. 

Recommendations: 

We recommend that the CEO of MCC take the following actions: 

* revise the public reporting of the Vanuatu compact's projected impact 
to clearly represent the underlying data and analysis; 

* assess whether similar statements in other compacts accurately 
reflect the underlying data and analysis; and: 

* improve economic analysis by phasing the costs and benefits in 
compact ERR calculations and by more fully accounting for risks such as 
those related to continuing maintenance, induced benefits, and 
monetized efficiency gains as part of sensitivity analysis. 

Agency Comments and Our Evaluation: 

MCC provided written comments regarding a draft of this report. We have 
reprinted MCC's comments, with our responses, in appendix VI and 
incorporated technical comments from MCC where appropriate. Although 
MCC did not directly acknowledge our recommendations, it questioned our 
finding of a gap between its portrayal of the compact's benefits and 
its underlying analysis. In addition, MCC responded to our discussion 
of beneficiary numbers and program risks. Following is a summary of 
MCC's comments and our evaluation of these comments. 

Compact Impact on Incomes and Growth: 

MCC acknowledged that its use of projected cumulative compact impact on 
income and growth was misleading, but it asserted that (1) it had no 
intention to mislead and (2) its portrayal of projected compact 
benefits was factually correct and consistent with its underlying data. 
Our report does not state or imply that MCC intended to mislead; we 
have not determined why MCC made its statements portraying projected 
benefits on income and growth. Further, MCC's portrayal of the 
projected benefits could be considered factually correct and consistent 
with the underlying data only if the reader knows that the portrayal 
represents cumulative income and growth over 5 years. However, MCC's 
public documents and communications to Congress never describe the 
benefits it projects as being cumulative over 5 years. At issue in this 
finding are both the facts represented by MCC's data and calculations 
and MCC's representation of these data and calculations in its public 
statements. As our report notes, MCC's statements can be understood 
accurately only by reviewing supporting source documents and 
spreadsheets, which are not publicly available. 

In addition, MCC's portrayal of the compact's effect in cumulative 
terms is not consistent with its description of per capita income in 
Vanuatu in annex I of the compact, which states that per capita income 
in Vanuatu declined by 15.4 percent between 1994 and 2003. This 
calculation is consistent with the World Bank's data, which show 
Vanuatu's per capita income in 2003 as approximately 15.4 percent less 
than its per capita income in 1994--a comparison of 2003 and 1994 
income levels, not a reflection of cumulative decline. If the 
methodology MCC uses to describe past per capita income in Vanuatu is 
used to describe the compact's projected future impact, then the 
projected increase in Vanuatu's per capita income in 2010 should be 
portrayed as 3.9 percent, not the cumulative 15 percent that MCC 
presents in its congressional notifications and public documents. 

MCC also commented that its data on the compact's effect on Vanuatu's 
GDP are consistent with its assessment that GDP will be perpetually 3 
percent higher with the MCC investment than without it. We agree that 
MCC's data reflect this, but MCC's portrayal of this benefit in its 
public statements implies an effect on the growth rate, rather than the 
level, of Vanuatu's GDP. MCC couples its portrayal of the GDP benefit 
with a description of the compact's impact as "transformational"--that 
is, as MCC defines the term, having "a dramatic and long-lasting impact 
on poverty reduction through sustainable economic growth." However, 
although MCC's data show projected benefits to Vanuatu's economy, 
including an increase in the level of GDP, the projected change to the 
overall Vanuatu economy, as reflected by the GDP growth rate, is 
smaller than 3 percent. In 2010-2015, for example, the GDP growth rate 
increase resulting from the compact will be 0.1 percent. 

Compact Beneficiaries: 

MCC commented that our report suggests that (1) nonpoor households 
could feasibly be excluded from the count of beneficiaries, (2) persons 
living on off-shore islets and in the hinterland should not be counted 
as beneficiaries, and (3) MCC's definition of the local beneficiary 
population includes urban dwellers. Regarding nonpoor households, we do 
not suggest that they could be excluded or that this would be 
desirable. Our report indicates that MCC's analysis shows there will be 
poor, rural beneficiaries as well as other beneficiaries and that MCC 
has not quantified what portion of the compact's benefits poor, rural 
beneficiaries will receive. Regarding off-shore and hinterland 
beneficiaries, we do question whether inhabitants of off-shore islets 
and of villages more than 10 kilometers from MCC projects on Santo and 
Efate would benefit fully from MCC's projects. However, these persons 
are less than half of the total number we question. In Santo and Efate, 
51 percent of the catchment area population are not inhabitants of the 
hinterland or off-shore islets, but rather residents of Efate, near the 
tourism center and capital of Port Vila and along a portion of the Ring 
Road that is already paved. It is not clear how construction of a road 
elsewhere on the island will benefit these residents when they already 
have access to Port Vila via the existing paved road. Regarding the 
definition of "local population"--which MCC's underlying documentation 
says will receive 43 percent of compact benefits--MCC's written 
comments asserted that its definition of this population does not 
include urban dwellers. Although we asked MCC to support this 
assertion, MCC did not provide any additional documents. In separate 
comments, MCC added that although its documentation does not contain a 
very detailed reference specifically saying exactly how many business 
people, transport providers and tourism operators are foreigners and 
urban, MCC believes it reasonable to conclude that urban dwellers are 
included in these categories. However, we note that not every urban 
dweller is a business person, transport provider, or tourism operator 
and that some urban beneficiaries would therefore fall into the "local 
population" that receives an estimated 43 percent of benefits. Further, 
MCC's definition of the term in its internal documents includes persons 
who are not poor and not rural. 

Program Risks: 

MCC commented that it undertakes sensitivity analysis for all of its 
ERR calculations, although it may focus on risks other than those we 
identified. In addition, MCC states that it differs significantly from 
us regarding the precise nature and severity of program risks in 
general. As our report notes, we offer several alternative scenarios to 
illustrate the maximum impact of certain areas of risk on projected 
compact benefits. The scenarios look at the effect of individual risk 
areas, rather than the possibility of projects' failing to achieve 
certain benefits in multiple areas simultaneously. Regarding induced 
benefits and efficiency gains, we note that MCC's documentation of its 
economic analysis questions the valuing of efficiency gains as induced 
benefits to the Vanuatu economy. Regarding maintenance, MCC states that 
its compact provides a significant amount of money to assist the 
Vanuatu PWD in providing timely and adequate maintenance. However, as 
our report notes, Vanuatu's record of maintenance is poor, and 
achieving a change in maintenance performance through donor assistance 
would show a significant break from past experience. Finally, regarding 
contingencies, MCC questions our use of the Flyvbjerg, Holm, and Buhl 
study as a basis for evaluating its use of contingencies and states 
that it provided us with price and contingency information to support 
its use of contingencies. The Flyvbjerg, Holm, and Buhl study and our 
report use the same basis of comparison for evaluating the adequacy of 
contingencies, and we have updated the language of our report to make 
this clear. In addition, as our report notes, we did not receive full 
documentation of the cost estimates and contingencies that MCC used in 
developing its economic analyses. We received no supporting 
documentation for the cost and contingency estimates for the Santo and 
Efate projects, which account for 56 percent of the total compact 
budget. A key document that MCC's contractor said it used as a basis 
for developing the Santo and Efate costs was unavailable to us, because 
the government of Vanuatu would not provide it to MCC. Cost overruns 
remain a risk to project benefits, and we have presented them as such. 

As agreed with your offices, unless you publicly announce the contents 
of this report earlier, we plan no further distribution until 30 days 
from the report date. At that time, we will send copies of this report 
to interested congressional committees as well as to the CEO of MCC and 
the Secretary of the Treasury. We will make copies available to others 
on request. In addition, this report will be available at no charge on 
the GAO Web site at http://www.gao.gov. 

If you or your staff have any questions about this report, please 
contact David Gootnick at (202) 512-3149 or gootnickd@gao.gov. Contact 
points for our Offices of Congressional Relations and Public Affairs 
may be found on the last page of this report. GAO staff who made major 
contributions to this report are listed in appendix VII. 

Sincerely yours, 

Signed by: 

David Gootnick: 
Director: 
International Affairs and Trade: 

[End of section] 

Appendix I: Objectives, Scope, and Methodology: 

At the request of the Chairman of the House Committee on Foreign 
Affairs, we examined the Millennium Challenge Corporation's (MCC) 
economic analyses of the Vanuatu compact. Specifically, we examined: 

* MCC's methods of projecting the compact's economic benefits and 
methods of calculating the benefits, 

* MCC's portrayal and analysis of the projected benefits, and: 

* risks that could affect the compact's impact on poverty reduction and 
economic growth. 

To accomplish our objectives, we reviewed MCC's record of due 
diligence, including its opportunity memo, investment memo, due 
diligence book, economic analysis spreadsheets, and contractor reports. 
These documents are restricted from public dissemination based on MCC 
policy, but MCC made them available to us for analysis. We also 
reviewed the supporting documents used in MCC's due diligence process, 
such as previous donor reports and studies. We supplemented this review 
with interviews with MCC officials in Washington, D.C., and MCA-Vanuatu 
officials in Vanuatu. We then used this information to (1) summarize 
the process that MCC used to develop its economic model and (2) 
identify the logic, data, methods, and assumptions used to determine 
the compact's projected costs and benefits, number of beneficiaries, 
and effects on per capita income, gross domestic product (GDP), and 
poverty. 

We evaluated MCC's statements of compact impact in its public documents 
such as the congressional notification and the compact itself--as well 
as MCC's internal investment memo--by comparing these statements with 
the underlying analyses and data used to support them. We analyzed the 
November 2005 investment memo spreadsheet containing the source data 
and calculations used for MCC's statements about per capita income and 
GDP and reviewed its contents and formulas to determine the methodology 
and source data for MCC's statements and to assess the accuracy of the 
calculations. We then used MCC's own data and formulas from the 
November 2005 investment memo to determine MCC's impact in individual 
years and compare the results with the statements made in the Vanuatu 
compact and congressional notification. We could not validate most of 
MCC's underlying data and assumptions, because some data, such as 
construction and maintenance cost data, were not available or could not 
be checked within the time frames of our engagement. Further, we used 
the calculations and formulas from MCC's November 2005 investment memo, 
corrected population figures, and the data from MCC's April 2006 final 
economic analysis to recalculate MCC's statements about per capita 
income impacts. We reviewed MCC's economic analysis spreadsheet to find 
the amount of benefits that MCC's analysis assumed would flow to poor, 
rural beneficiaries and reviewed spreadsheets provided by MCA-Vanuatu 
to determine the assumptions, methodology and data used to calculate 
the beneficiary population in Santo and Efate. We compared the 
spreadsheet data with Vanuatu village maps and population figures from 
the 1999 National Population and Housing Census provided by MCA-Vanuatu 
to evaluate MCC's calculations and assumptions about the villages 
included as beneficiaries. 

We identified risks to MCC's compact results based on our review of 
MCC's internal documentation and donor and academic literature. We also 
met with Vanuatu, MCC, and contractor officials and interested parties 
such as tourism and agriculture business owners in Vanuatu. We focused 
our field work on the three projects representing more than half of the 
compact budget and on Vanuatu's two most populous islands, Santo and 
Efate. We examined MCC's due diligence documentation and contacted 
MCC's contractor to determine the methodology used to estimate 
construction costs. We then compared MCC's procedures to findings and 
GAO's best practice criteria for cost estimation. We compared MCC's 
assumptions about maintenance continuing for 20 years to previous donor 
experiences gleaned from donor reporting or interviews with the Asian 
Development Bank, Australian Agency for International Development, 
European Union, International Monetary Fund, New Zealand Agency for 
International Development, and World Bank. We determined the need for 
the phasing of benefits based on discussions with knowledgeable 
agriculture and tourism representatives in Vanuatu and a review of 
MCC's source documentation. We further compared MCC's treatment of 
induced benefits to the approach taken by the World Bank and MCC's 
treatment of direct benefits to economic principles about the 
monetization of efficiency gains. We modeled risk scenarios that 
provide alternative methods of looking at compact benefits using MCC's 
April 2006 economic model spreadsheet. In modeling these risks, we used 
the data from MCC's economic analyses; we did not validate these data. 
In these scenarios, we determined the effect of the scenario on the 
economic rate of return by assuming the phasing in of costs over 3 
years and of benefits over 5 years, first determining the effect of 
phasing alone, then examining the phasing of costs and benefits coupled 
with the following: 

* Lack of maintenance--eliminating all future large maintenance 
expenditures and taking away a fraction of the growing expected 
benefits each year, so that at the end of the 20-year period, 2027, 
benefits basically return to their 2012 level. 

* Lack of induced benefits--deleting those benefits from the cost 
model. 

* Lack of monetized efficiency gains--deleting those benefits from the 
cost model. 

[End of section] 

Appendix II: Prior Development Assistance to Vanuatu: 

Of Pacific island countries, Vanuatu receives among the largest amounts 
of aid; however, Vanuatu's economic performance and social development 
since 1990 have lagged behind other countries in the region. Vanuatu 
depends on donor assistance for its development budget; from its 
gaining independence, in 1980, to 2002, aid receipts averaged around 17 
percent of GDP. In 1997, Vanuatu initiated structural adjustment 
efforts with ADB assistance designed to stimulate the private sector 
and help realize the country's growth potential in agriculture and 
tourism. However, these reforms have had a limited impact on growth in 
private-sector investment and GDP. In 2003, the government of Vanuatu 
prepared the Priorities and Action Agenda to focus available resources, 
including donor assistance, on Vanuatu's development priorities. The 
current strategies of Vanuatu's major donors--which include Australia, 
France, New Zealand, the European Union, Japan, and MCC--are aimed at 
strengthening Vanuatu's productive sectors and supporting private 
sector-led development and are based on the Vanuatu government's stated 
goals. MCC anticipates that its average annual assistance level of 
$13.1 million over the next 5 years will position the United States as 
one of Vanuatu's top two official donors (see table 1). 

Table 1: Average Annual Contributions of Top Five Donors to Vanuatu in 
2004 and 2005: 

(Dollars in millions). 

Donor: Australia; 
Average annual contribution: $19.1. 

Donor: France; 
Average annual contribution: $5.3. 

Donor: New Zealand; 
Average annual contribution: $4.8. 

Donor: European Union; 
Average annual contribution: $4.6. 

Donor: Japan; 
Average annual contribution: $3.5. 

Source: Organization for Economic Cooperation and Development. 

[End of table] 

According to the ADB, Vanuatu's development budget has concentrated on 
building new capital infrastructure; however, according to donor 
reporting, these assets have not been adequately maintained. From 1996 
to 2004, Vanuatu received about $35 million in assistance for 
transportation infrastructure projects including the construction and 
rehabilitation of roads, bridges, and wharfs, and the extension and 
upgrading of airport terminals and runways. 

According to studies by the World Bank and ADB, Vanuatu faces 
significant development challenges. Some of Vanuatu's challenges are 
shared by other small states and island economies,[Footnote 33] while 
others are either the result of, or exacerbated by, political 
instability and, according to ADB, inappropriate policies or 
regulations. Like other small island economies, Vanuatu's agriculture 
and tourism sectors are sensitive to extreme weather conditions and 
natural disasters. For example, frequent cyclones cause production 
shocks and restrict the range of viable crops and tree species. 
Vanuatu's narrow production base also leaves the economy open to 
economic shocks from changes in commodity prices, especially for copra. 
Despite these constraints, Vanuatu has a wealth of natural resources 
and growth opportunities in agriculture, fisheries, and tourism. 
However, growth has been hindered by substantial barriers to private- 
sector development, including political uncertainty,[Footnote 34] high 
costs of doing business, poor and costly infrastructure, lack of a 
secured transactions framework,[Footnote 35] and issues with land 
tenure. 

[End of section] 

Appendix III: MCC Estimates of Per Capita Income Impact and Poor, Rural 
Beneficiaries: 

Per Capita Income Impact: 

In its economic analysis for Vanuatu, MCC did not correctly account for 
population growth in projecting per capita income and incorrectly 
deflated the value of the benefit stream that was already in real 
terms. When the benefit stream is no longer deflated, its value 
increases. However, when population growth is accounted for, the per 
capita income benefit is reduced. MCC assumes that the population is 
growing by approximately 2.6 percent each year, but MCC's analysis of 
cumulative income per capita benefits for 2006 to 2015 is based on 2005 
population estimates, rather than on an estimate of population that 
increases each year. Figure 11 contrasts the estimated increase in 
income per capita over the 2005 baseline in the given year with and 
without our corrections to deflating of benefits and to population 
growth. The errors largely cancel out, reducing the per capita income 
benefit projected by MCC from $51 (3.9 percent) to $49 (3.7 percent) in 
2010 and from $61 (4.6 percent) to $57 (4.3 percent) in 2015. 

Figure 11: MCC Compact's Projected Impact on Vanuatu Per Capita Income 
with and without Adjusted Population Estimates: 

[See PDF for image] 

Source: GAO analysis of MCC data. 

Note: In addition to adjusting for population growth and eliminating 
the deflating of benefits already in real terms, our revised 
projections of per capita income benefits use April 2006 data from MCC, 
including a benefit start date of 2007 for construction spending, a 
benefit start date of 2008 for some projects, and updated numeric 
values for costs and benefits. 

[End of figure] 

The impact of the MCC compact on per capita income is a relatively 
small addition to Vanuatu's expected per capita income without MCC. 
Figure 12 shows, in dollar terms, MCC's projected impact on Vanuatu's 
per capita income compared with the expected levels of per capita 
income, using MCC's assumptions of 3 percent annual growth in GDP and 
2.6 percent in population. 

Figure 12: MCC Compact's Projected Impact on Vanuatu Per Capita Income 
in Real Terms Relative to Per Capita Income without MCC: 

[See PDF for image] 

Source: GAO analysis of MCC data. 

Note: Assumes 0.4 percent growth in per capita income, based on MCC's 
assumed GDP growth of 3 percent without the compact and MCC's assumed 
Vanuatu population growth rate of 2.6 percent. 

[End of figure] 

Calculation of Beneficiaries: 

MCC calculated approximately 65,000 beneficiaries on the eight islands 
receiving MCC projects (see table 2).[Footnote 36] However, MCC's 
determination of beneficiary numbers in Santo and Efate contained 
calculation errors and relied on problematic assumptions that villages 
near portions of the road not improved by MCC, as well as those on off- 
shore islets, would fully benefit from the compact. Correcting the 
calculation and data errors and fully discounting MCC's assumptions 
would reduce the beneficiary count on Efate and Santo by 32 percent, 
from MCC's stated 26,553 to 18,070. 

Table 2: Number of Poor, Rural Beneficiaries of Vanuatu Compact, as 
Calculated by MCC: 

Island: Efate; 
Project: Round Island Road; 
Number of beneficiaries: 13,819. 

Island: Santo; 
Project: Port Olry Road; 
Number of beneficiaries: 7,404. 

Island: Santo; 
Project: South Coast Road Bridges; 
Number of beneficiaries: 5,330. 

Subtotal, Santo and Efate; 
Project: [Empty]; 
Number of beneficiaries: 26,553. 

Other projects: 

Island: Tanna; 
Project: Whitesands Road; 
Number of beneficiaries: 6,190. 

Island: Malekula; 
Project: Lits Lits Road; 
Number of beneficiaries: 5,676. 

Island: Malekula; 
Project: Southwest Bay Airstrip; 
Number of beneficiaries: 4,570. 

Island: Pentecost; 
Project: Loltong Wharf and North-South Road; 
Number of beneficiaries: 7,858. 

Island: Epi; 
Project: Lamen Bay Wharf; 
Number of beneficiaries: 1,560. 

Island: Malo; 
protect: Malo Roads Upgrading; 
Number of beneficiaries: 3,480. 

Island: Ambae; 
Project: Ambae Roads Reconstruction; 
Number of beneficiaries: 9,340. 

Total; 
Project: [Empty]; 
Number of beneficiaries: 65,227. 

Source: GAO analysis of MCC data. 

[End of table] 

Our examination of the calculation of MCC's beneficiary estimate for 
the Efate Ring Road, Santo East Coast Road, and Santo South Coast Road 
Bridges shows that MCC's estimate of rural beneficiaries in Santo and 
Efate would be increased by approximately 5,100 persons, or 19 percent, 
if adjustments were made to correct data errors[Footnote 37] (see table 
3). 

Table 3: Effect of Calculation and Data Errors on Beneficiary 
Population in Efate and Santo: 

MCC Statement: beneficiary count in monitoring and evaluation plan; 
Efate Ring Road: 13,819; 
Santo East Coast Road: 7,404; 
Santo South Coast Road Bridges: 5,330; 
Total: 26,553. 

GAO recalculation of MCC data[A]; 
Efate Ring Road: 15,439; 
Santo East Coast Road: 7,518; 
Santo South Coast Road Bridges: 8,732; 
Total: 31,689. 

Difference; 
Efate Ring Road: 1,620; 
Santo East Coast Road: 114; 
Santo South Coast Road Bridges: 3,402; 
Total: 5,136. 

Source: GAO analysis of MCC data. 

[A] GAO's recalculation uses 1999 Vanuatu census data, removes double- 
counted villages, and updates the population to 2005 using MCC's 
assumed population growth factor in rural areas of 2 percent per year. 

[End of table] 

MCC also made problematic assumptions about the geographic reach of 
project benefits that increased the number of beneficiaries by 
including more rural areas as benefiting from the project. Fully 
discounting these assumptions would reduce the beneficiary count from 
our adjusted total of 31,689 to 18,070 (see table 4). 

Table 4: Effect of MCC Assumptions on Beneficiary Population in Efate 
and Santo: 

GAO calculation using MCC data; 
Efate Ring Road: 15,439; 
Santo East Coast Road: 7,518; 
Santo South Coast Road Bridges: 8,732; 
Total: 31,689. 

Discounting population of areas not adjacent to MCC road project; 
Efate Ring Road: (6,889); 
Santo East Coast Road: 0; 
Santo South Coast Road Bridges: (3,222); 
Total: (10,111). 

Discounting population of off-shore islets; 
Efate Ring Road: (2,779); 
Santo East Coast Road: (196); 
Santo South Coast Road Bridges: (533); 
Total: (3,508). 

Total[A]; 
Efate Ring Road: 5,770; 
Santo East Coast Road: 7,322; 
Santo South Coast Road Bridges: 4,978; 
Total: 18,070. 

Source: GAO analysis of MCC data. 

[A] Because of rounding, numbers in columns may not sum to totals. 

[End of table] 

In estimating beneficiaries of both the Efate Ring Road and the Santo 
South Coast Road Bridges projects, MCC included the populations of 
villages not located near MCC's road projects and populations that face 
geographic barriers to using the projects. These population counts were 
not prorated or adjusted to account for the barriers that some villages 
face in realizing benefits from MCC's projects. 

* In Efate, MCC assumed that the populations of villages located along 
an already improved portion of the road adjacent to Port Vila in the 
Mele, Eratap, and Erakor areas--which will not be addressed by the MCC 
compact--would benefit from construction of the road elsewhere on the 
island (see fig. 13). MCA-Vanuatu stated that the populations of these 
villages were included because they own land where the road will be 
constructed and most of the villagers' gardens are located in these 
areas. (The source for this assumption is not documented.) The 
inhabitants of these villages represent 45 percent of the total number 
(15,439) of poor, rural Efate beneficiaries. MCC also did not account 
for "most" villagers and instead included the entire village population 
in these areas in its totals. 

Figure 13: MCC Efate Ring Road Catchment Area: 

[See PDF for image] 

Source: GAO based on MCC data; Vanuatu government (map). 

[End of figure] 

* For the Santo South Coast Road Bridges, MCC included villages in West 
Santo more than 10 kilometers from the bridge projects and villages 
across a river that vehicles can cross only by fording; after heavy 
rains, the river becomes impassable. MCC's compact program does not 
include a bridge across this river. Using a 2005 population estimate, 
we determined that MCC included 3,222 people in the count of 
beneficiaries of South Coast Road Bridges based on this assumption--37 
percent of the total number (8,732) of South Coast Road Bridges 
beneficiaries. 

For all three projects, MCC assumed that inhabitants of off-shore 
islets would fully benefit, although they are not connected to the 
improved roads and must travel by boat to reach them. MCC officials 
told us that they expect that residents of off-shore islets will 
benefit from increased tourism spending after construction of roads and 
would benefit from decreased transportation costs after coming to the 
mainland. Using 2005 population estimates, we determined that 2,779 
persons (18 percent of the Efate Ring Road beneficiary population) live 
on off-shore islets. For the Santo East Coast Road and South Coast Road 
Bridges, respectively, 196 (3 percent of the beneficiary population) 
and 533 (6 percent of the beneficiary population) live on off-shore 
islets. 

[End of section] 

Appendix IV: MCC Cost Estimates for the Vanuatu Compact: 

The reliability of MCC's cost estimates for the Vanuatu compact 
construction cannot be evaluated, because MCC was unable to fully 
provide the data that it used to produce them. MCC's contractor based 
its construction cost estimates for MCC's transportation infrastructure 
activities primarily on two sources. To build cost estimates for seven 
of the compact projects, representing 15 percent of total compact cost, 
the contractor used a 2003 analysis it had performed for the ADB. To 
prepare cost estimates for the three projects on Santo and Efate, 
representing 56 percent of compact cost, the contractor used estimates 
prepared by a second contractor for the government of Vanuatu.[Footnote 
38] 

* For the seven ADB projects, MCC's file of documents used in its due 
diligence included the original cost estimate but did not include an 
analysis that showed how these costs were updated to account for 
inflation and changes in the scope of the work. MCC's documentation 
states that these previous costs were multiplied by 1.17 to account for 
inflation and also adjusted for changes in the scope.[Footnote 39] 

* For the three projects on Santo and Efate, MCC's file did not include 
any source information or analysis. MCC's contractor later told us that 
it derived the costs of the projects on Santo and Efate from an 
existing report by a second contractor to the government of Vanuatu, 
dated approximately 2004. MCC's contractor told us that because MCC's 
scope of work was more realistic than that proposed in 2004, which 
included realigning portions of the roadways, the MCC cost estimate was 
prepared anew using source information from the previous report. We 
requested, but did not receive, a copy of this report to the government 
of Vanuatu and a copy of the analysis showing how MCC developed its 
cost estimates using this source data. According to MCC officials, the 
government of Vanuatu was not willing to provide the report to MCC for 
our review. 

According to previous GAO work, high-quality, reliable cost estimates 
should be accurate, comprehensive, well-documented, and 
validated.[Footnote 40] MCC cost estimates used in the economic 
analyses have shortcomings in each of these areas. 

* Accurate. The source data for the estimates is from 2003 and 
approximately 2004, and changes in the cost environment beyond general 
inflation may not be accounted for. MCC also incorrectly calculated the 
length of the Santo East Coast Road as 70 kilometers in its economic 
analyses, instead of the 55 kilometers found by a PWD survey, according 
to a PWD official. Other elements of this criterion cannot be 
independently assessed without the source data for the cost estimates 
used in the model. 

* Comprehensive. We could not assess the cost estimates against this 
criterion without the source data for the estimates used in the model. 

* Well-documented. MCC did not fully document any cost estimation 
procedures and checks that it performed as part of its due diligence. 
The due diligence documents we received from MCC contain a narrative 
describing the cost estimate procedures, but no documentation showing 
the analysis that MCC's contractor performed to update previous cost 
estimates. 

* Validated. MCC's due diligence documentation contains no evidence 
that it obtained an independent cost estimate to verify the work of its 
contractor in updating its own estimate for the ADB projects and 
reworking the estimates of another contractor for the Efate and Santo 
projects. An independent cost estimate provides the estimator with an 
unbiased test of the reasonableness of the estimate and reduces the 
cost risk associated with the project by demonstrating that alternate 
methods generate similar results. 

[End of section] 

Appendix V: Illustrative Alternative Calculations of Vanuatu Compact 
Impact: 

This appendix describes the results from several alternative scenarios, 
which disaggregate the effects of the various types of benefits on the 
overall ERR, GDP and per capita income in order to illustrate the 
significance of each of the components. 

MCC reported different values of summary statistics in the investment 
memo dated November 2005 as compared with the final analysis that 
reported more recent data (see table 5). 

Table 5: Summary of Compact Impacts as Presented by MCC, with GAO 
Recalculations: 

MCC's anticipated effect as presented in investment memo; 
Compact ERR: 24.7 percent; 
Increase in level of GDP with MCC in year 5 vs. GDP without MCC in year 
5: 3.2 percent; 
Increase in level of real income per capita in 2010 vs. 2005 baseline: 
3.9 percent[A]. 

MCC's anticipated effect using corrected or updated data; 
Compact ERR: 24.2 percent[B]; 
Increase in level of GDP with MCC in year 5 vs. GDP without MCC in year 
5: 3.2 percent; 
Increase in level of real income per capita in 2010 vs. 2005 baseline: 
3.7 percent[C]. 

Source: GAO analysis of MCC data. 

[A] Derived from MCC's own data and calculations. 

[B] In its updated and final April 2006 economic analysis, MCC adjusted 
the 24.7 percent reported in its congressional notification downward 
slightly to 24.2 percent. 

[C] GAO calculation after correcting MCC calculation errors. 

[End of table] 

To illustrate the maximum impact of certain areas of risk on projected 
compact benefits, we modified the model using MCC data for different 
scenarios. We phased costs and benefits to mirror more accurately the 
rate of construction and accrual of potential benefits. Phasing costs 
has the effect of increasing ERR, while phasing benefits decreases it; 
the net effect is to decrease ERR. Within this modified framework that 
includes phasing, we eliminated each main type of benefit--direct and 
induced--one at a time to understand their relative importance. 

* For "no induced benefits," we eliminated all induced benefits from 
expanded tourism, agriculture, or land development as a result of the 
compact but did not change the direct benefits. 

* For "efficiency gains are not monetized," we eliminated all direct 
benefits such as cost savings for existing users of a road or benefits 
from newly generated traffic, which are attributable to the project, 
but did not change the induced benefits. 

Finally, we examined the impact of poor maintenance performance after 
compact expiration in 2011 on total benefits from the phased project. 
We modeled this by eliminating all future large maintenance 
expenditures and taking away a fraction of the growing expected 
benefits each year, so that at the end of the 20-year period, 2027, 
benefits basically return to their 2012 level. 

These scenarios present illustrative cases and should be considered as 
extremes of a possible range of cases that capture uncertainty effects 
on summary statistics, such as the ERR, GDP, and income per capita. MCC 
has observed that overall benefits may understate the true impact of 
the compact because spillover effects, such as easier access to schools 
and medical facilities due to the improved infrastructure, are not 
captured by these measures. These effects represent long-term benefits 
to society as whole, which are not accounted for by the analysis 
performed by either MCC or GAO. 

Table 6: Summary of Compact ERR, GDP, and Per Capita Income impacts 
under Alternative Scenarios of Accounting for Risks to Benefits: 

GAO analysis[A]: (1) Costs are phased over 3 years and benefits are 
phased over 5 years; 
Compact ERR: 16.5 percent; 
Increase in GDP with MCC in year 5 relative to GDP without MCC in year 
5: 1.1 percent; 
Increase in real income per capita level in year 5 relative to 2005 
baseline: 1.4 percent. 

Costs are phased over 3 years and benefits are phased over 5 years, 
and: (2) induced benefits are not realized[B]; 
Compact ERR: 5.5 percent; 
Increase in GDP with MCC in year 5 relative to GDP without MCC in year 
5: 0.7 percent; 
Increase in real income per capita level in year 5 relative to 2005 
baseline: 0.8 percent. 

Costs are phased over 3 years and benefits are phased over 5 years, 
and: (3) efficiency gains are not monetized[C]; 
Compact ERR: 11.8 percent; 
Increase in GDP with MCC in year 5 relative to GDP without MCC in year 
5: 0.8 percent; 
Increase in real income per capita level in year 5 relative to 2005 
baseline: 1.0 percent. 

Costs are phased over 3 years and benefits are phased over 5 years, 
and: (4) large-scale maintenance is not undertaken[D]; 
Compact ERR: 13.8 percent; 
Increase in GDP with MCC in year 5 relative to GDP without MCC in year 
5: 1.1 percent; 
Increase in real income per capita level in year 5 relative to 2005 
baseline: 1.3 percent. 

Source: GAO analysis of MCC data. 

[A] In our analysis, benefits start in 2010 and are phased in equal 
increments over 5 years, from 2010 to 2014, with phasing completed by 
year 5. Costs are phased over 3 years to reflect projected timing of 
construction. 

[B] In addition to phasing benefits and costs, we eliminated induced 
effects of the project on agriculture, tourism, fisheries, and the 
development of subdivided beachfront land. 

[C] In addition to phasing benefits and costs, we eliminated road user 
cost savings and savings from wasted surface trips, lost trips, longer 
diversions, and enforced longer trips from road closures. 

[D] In addition to phasing benefits and costs, we assumed that total 
benefits will increase, peak, and decrease such that their value in 
2027 will equal their original value in 2012. The large capital outlays 
for road rehabilitation in 2017 and 2026 in Santo and Efate have been 
eliminated. 

[End of table] 

Table 7 reports the ERR for individual projects as modified in our 
scenarios. 

Table 7: Summary of Selected Compact Project Impacts under Alternative 
Scenarios Accounting for Risks to Benefits: 

Projects: Overall compact; 
MCC's anticipated ERR: 24.2 percent; 
Costs are phased over 3 years and benefits are phased over 5 years: 
16.5 percent; 
Costs are phased over 3 years, benefits are phased over 5 years, and: 
Induced benefits are not realized: 5.5 percent; 
Costs are phased over 3 years, benefits are phased over 5 years, and: 
Efficiency gains are not monetized: 11.8 percent; 
Costs are phased over 3 years, benefits are phased over 5 years, and: 
Large-scale maintenance is not undertaken: 13.8 percent. 

Projects: Efate: Ring Road; 
MCC's anticipated ERR: 20.6 percent; 
Costs are phased over 3 years and benefits are phased over 5 years: 
13.6 percent; 
Costs are phased over 3 years, benefits are phased over 5 years, and: 
Induced benefits are not realized: negative; 
Costs are phased over 3 years, benefits are phased over 5 years, and: 
Efficiency gains are not monetized: 10.8 percent; 
Costs are phased over 3 years, benefits are phased over 5 years, and: 
Large-scale maintenance is not undertaken: 11.2 percent. 

Projects: Santo: Port Olry Road; 
MCC's anticipated ERR: 33.8 percent; 
Costs are phased over 3 years and benefits are phased over 5 years: 
22.6 percent; 
Costs are phased over 3 years, benefits are phased over 5 years, and: 
Induced benefits are not realized: 10.3 percent; 
Costs are phased over 3 years, benefits are phased over 5 years, and: 
Efficiency gains are not monetized: 15.5 percent; 
Costs are phased over 3 years, benefits are phased over 5 years, and: 
Large-scale maintenance is not undertaken: 19.4 percent. 

Projects: Santo: South Coast Bridges; 
MCC's anticipated ERR: 24.3 percent; 
Costs are phased over 3 years and benefits are phased over 5 years: 
14.3 percent; 
Costs are phased over 3 years, benefits are phased over 5 years, and: 
Induced benefits are not realized: -4.0 percent; 
Costs are phased over 3 years, benefits are phased over 5 years, and: 
Efficiency gains are not monetized: 12.4 percent; 
Costs are phased over 3 years, benefits are phased over 5 years, and: 
Large-scale maintenance is not undertaken: 10.8 percent. 

Projects: Tanna: Whitesands Road; 
MCC's anticipated ERR: 17.7 percent; 
Costs are phased over 3 years and benefits are phased over 5 years: 
12.1percent; 
Costs are phased over 3 years, benefits are phased over 5 years, and: 
Induced benefits are not realized: 8.6 percent; 
Costs are phased over 3 years, benefits are phased over 5 years, and: 
Efficiency gains are not monetized: -0.3 percent; 
Costs are phased over 3 years, benefits are phased over 5 years, and: 
Large-scale maintenance is not undertaken: 8.4 percent. 

Projects: Malekula: Lits Lits Road; 
MCC's anticipated ERR: 22.5 percent; 
Costs are phased over 3 years and benefits are phased over 5 years: 
15.1 percent; 
Costs are phased over 3 years, benefits are phased over 5 years, and: 
Induced benefits are not realized: 15.1 percent; 
Costs are phased over 3 years, benefits are phased over 5 years, and: 
Efficiency gains are not monetized: negative; 
Costs are phased over 3 years, benefits are phased over 5 years, and: 
Large-scale maintenance is not undertaken: 14.3 percent. 

Projects: Malekula: South West Bay Airstrip; 
MCC's anticipated ERR: 11.4 percent; 
Costs are phased over 3 years and benefits are phased over 5 years: 6.3 
percent; 
Costs are phased over 3 years, benefits are phased over 5 years, and: 
Induced benefits are not realized: 1.1 percent; 
Costs are phased over 3 years, benefits are phased over 5 years, and: 
Efficiency gains are not monetized: 6.1 percent; 
Costs are phased over 3 years, benefits are phased over 5 years, and: 
Large-scale maintenance is not undertaken: 2.4 percent. 

Projects: Pentecost: Loltong Wharf/N-S Road; 
MCC's anticipated ERR: 15.6 percent; 
Costs are phased over 3 years and benefits are phased over 5 years: 8.9 
percent; 
Costs are phased over 3 years, benefits are phased over 5 years, and: 
Induced benefits are not realized: 5.6 percent; 
Costs are phased over 3 years, benefits are phased over 5 years, and: 
Efficiency gains are not monetized: 4.4 percent; 
Costs are phased over 3 years, benefits are phased over 5 years, and: 
Large-scale maintenance is not undertaken: 4.7 percent. 

Source: GAO analysis of MCC data. 

[End of table] 

[End of section] 

Appendix VI: Comments from the Millennium Challenge Corporation: 

Note: GAO comments supplementing those in the report text appear at the 
end of this appendix. 

Millennium Challenge Corporation: 

June 8, 2007: 

Mr. David B. Gootnick, Director: 
International Affairs and Trade: 
Government Accountability Office: 
441 G Street NW: 
Washington, DC 20548: 

Dear Mr. Gootnick: 

We appreciate the opportunity to review and comment on your draft 
report, "Vanuatu Compact Overstates Projected Program Impact." 

The enclosed Millennium Challenge Corporation views are provided for 
incorporation with this letter as an appendix to the final report. 

If you have any questions concerning this response, please contact Kick 
Stilgenbauer, Program Officer. Department of Congressional and Public 
Affairs, at (202) 521-3629. 

Sincerely, 

Signed by: 

Frances C. McNaught: 
Vice President: 
Congressional and Public Affairs: 

Millennium Challenge Corporation Comments on GAO Draft Report: 

Vanuatu Compact Overstates Projected Program Impact: 

General: 

The Millennium Challenge Corporation (MCC) thanks the Government 
Accountability Office (GAO) for their dedication and care in producing 
this audit of the MCC Compact with Vanuatu, and in particular their 
investment of time and resources to see conditions on the ground 
firsthand. We are pleased with the many opportunities that we had for 
frank and open exchanges of views with GAO staff, in a spirit of mutual 
cooperation, MCC maintains that the Vanuatu Compact will benefit 
significantly the county's poorest people and positively transform the 
economy, 

At issue between the GAO and the MCC is the degree of benefit. The 
language in the MCC Compact reads: "The Transport Infrastructure 
Project is expected to have a transformational impact oil Vanuatu's 
economic development, increasing average income per capita (in real 
terms) by approximately $200 or 15% of current income per capita by 
2010. 

The GAO report states: "This statement suggests that per capita income 
in 2010 and 2015 will be, respectively, 15 percent and 37 percent 
higher than they would be without the compact; however, MCC's data and 
calculations show that this projected impact on per capital income is 
cumulative rather than annual, relative to 2005 per capita income. " 
(page 3, GAO draft report) " 

In short, the GAO report asserts that the MCC statement "suggests" a 
larger increase in annual income while the underlying data show that it 
is a cumulative increase. This suggestion then becomes the basis for 
the headline of the report which is repeated multiple times throughout 
the body of the GAO report. 

We understand the basis for the GAO assertion, although we have 
emphasized repeatedly in discussions with GAO staff that the key 
statements in question were factually correct, and that it was in no 
way our intention to overstate benefits. That said, in all future MCC 
reports, we will use wording that cannot be subject to the 
misinterpretation that the GAO attributes to MCC in this case. MCC 
notes that the GAO presentation of its own assertion, in word and 
graphics, is itself likely to mislead readers. 

Compact impact on incomes and growth: 

The summary page presents a graphic entitled, "Comparison of MCC 
Vanuatu Compact Per Capita Income Effect Stated in Congressional 
Notification to Effect Supported by Underlying Data." The caption in 
the legend of this diagram incorrectly suggests that MCC erred 
factually in describing the cumulative benefits of the program after 
five years. Specifically. MCC stated that per capita income would 
increase by a cumulative amount of 15,4% by 2010 relative to the 2005 
baseline. This is, in fact, how MCC understood the underlying data in 
2005. However. the GAO includes all additional figure in the graphic 
with the caption, "MCC's underlying data," that would lead a reasonable 
reader to believe that the cumulative growth over these five years 
would be only 3,9%. The underlying data do not support the conclusion 
that cumulative growth over five years would be only 3.9%, MCC accepts 
the GAO's assertion that the use of five-year cumulative figures is 
misleading and that annualized figures should be used, but we object to 
the characterization of the Compact figures as inconsistent with the 
underlying data. 

A second example given by the GAO, cited on page three, faults the MCC 
for its claim that "Vanuatu's GDP will increase by 'an additional three 
percent a year." The underlying data are consistent with the assessment 
by MCC that Vanuatu's GDP would be perpetually three percent higher 
with the MCC investment than without, The data also indicate that the 
effect on the per capita GDP growth rate will be moderate but rising, 
albeit much smaller than three percentage points. GAO interpreted the 
statement about three percent higher GDP as indicating that the growth 
rate itself would be three percentage points higher, and we take the 
point that a reasonable reader might make this error. We will certainly 
strive to reduce potential ambiguity in official language in future 
documentation. However, we maintain that the original language was 
factually correct and in no way intended to mislead. 

Compact beneficiaries: 

GAO suggests on page four that it is misleading to indicate that the 
Compact is "expected to benefit approximately 65,000 poor, rural 
inhabitants, [because] these 65,000 beneficiaries will not receive the 
majority of the benefits,," This suggests a significant difference 
between MCC and GAO, We expect that near-and non-poor households will 
benefit from the Compact in addition to poor households. GAO's 
presentation suggests that such households (local business people, 
government personnel, transport providers, and tourism service 
providers) could feasibly be excluded and that this would be desirable, 
We know of no growth strategy by which this would be possible for a 
road project. If MCC is true to its mandate of targeting economic 
growth as a means of reducing poverty, it is inevitable that households 
above the poverty line (some not so very far above the poverty line) 
will also benefit significantly from MCC investments. While the 65,000 
poor, rural beneficiaries cited are not the exclusive beneficiaries of 
the projects, they will still benefit significantly nonetheless. 

GAO further argues that the total number of rural beneficiaries 
included in MCC's analysis should be reduced by one-third. This is 
based on the assumption that households living in the hinterland away 
from the road and also those living on islets off the coast of the main 
islands will not be beneficiaries. We dispute this assumption on the 
grounds that these individuals certainly will benefit from greater 
traffic along the MCC-funded roads. Households that live inland or on 
islets are connected to the greater economy of Vanuatu, and MCC-funded 
roads make those connections all the more vibrant and growth-enhancing. 
Naturally, those households closer to the road may benefit more in 
absolute terms. In counting beneficiaries, however, we have not 
suggested that benefits are distributed equally across beneficiaries. 
Variations do exist, and this is unavoidable. 

The GAO report also asserts on page 20 that MCC fails to identify the 
share of benefits that go to rural poor, of the 43% of benefits that 
accrue to the "local population," MCC's definition of "local 
population" does not include urban dwellers, and we have a reasonable 
expectation that the majority of the 65,000 are poor. 

Program risks - general: 

We appreciate the care and attention the GAO has placed on our analysis 
of the economic returns of MCC investments, As a means of checking the 
robustness of our economic analysis, GAO has integrated several risks 
that it believes important into a sensitivity analysis of the projected 
economic rates of return (ERRS), Sensitivity analysis is an important 
exercise that we undertake for all of our ERR calculations, although 
the focus may be on risks other than those identified by the GAO. While 
MCC accepts the concerns raised by several risks highlighted by GAO, 
including phasing of benefits, project start times, and sustainability, 
we differ significantly on the precise nature and severity of risks 
related to induced benefits, efficiency gains, maintenance and 
contingencies, MCC believes that taken as a whole the judgments that 
were made with respect to risks and assumptions, and the sensitivity 
analysis of them, were reasonable, The following discussions highlight 
MCC's different perceptions of risks discussed in the GAO report. 

Risks - Induced benefits: 

As GAO states on page five, "the projected induced benefits from 
expanded tourism and agriculture depend on businesses and rural 
inhabitants responding to opportunities created by improved 
infrastructure." GAO views the speculative nature of these induced 
benefits as problematic despite its own admission that "[GAO] fieldwork 
and meetings in Vanuatu generally affirmed MCC's assumptions about 
benefits." MCC aims to catalyze economic growth by providing 
opportunities to which beneficiaries can respond. Although we 
acknowledge the difficulty of assessing projected impacts of MCC 
investments, we feel it is important that we attempt to quantify them 
given our unique mandate of promoting economic growth. 

Risks - Efficiency gains: 

GAO suggests that "efficiency gains - such as time saved because of 
better roads - . may not result in measurable changes in per capita 
income or GDP. Although efficiency gains could improve social welfare, 
they may not be directly measurable as net additions to the economy." 
While it is true that time and cost savings may not translate 
completely into increased income-generating activity, it is highly 
plausible in a poor country that the vast majority of these savings 
would be used to generate income. In recognition that the assumption of 
full translation of efficiency gains into income may overstate 
benefits, MCC was careful to include a scenario in which benefits are 
lower than expected in its sensitivity analysis of overall Compact, It 
is also important to note that, despite the risks identified by the 
GAO, other factors not included in the MCC analysis could potentially 
raise the returns on MCC investments (including better access to health 
and education, which would raise overall worker productivity). 

Risks - Maintenance: 

The report makes several references to maintenance risk. We believe 
this risk is overstated and does not reflect a number of recent 
measures taken by Vanuatu over the past three years to improve its 
transport infrastructure maintenance capacity, nor the content of the 
Compact itself. For example, the Management Improvement Plan that was 
developed with the assistance of the Asian Development Bank has led to 
a significant increase in meaningful capacity within the Public Works 
Department (PWD) for maintenance. The European Union continues to 
provide assistance to expand and strengthen PWD's maintenance 
functions, especially in the outer islands, The Government increased 
the PWD's budget by an unprecedented amount, 499 million Vatu, for 
2007. MCC's Compact will provide both technical assistance and a 
significant amount of money to alleviate one of the key constraints 
that prevents timely and adequate level of maintenance by the PWD, 
namely maintenance equipment. These are sustainable measures, which 
will continue to yield benefits beyond 2011. 

Risks - Contingencies: 

The report concludes that insufficient contingencies have been allowed 
in the build up of construction costs, It cites a survey by Flyvbjerg, 
Holm and Buhl, which concluded that road projects averaged escalations 
of 20.4 percent from amounts forecasted at the alternatives analysis 
stage. The projects that were under consideration in the Compact were 
at an advanced stage in preparation, either at preliminary design or 
final design stage, not at an alternatives analysis stage, Appropriate 
contingencies should be applied to project cost depending on the stage 
of the project. MCC has used certified civil engineers and cost 
estimators in reviewing and finalizing project specific costs during 
the due diligence phase. MCC is confident in the final cost estimates. 
including price and physical contingencies, for each subproject within 
the Compact. This information has been provided to GAO.

Conclusion: 

In conclusion, we wish to emphasize that our analysis excluded a range 
of important but difficult to quantify benefits (such as an increased 
ability to invest in education, healthcare, and land improvements). In 
this sense, our analysis was designed conservatively, to prevent the 
very outcome that GAO claims to have discovered in its audit, namely an 
explicit overstatement of benefits. While some of the language used by 
MCC may unintentionally mislead some readers, it would be wrong to 
conclude that the analysis was designed intentionally to overstate 
benefits, but this is the overwhelming impression a reader will get 
from the GAO report. 

The following are GAO's comments on the Millennium Challenge 
Corporation letter dated June 8, 2007. 

GAO Comments: 

1. MCC quotes from its statement in the compact and our statement in 
the text of the draft report. However, the figure we cite for per 
capita income in 2015--37 percent--originates in MCC's investment memo, 
not in our analysis. 

2. MCC notes that it emphasized repeatedly that its key statements in 
question were factually correct and that it did not intend to overstate 
benefits. At issue in this finding are both the facts represented by 
MCC's data and calculations and MCC's representation of these data and 
calculations in its public statements. As our report notes, MCC's 
portrayal of the compact's projected impact on income and economic 
growth can be understood as cumulative only by reviewing supporting 
source documents and spreadsheets, which are not publicly available. 
Our report does not assert or imply that MCC's portrayal was 
intentionally misleading; we did not determine through our audit, or 
address in the report, the reasons for the gap between MCC's portrayal 
and its underlying analysis. 

3. MCC comments that our assertion about its portrayal of income 
benefits is itself likely to mislead. In response to this comment, and 
as part of our final review process, we clarified the terms and 
presentation of our analysis of MCC's statements and data and made 
modifications as appropriate. 

4. MCC asserts that it "stated that per capita income would increase by 
a cumulative amount of 15.4% by 2010 relative to the 2005 baseline" and 
that this is how it understood the underlying data in 2005. However, 
MCC documents in 2005 and later do not state that the portrayed effect 
on per capita income is cumulative. These documents include MCC's 
November 2005 Investment Memo; the March 2006 Vanuatu compact; the 
January 4 and March 7, 2006, congressional notifications; MCC press 
release; and the April 2007 MCC Annual Report for 2006. We asked MCC to 
provide any publicly available document that discloses that the per 
capita income effect is cumulative; however, MCC did not provide any 
such document. 

5. MCC comments that its "underlying data do not support the conclusion 
that cumulative growth over 5 years would be only 3.9%." Our report 
does not make any assertions about the cumulative growth rate in income 
per capita. As stated in figure 7, we graph the increase in the level 
of income per capita in each year relative to the 2005 baseline. In 
2010, this increase is 3.9 percent. 

6. We agree with MCC that "the use of five-year cumulative figures is 
misleading." Since MCC's public documents do not state that the 
increase in per capita income is cumulative, we maintain that MCC's 
statements regarding the increase in average per capita income are 
inconsistent with its data. 

7. MCC comments that "the data also indicate that the effect on the per 
capita GDP growth rate will be moderate but rising, albeit much smaller 
than three percentage points." (We understand this statement to mean 
that the effect on aggregate GDP growth will be smaller than 3 
percentage points, given that the sentences that follow the statement 
discuss aggregate, not per capita, GDP.) As the note to figure 9 of our 
report indicates, we agree with MCC that its data indicate that the 
compact will have a positive, albeit small, effect on GDP growth rate. 
MCC provided us with its computation showing that the compact's effect 
on GDP growth rate will increase over a 20-year period and will reach 
0.54 percentage points in year 20--not the 3 percent growth rate 
increase implied by its portrayal of the compact's projected results. 
Based on MCC's calculation, the compact will enable the Vanuatu economy 
to double in size in 20 years. Without the compact, the Vanuatu economy 
will double in size in 24 years. 

8. MCC comments that "GAO's presentation suggests that [near and non-
poor] households (local business people, government personnel, 
transport providers, and tourism service providers) could feasibly be 
excluded and that this would be desirable." We are not suggesting that 
such people could be feasibly excluded or that this would be desirable. 
Our report notes, "Given that many of MCC's benefits flow to the 
expatriate-dominated tourism sector, MCC makes an appropriate 
assumption that the local population will not receive all the 
benefits." Our report's analysis indicates that there will be poor, 
rural beneficiaries in addition to other beneficiaries but observes 
that MCC has not quantified what portion of the compact's benefits the 
poor, rural beneficiaries will receive. 

9. MCC states that we argue that the total number of rural 
beneficiaries should be reduced by one-third. In fact, our report 
observes that the total number of beneficiaries may be overstated. MCC 
specifically disputes our observation that households living in the 
hinterland or on off-shore islets may not be beneficiaries. However, of 
the 13,619 persons in the catchment areas that we question, 6,889 (51 
percent) reside not in the hinterland or off-shore islets but near the 
Efate tourism center and capital of Port Vila and along a portion of 
the Ring Road that is already paved. Given that MCC's benefits in Efate 
are primarily based on increased tourist activity and road user cost 
savings from the improved road, it is not clear how construction of a 
road elsewhere on the island will lead to benefits when the existing 
paved road already provides access to Port Vila. 

As documented in appendix III, we do question whether inhabitants of 
off-shore islets and of villages more than 10 kilometers from MCC 
projects would benefit fully from MCC's projects. MCC includes these 
people in its count of beneficiaries without accounting for, and 
quantifying, the likely decrease in benefits that will accrue to a 
person who lives both across an occasionally impassable river and more 
than 10 kilometers from an MCC project bridge. As MCC's internal 
analysis states regarding the Santo South Coast Road, "the project does 
not entail a general improvement of the road, only of river crossings. 
Neither would it give all-weather access all the way to Tasiriki, and 
so to the west coast. It must be said, therefore, that the scale of 
these benefits are more than usually doubtful." Elsewhere in the 
report, it notes that the "benefits to the west coast [Santo] 
population will be slight." 

10. MCC asserts that its definition of "local population" does not 
include urban dwellers. Although we asked MCC to support this 
assertion, we received no additional documents. In separate comments, 
MCC added that although its documentation does not contain a very 
detailed reference specifically saying exactly how many business 
people, transport providers and tourism operators are foreigners and 
urban, MCC believes it is reasonable to conclude that urban dwellers 
are included in these categories. However, we note that not every urban 
dweller is a business person, transport provider, or tourism operator 
and that some urban beneficiaries would therefore fall into the "local 
population" that receives an estimated 43 percent of benefits. 

On the basis of MCC's definitions of "local population" and other 
terms, we determined that "local population" includes residents of 
nonrural areas. MCC's definition of the term local population states, 
"We have combined three categories: local producers, local consumers 
and inhabitants of remote communities. In reality these are the same 
people and we have labeled them simply 'Local population.'" MCC defines 
local producers as including "landowners; existing and potential 
lessees of land; processors of primary produce, chiefly for export; and 
the owners and operators of tourist facilities." This definition does 
not distinguish between rural and urban dwellers. Further, given that 
expatriates dominate in Vanuatu's formal economy sectors and that the 
urban center of Port Vila is the center of Vanuatu's tourist industry, 
it is likely that "processors of primary produce, chiefly for export; 
and the owners and operators of tourist facilities" include urban 
dwellers and those that are not poor. Finally, MCC's definition of 
"inhabitants of remote communities" includes the poor inhabitants of 
the "informal urban settlements" on the outskirts of Port Vila. 
Although these people come from rural areas, they are now in an urban 
environment and included within MCC's definition of the local 
population. 

11. MCC states that it has "a reasonable expectation that the majority 
of the 65,000 [beneficiaries] are poor." However, MCC's public 
documents use the phrase "65,000 poor, rural inhabitants," without 
acknowledging that not all of the 65,000 beneficiaries live in poverty. 
We agree that MCC would be correct to suggest that the majority of the 
projected beneficiaries are poor, as our report states: "The poverty 
level, defined as having an income of one U.S. dollar per day, is 51 
percent in rural areas." 

12. About program risks in general, MCC states that it differs from us 
significantly regarding the precise nature and severity of risks. As we 
note in our report, we offer several alternative scenarios to 
illustrate the maximum impact of individual areas of risk on projected 
compact benefits. These scenarios examine the effect of individual risk 
areas and do not consider the possibility of projects' failing to 
achieve certain benefits in multiple areas simultaneously. 

13. We agree with MCC that attempting to quantify induced benefits is 
important and that our fieldwork generally affirmed MCC's assumptions. 
We maintain that quantifying induced benefits constitutes a risk to 
projected impact, because realization of such benefits depends on 
beneficiaries' response to the opportunities that the compact provides. 

14. We asked MCC to support its assertion that "it is highly plausible 
in a poor country that the vast majority of [time and cost] savings 
would be used to generate income." We did not receive additional 
documentation supporting this assertion. However, MCC stated that its 
assertion is based on extensive field consultations with beneficiary 
groups. 

MCC's documentation of its methodology for assessing efficiency gains 
states that its valuation of the average value of passengers' time is 
"extremely crude." MCC's documentation adds: "The very concept of 
'working hours' in a largely subsistence economy is open to question. 
And what is the value of tourists' time to the national economy? It is 
arguable that faster journey times allow tourists more time to spend on 
things other than travel. But there is no research to support a more 
sophisticated approach to valuing time." 

15. MCC notes other factors that may benefit the economy. As our report 
indicates (see footnote 15), we agree that "other benefits may accrue 
to Vanuatu from the MCC compact. Increased economic activity in tourism 
may have spillover benefits for other sectors of the economy, and the 
welfare of Vanuatu's citizens may improve due to increased access to 
health care and educational opportunities." 

16. MCC believes that we have overstated the risk from poor maintenance 
of the infrastructure investment and have not accounted for Vanuatu's 
increasing its maintenance spending by $4.6 million or the assistance 
and maintenance equipment MCC is providing to Vanuatu. We have added 
additional information to our report regarding MCC's program to improve 
maintenance capability. Nevertheless, maintenance remains a significant 
risk. As we noted in our report, Vanuatu's record on transport 
infrastructure maintenance is poor. Other donors, including officials 
from the ADB and the World Bank, identified maintenance as a 
significant challenge. 

17. MCC states that "the projects that were under consideration in the 
compact were at an advanced stage in preparation, either at preliminary 
design or final design stage," not at the alternatives analysis stage. 
Our previous report, which cited a study by Flyvbjerg, Holm, and Buhl 
as part of a larger report on improving cost and benefit analysis for 
transportation projects, used the term "alternatives analysis stage"; 
however, the study uses the term "time of decision to build" to define 
the point of comparison. MCC's "time of decision to build" is when it 
has completed its due diligence process and decided to sign a compact. 
As such, the basis for comparison in the study by Flyvbjerg et al. and 
in our report is identical. To help clarify the basis of comparison, we 
have replaced "alternatives analysis stage" with "time of decision to 
build." 

MCC states that it provided us the final cost estimates, including 
price and physical contingencies, for each subproject within the 
compact. However, as our report notes, we did not receive full 
documentation of the cost estimates and contingencies that MCC used in 
developing its economic analyses. We received no supporting 
documentation for the cost and contingency estimates for the Santo and 
Efate projects, which account for 56 percent of the total compact 
budget. A key document that MCC's contractor told us it used as a basis 
for developing the Santo and Efate costs was not available to us 
because the government of Vanuatu would not provide it to MCC. The 
documentation of MCC's cost estimates and contingencies for specific 
projects that we received shows a design contingency of 20 percent. 
(One project--MCC's smallest project, the $400,000 South West Bay 
airstrip--adds an additional 10 percent construction contingency.) This 
20 percent contingency is slightly less than the average cost overrun 
found by Flyvbjerg et al. MCC may complete the projects at, under, or 
above a 20 percent contingency. However, cost overruns remain a risk to 
project benefits, and we have presented them as such. 

18. See comment 2. 

[End of section] 

Appendix VII: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

David B. Gootnick, Director, 202-512-3149: 

Staff Acknowledgments:

In addition to the person named above, Emil Friberg, Jr. (Assistant 
Director), Gergana Danailova-Trainor, Reid Lowe, Angie Nichols- 
Friedman, Michael Simon, and Seyda Wentworth made key contributions to 
this report. Also, David Dornisch, Etana Finkler, Ernie Jackson, and 
Tom McCool provided technical assistance. 

(320451): 

FOOTNOTES 

[1] An MCC compact is an agreement between the U.S. government, acting 
through MCC, and the government of a country eligible for MCA 
assistance. 

[2] MCC's $65.7 million compact with Vanuatu provides $317 per capita; 
in contrast, MCC's $547 million compact with Ghana--its largest 
compact--provides $25 per capita. The amounts provided per capita by 
the 11 compacts signed to date range from $6 for Madagascar to $317 for 
Vanuatu. 

[3] Project cash flows are determined by comparing program spending 
against future expected increases in value added or income. The 
internal rate of return is calculated for these cash flows to summarize 
the economic impact. MCC refers to this internal rate of return as the 
economic rate of return. 

[4] Millennium Challenge Account, Best Practices in Compact Development 
(Washington, D.C.: 2006). 

[5] For example, in Nicaragua, MCC expects that the compact will 
transform project areas into an engine of economic growth; in El 
Salvador, MCC states that the compact provides an historic opportunity 
to transform the country's economic development; and in Armenia, MCC is 
undertaking road and irrigation projects to transform the economic 
performance of Armenia's agricultural sector. 

[6] The "investment memo" is an MCC internal document prepared by MCC's 
compact assessment team and submitted to MCC's investment committee-- 
consisting of MCC's Chief Executive Officer (CEO), vice presidents, and 
other senior officials. The committee reviews the memo and decides 
whether to recommend proceeding to compact negotiations. 

[7] We modeled risk scenarios by first assuming the phasing of costs 
and benefits. We then combined this phasing with three additional 
models that assumed a lack of maintenance, a lack of induced benefits, 
and a lack of monetized efficiency gains. See appendix V for more 
information about our methodology. 

[8] The Millennium Challenge Act of 2003 requires MCC to determine 
whether countries are eligible for MCA assistance each fiscal year. 
Countries with per capita income at or below a set threshold may be 
selected as eligible for assistance if they meet MCC indicator criteria 
and are not statutorily barred from receiving U.S. assistance. MCC uses 
16 indicators divided into three categories: Ruling Justly, Encouraging 
Economic Freedom, and Investing in People. To be eligible for MCA 
assistance, countries must score above the median relative to their 
peers on at least half of the indicators in each category and above the 
median on the indicator for combating corruption. GAO, Millennium 
Challenge Corporation: Compact Implementation Structures Are Being 
Established; Framework for Measuring Results Needs Improvement, GAO-06-
805 (Washington, D.C.: July 28, 2006). 

[9] Prior to the due diligence process, a transaction team conducts a 
preliminary assessment of each proposal and reports its findings in an 
internal "opportunity memo" to the MCC investment committee. If the 
opportunity memo is approved, the team launches a detailed due 
diligence review, using MCC's published proposal guidelines and 
criteria to analyze project costs and benefits and assess the 
proposal's consultative process, rationale, environmental and social 
impact, and sustainability. The transaction team makes recommendations 
based on its assessment of the proposal in an investment memo to the 
MCC investment committee. 

[10] MCC and the country's accountable entity must complete 
supplemental agreements, including a disbursement agreement and a 
procurement agreement, before the compact enters into force and funds 
are disbursed. 

[11] By U.S. law, all compacts must end within 5 years of entry into 
force. 

[12] The institutional strengthening program includes $5.74 million for 
equipment purchases; of this amount, $1.4 million is provided directly 
to PWD and the remainder will purchase equipment for the use of the MCC 
construction contractor, to be turned over to the PWD in specified 
condition 4 years later. 

[13] MCC used Vanuatu statistical surveys such as the Vanuatu 1999 
Household Income and Expenditure Survey, 1993 Agricultural Survey, 2004 
Tourist Survey, and a monthly Hotel Occupancy Survey as baseline data 
sources. Some data were updated to 2005 values from earlier data based 
on growth factors. MCC assessed the data it used as the best available, 
but noted that some data have shortcomings. Other data--such as the 
amount of spending by expatriates in Vanuatu, and the percentage of 
tourist spending that remains in the country--were not available, and 
MCC had to estimate their value. Since signing the compact with 
Vanuatu, MCC has funded additional surveys to improve the quality of 
baseline data and track key compact performance indicators. 

[14] MCC determined reduced road user costs by means of (1) traffic 
counts prepared by Vanuatu's PWD and MCA-Vanuatu for the MCC and (2) an 
estimation of road user costs per kilometer based on the Highway Design 
and Maintenance Standards Model (HDM)-III developed by the World Bank. 
HDM-III is a software package that provides the economic decision 
criteria for evaluating road construction or maintenance strategies. 
MCC determined the costs to road users by multiplying the average 
annual daily traffic on the road by the length of the road and by the 
average road user cost per kilometer. Road user cost includes vehicle 
operating costs and the value of passengers' time. The difference in 
user cost before and after construction provides the benefit. 

[15] Although not included in the economic analyses, other benefits may 
accrue to Vanuatu from the MCC compact. Increased economic activity in 
tourism may have spillover benefits for other sectors of the economy, 
and the welfare of Vanuatu's citizens may improve due to increased 
access to health care and educational opportunities. 

[16] MCC assumed that 16 percent of the total cost of construction 
would be captured in the local economy and counted this as a benefit. 

[17] In its final April 2006 economic analysis, MCC adjusted this 
calculation downward slightly to 24.2 percent. 

[18] In its Vanuatu Monitoring and Evaluation Plan, MCC is using the 
poverty measure proposed by Vanuatu, the fraction of individuals with 
monthly cash income less than 20,000 vatu (approximately $185 U.S. 
dollars or $6 per day). 

[19] We modeled risk scenarios with alternative methods of looking at 
compact benefits that assume phasing of costs and benefits and phasing 
coupled with lack of maintenance, lack of induced benefits, and lack of 
monetized efficiency gains. See appendix V for more information about 
our methodology. 

[20] For one of these five, the $400,000 South West Bay airstrip 
project, the estimate also included an additional 10 percent 
construction contingency. 

[21] GAO recently reported this occurring as part of U.S. Agency for 
International Development (USAID)-funded road construction in a tsunami-
affected area of Indonesia. In Indonesia, construction cost per mile 
increased by 75 percent, USAID reduced the length of road to be built 
by more than one third and the agency may extend the planned completion 
date by 5 months. GAO, Foreign Assistance: USAID Signature Tsunami 
Reconstruction Efforts in Indonesia and Sri Lanka Exceed Initial Cost 
and Schedule Estimates and Face Further Risks, GAO-07-357 (Washington, 
D.C.: February 28, 2007). 

[22] Bent Flyvbjerg, Mette Skamris Holm, and Soren Buhl, 
"Underestimating Costs in Public Works Projects: Error or Lie?," 
Journal of the American Planning Association, Vol. 68, No. 3 (2002), 
cited in GAO, Highway and Transit Investments: Options for Improving 
Information on Projects' Benefits and Costs and Increasing 
Accountability for Results, GAO-05-172 (Washington, D.C.: January 24, 
2005). 

[23] MCC cites the "design-construct" contract proposed for the MCA 
program, which will include design and construction of all the projects 
as one package, as key to mitigating this risk. However, MCC's analysis 
also recognized that nonconstruction-related issues (such as access to 
parts of the project site) have the potential to delay the contractor 
and increase costs and that such issues can be significant for major 
road upgrade projects where the competing interests of the contractor, 
adjacent villages, and the general public must be balanced. MCC's 
analysis states that, to help manage the risk of project-related 
disputes and delays, MCC plans to have experienced consultants work 
with local PWD staff who have an understanding of the social and 
cultural issues. 

[24] According to the compact, the government of Vanuatu must pay any 
environmental mitigation and remediation costs in excess of the budget. 

[25] Benefits from construction activities may also be reduced by a 
delayed procurement. MCA-Vanuatu officials initially told us they 
anticipated issuing an invitation for bid to contractors by the end of 
February 2007. As of May 2007, the invitation had not yet been issued. 
MCC currently expects construction to begin in 2008, further reducing 
the likelihood of benefits starting in 2007 as MCC anticipated in its 
analyses. 

[26] It is unclear whether responsibility for feeder roads lies with 
the government at the central or provincial level; neither takes 
responsibility. The Santo PWD expects to focus on maintaining feeder 
roads once the East Coast Road is done. 

[27] Conditions precedent are specific steps that must be completed by 
the compact country prior to MCC's providing a disbursement. According 
to MCC officials, the government has thus far met or exceeded the 
requirements of these conditions. 

[28] In technical comments on a draft of this report, the World Bank 
noted that initial construction quality and subsequent traffic can 
affect a road's projected benefits and, if not included in the 
investment analysis, can have important implications for the actual ERR 
that may be equal to, or larger than, the impact of poor maintenance. 
The World Bank also noted that weak governance and political corruption 
can also have a large impact on a road's sustainability. 

[29] This simulation results in a modest impact on ERR because it 
reflects both a reduction in future benefits and a reduction in future 
costs for road maintenance. 

[30] In technical comments on a draft of this report, the World Bank 
noted that it measures induced benefits directly, comparing base 
traffic levels with expected growth in demand for transport as a result 
of investments. This method results in relatively straightforward and 
transparent calculations. The World Bank observed that models using 
other, more indirect approaches to estimating traffic, such as 
examining the impact of increased agricultural production, require 
careful consideration and that it has generally used such methods less 
often over the last 10 years because the results are sometimes unclear 
and sensitive to assumptions. Typically, it uses such methods only when 
a road project is expected to have significant restructuring effects-- 
such as a completely new road or bridge in an urban environment--that 
could drastically change land use patterns. 

[31] However, the lack of consideration of induced benefits is not 
without its flaws. World Bank officials told us that in the fast- 
growing economy of Vietnam, the World Bank funded a road on the basis 
of current use of traffic without projecting the gains to the economy. 
The road is now very congested, and the project was underinvested. 
According to the World Bank, consideration of projected traffic 
increases--that is, induced traffic--are taken into account only when 
the project is not justified on the basis of existing traffic alone. 
However, rapid traffic growth is considered exceptional. 

[32] The ERR in this case does not take into account opportunity costs 
and is therefore an internal rate of return. 

[33] According to the World Bank, some of the characteristics shared by 
small states such as Vanuatu include remoteness and insularity, 
susceptibility to natural disasters, limited institutional capacity, 
limited diversification, reliance on external trade, and limited access 
to external capital. Commonwealth Secretariat/World Bank, Small States: 
Meeting Challenges in the Global Economy (Washington, D.C.: 2000). 

[34] Beginning in the early 1990s, Vanuatu has witnessed a series of 
short-lived coalition governments, comprising numerous parties and 
involving frequent changes of ministers. There have been eleven 
governments in power since 1990, with most lasting only about 1 year. 
This constant turnover has delayed reforms as coordination of policies 
and obtaining consensus for needed reforms has proved difficult. 
International Monetary Fund, Vanuatu: Selected Issues, IMF Country 
Report No. 07/93 (Washington, D.C.: 2007). 

[35] The legal framework for secured transactions is incomplete. 
According to the ADB, the legal system does not allow for property to 
be used as collateral for loans, particularly outside of Port Vila and 
Luganville. Thus borrowers can neither purchase property on credit nor 
obtain loans against their own assets. This accounts for high borrowing 
costs, higher interest rates, smaller loans offered, and loans of 
shorter maturity, which limit the private sector's access to credit and 
entrepreneurship. IMF, Vanuatu: Selected Issues. 

[36] Before signing the compact, MCC determined the number of potential 
beneficiaries but did not quantify targets for the compact's effect on 
poverty. MCC has funded an additional survey that will provide updated 
baselines and the basis for establishing these targets. 

[37] We identified the following errors: (1) MCC's beneficiary 
estimates use data from the 1999 census and have not been updated to 
2005. This results in an underestimate of the beneficiary population. 
(2) In the data provided to us, the total of the populations of 
villages included in the MCC analysis does not match the beneficiary 
count in the monitoring and evaluation plan. (3) For the Efate Ring 
Road project, MCC's data double-counted some villages. (4) For all 
three projects, the 1999 population figures used for some villages do 
not match the populations shown in the 1999 Vanuatu census. For the 
Santo East Coast Road, the population figures do not match for 53 of 56 
villages in the catchment area. 

[38] The remaining 29 percent of costs are for warehouses, and for 
oversight and administrative functions. 

[39] MCC's contractor provided us with spreadsheet calculations of the 
costs for five of the compact projects for MCC. The sheets included 
estimated quantities and unit costs but did not indicate how these 
quantities and costs were derived. 

[40] We developed these criteria as part of a forthcoming publication 
on best practices in cost estimation and implemented the criteria in a 
recent report. See GAO, Telecommunications: GSA Has Accumulated 
Adequate Funding for Transition to New Contracts but Needs Cost 
Estimation Policy, GAO-07-268 (Washington, D.C.: February 23, 2007). 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site (www.gao.gov). Each weekday, GAO posts 
newly released reports, testimony, and correspondence on its Web site. 
To have GAO e-mail you a list of newly posted products every afternoon, 
go to www.gao.gov and select "Subscribe to Updates." 

² Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office 441 G Street NW, Room LM 
Washington, D.C. 20548: 

To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202) 
512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400 U.S. 
Government Accountability Office, 441 G Street NW, Room 7125 
Washington, D.C. 20548: 

Public Affairs: 

Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800 
U.S. Government Accountability Office, 441 G Street NW, Room 7149 
Washington, D.C. 20548: