This is the accessible text file for GAO report number GAO-06-805
entitled 'Millenniums Challenge Corporation: Compact Implementation
Structures Are Being Established; Framework for Measuring Results Needs
Improvement' which was released on July 28, 2006.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Chairman, Committee on Foreign Relations, U.S. Senate:
July 2006:
Millennium Challenge Corporation:
Compact Implementation Structures Are Being Established; Framework for
Measuring Results Needs Improvement:
GAO-06-805:
GAO Highlights:
Highlights of GAO-06-805, a report to the Chairman, Committee on
Foreign Relations, U.S. Senate.
Why GAO Did This Study:
In January 2004, Congress established the Millennium Challenge
Corporation (MCC) to administer the Millennium Challenge Account. MCC’s
mission is to reduce poverty by supporting sustainable, transformative
economic growth in developing countries that create and maintain sound
policy environments. MCC has received more than $4.2 billion in
appropriations, and, as of May 2006, it had disbursed $22.4 million to
four countries whose signed MCC compacts have entered into force. For
the first three countries with compact entry into force—Madagascar,
Cape Verde, and Honduras—GAO was requested to examine (1) key aspects
that MCC reviewed, and the criteria it used, in its due diligence
assessments; and (2) the structures that have been established for
implementing the compacts.
What GAO Found:
MCC undertook a wide range of activities in its due diligence,
including five key aspects of the Madagascar, Cape Verde, and Honduras
proposals: (1) countries’ consultation with local groups in developing
compact proposals, (2) projects’ coherence with compact goals, (3)
environmental and social impacts, (4) institutional and financial
sustainability, and (5) impact on economic growth and poverty
reduction. MCC based its assessments on an evolving set of criteria:
early, general guidance to the countries followed by later, more
specific guidance. MCC’s analyses of the projects’ economic impact were
limited in that some of the assumptions and data used may not reflect
country conditions. As a result, the projects selected on the basis of
the analyses may not achieve compact goals. In the two countries we
visited, Madagascar and Cape Verde, MCC conducted the analyses with
limited country participation, which resulted in countries’ having
little understanding of the process.
MCC and the three countries have made progress in establishing compact
country structures for oversight and management, procurement, fiscal
accountability, and monitoring and evaluation, although some of these
structures are not yet complete. The oversight structures allow for
country management with MCC review, but some organizations were not
fully staffed for months after the compacts entered into force.
Madagascar and Cape Verde have implemented fiscal accountability
structures for MCC-funded projects, and established procurement
structures with effective characteristics; however, these structures
are still largely untested and some are still under development.
Finally, MCC and the countries have established monitoring and
evaluation frameworks to track and account for program results.
However, limitations in the baseline data collected, linkage to
economic analyses, methods of addressing uncertainty associated with
program results, and the timely design of randomized controlled trials
may constrain MCC’s ability to monitor and evaluate program results.
Figure: Status of MCC Compact Development and Appropriations:
[See PDf for Image]
Source: GAO analysis of MCC information.
[End of Figure]
What GAO Recommends:
GAO recommends that the Chief Executive Officer of MCC (1) ensure that
economic analyses of compact proposals better reflect country
conditions and involve country participation and (2) improve monitoring
and evaluation by obtaining more reliable baseline data, ensuring a
clear linkage to economic analyses, developing criteria for
establishing and adjusting targets, and ensuring timely development of
evaluation designs. MCC generally agreed with GAO’s recommendations;
State commented that some of GAO’s findings reflect transitory
problems.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-805.]
To view the full product, including the scope and methodology, click on
the link above. For more information, contact David Gootnick at (202)
512-3149 or gootnickd@gao.gov.
[End of Section]
Contents:
Letter:
Results in Brief:
Background:
MCC Assessed Five Key Aspects of Proposals While Developing Guidance,
but Limitations Affected the Accuracy of Economic Analyses:
MCC-Approved Compact Implementation Structures Are Not Yet Complete;
Weaknesses in Monitoring and Evaluation Framework May Limit Measurement
of Results:
Conclusions:
Recommendations:
Agency Comments and Our Evaluation:
Appendixes:
Appendix I: MCC Actions in Response to April 2005 GAO Recommendations:
MCC Has Developed a Strategic Plan to Enhance Accountability and
Completed a 2006 Performance Plan:
MCC Has Made Significant Progress in Establishing Internal Controls:
MCC Has Taken Steps to Develop Human Capital, but Does Not Track
Allocation of Human Capital to Key Activities:
MCC Has Taken Steps to Define Corporate Governance:
Appendix II: Scope and Methodology:
Appendix III: MCC Projects in Madagascar and Cape Verde:
Appendix IV: Summary of MCC Procurement Agents, Standards, and
Provisions in Madagascar, Cape Verde, and Honduras:
Appendix V: Comments from the Millennium Challenge Corporation:
GAO Comments:
Appendix VI: Comments from the Department of State:
GAO Comments:
Appendix VII: GAO Contacts and Staff Acknowledgments:
Tables:
Table 1: Hiring Dates for Key Positions Identified in Madagascar, Cape
Verde, and Honduras Compacts, as of April 2006:
Table 2: Status of MCA-Madagascar Internal Controls, as of February
2006:
Table 3: Status of MCA-Cape Verde Internal Controls, as of February
2006:
Table 4: MCC Response to GAO Recommendations for Corporatewide
Accountability:
Table 5: MCC Response to GAO Recommendations for Internal Controls over
Program and Administrative Operations:
Table 6: MCC Response to GAO Recommendations for Human Capital
Infrastructure:
Table 7: MCC Response to GAO Recommendations for Corporate Governance:
Table 8: Procurement Agents and Standards in Madagascar, Cape Verde,
and Honduras:
Table 9: Procurement Provisions, Roles, and Responsibilities Stated in
MCC Agreements with Madagascar, Cape Verde, and Honduras:
Figures:
Figure 1: Summary of the MCC Compact Development and Implementation
Process and Status of Countries Eligible to Apply for Compacts, as of
June 2006:
Figure 2: MCC-Eligible Countries with Signed Compacts, as of May 2006:
Figure 3: Number of Days Elapsed from Eligibility to Compact Signature
or Entry into Force, as of May 2006:
Figure 4: Timeline of MCC Guidance Issuance and Investment Memo Dates:
Figure 5: Issuance of Key MCC Criteria for the Consultative Process
Relative to Completion of Due Diligence for Madagascar, Honduras, and
Cape Verde:
Figure 6: Issuance of Key MCC Criteria for Project Coherence Relative
to Completion of Due Diligence for Madagascar, Honduras, and Cape
Verde:
Figure 7: Issuance of Key MCC Criteria for Environmental Impact
Relative to Completion of Due Diligence for Madagascar, Honduras, and
Cape Verde:
Figure 8: Issuance of Key MCC Criteria for Project Sustainability
Relative to Completion of Due Diligence for Madagascar, Honduras, and
Cape Verde:
Figure 9: Illustration of MCC's Methodology for Calculating an Economic
Rate of Return: Advanced Crop Irrigation Project:
Figure 10: Issuance of Key MCC Criteria for Economic Analysis Relative
to Completion of Due Diligence for Madagascar, Honduras, and Cape
Verde:
Figure 11: Compact Country Oversight and Management Structure:
Figure 12: Elements of the MCC Fiscal Accountability Framework:
Figure 13: Indicator Structure for Honduras's Rural Development
Project:
Figure 14: Jatropha Plantation, outside Antsirabe, Madagascar, Targeted
for Assistance by the MCA-Madagascar Agricultural Business Investment
Project:
Figure 15: Antananarivo Land Records Storage Room, Programme National
Foncier Offices:
Figure 16: Port of Praia, Cape Verde:
Figure 17: The Road from Assomada to Rincao, Cape Verde:
Abbreviations:
CEO: Chief Executive Officer:
ERR: economic rate of return:
FAP: Fiscal Accountability Plan:
FISMA: Federal Information Security Management Act of 2002:
FMFIA: Federal Managers Financial Integrity Act of 1982:
GDP: gross domestic product:
GPRA: Government Performance and Results Act of 1993:
IG: Inspector General:
MCA: Millennium Challenge Account:
MCC: Millennium Challenge Corporation:
NBC: National Business Center:
NGO: nongovernmental organization:
OMB: Office of Management and Budget:
PIU: Program Implementation Unit:
SAS 70: Statement on Auditing Standards No. 70:
USAID: U.S. Agency for International Development:
July 28, 2006:
The Honorable Richard G. Lugar:
Chairman:
Committee on Foreign Relations:
United States Senate:
Dear Mr. Chairman:
In January 2004, Congress established the Millennium Challenge
Corporation (MCC) to administer the Millennium Challenge Account (MCA)
for foreign assistance.[Footnote 1] MCC's mission is to reduce poverty
by supporting sustainable, transformative economic growth in
partnership with developing countries that create and maintain sound
policy environments. MCC carries out its mission by funding projects or
activities in developing countries that demonstrate a commitment to MCA
objectives. MCC expects to raise incomes and lift thousands out of
poverty in countries receiving MCC assistance. MCC has received
appropriations for fiscal years 2004 to 2006 totaling more than $4.2
billion, about $3.8 billion of which has been set aside for compact
assistance; as of May 2006, it had committed $1.5 billion and disbursed
$22.4 million in compact assistance.[Footnote 2] The President has
requested an additional $3 billion in MCC funding for fiscal year 2007.
In April 2005, we reported on MCC's first year of operations, focusing
on its process for determining country eligibility, progress in
developing compacts, coordination with stakeholders, and progress in
establishing management and accountability structures.[Footnote 3] We
made several recommendations to the Chief Executive Officer (CEO) of
MCC and to the Secretary of State, in her capacity as board chair, on
ways to improve MCC's strategic planning, internal controls, and human
capital and corporate governance policies. At that time, MCC's
assessment--"due diligence"--of eligible countries' compact
proposals[Footnote 4] had resulted in its signing one compact for $110
million with Madagascar; by the end of May 2006, MCC had signed
compacts with seven more countries.[Footnote 5] As of the end of May
2006, six compact countries--Madagascar, Cape Verde, Honduras, Georgia,
Vanuatu, and Nicaragua--had signed the supplemental agreements that MCC
requires before compacts can "enter into force," when MCC begins to
disburse funds for compact implementation. MCC has disbursed funds to
four of these six countries--Madagascar, Cape Verde, Honduras, and
Georgia--to begin implementing their compacts.
At your request, we examine in this report MCC's procedures and
structures for developing and implementing compacts. We focused our
work on the first three countries to sign compacts: Madagascar, Cape
Verde, and Honduras. Specifically, we reviewed:
* the key areas that MCC examined in its due diligence assessments of
the three countries' compact proposals and the criteria that MCC used
in these assessments and:
* the implementation structures that MCC and the three countries have
established for oversight and management, fiscal accountability,
procurement, and monitoring and evaluation of compacts.
We also report on MCC's response to our April 2005 recommendations (see
app. I). In conducting our work, we analyzed MCC's process for
evaluating eligible country proposals, including MCC's guidance
documents for eligible countries, policies and procedures, economic
analysis, and documentation of its assessment of proposals submitted by
Madagascar, Cape Verde, and Honduras. In addition, we reviewed MCC's
compacts with the three countries and the supplemental agreements
required for those compacts to enter into force.[Footnote 6] We also
reviewed the countries' implementation plans, such as those for
monitoring and evaluation. We supplemented our evaluation of due
diligence and implementation documents with interviews with MCC
officials and site visits to Madagascar and Cape Verde, in January and
February 2006. We selected these two countries because they had
advanced further than Honduras in filling key positions and beginning
compact implementation. These site visits provided us with a firsthand
perspective of the status of the program and the challenges these
countries faced in developing their compacts and in beginning to
implement their programs. Furthermore, we examined MCC's planning
documents, policies, and procedures and interviewed senior MCC
officials. We conducted our review from June 2005 through May 2006 in
accordance with generally accepted government auditing standards. (See
app. II for additional details of our scope and methodology.)
Results in Brief:
MCC undertook a wide range of activities in its due diligence of the
Madagascar, Cape Verde, and Honduras proposals, while at the same time
developing guidance on key aspects of the countries' proposals. In
conducting its assessments, MCC applied criteria from general guidance
issued in 2004, which was shortly before the countries began developing
their proposals, and from more specific guidance issued in 2005 and
2006. MCC's analyses of the projects' economic impact had limitations
in their use of assumptions and data and degree of country involvement.
We focused on MCC's assessment of five key aspects: the consultative
process during proposal development, project coherence, environmental
impact, institutional and financial sustainability, and economic
analyses of the projects' impact on economic growth and poverty
reduction:
* Consultative process. MCC obtained the views of government, civil
society, and private sector officials to determine whether the
countries' consultative process had been timely, participatory, and
meaningful. MCC officials told us that, on the basis of these
assessments, they had returned proposals that were not sufficiently
founded on the consultative process. However, in assessing the
proposals, MCC used criteria contained in guidance issued after or
shortly before the countries submitted their proposals.
* Project coherence. MCC assessed whether the countries' proposed
projects were linked to their compact goals and whether the projects
addressed key impediments to achieving those goals. MCC's initial
published guidance to the countries did not inform them that projects
should be linked to compact goals. When MCC issued more specific
guidance, in November 2005, it had already signed compacts with all
three countries. Generally, projects were linked to compact goals in
the three countries that we reviewed.
* Environmental and social impact. MCC reviewed the probable
environmental and social impact of proposed projects. For projects that
MCC deemed likely to have a negative impact, it required further
assessment as well as an impact management plan. In conducting its
assessments, MCC used criteria in guidelines issued in March 2005.
* Project sustainability. MCC assessed whether projects could be
sustained institutionally and financially after the compacts expired.
MCC reviewed (1) the countries' commitment to sustaining the projects
by making policy changes and providing financial resources, (2) the
countries' institutional capacity to achieve project objectives, and
(3) the projects' likelihood of being financially self-sustaining based
on MCC's economic analysis.
* Economic analyses. To predict the projects' impact on economic
growth, MCC calculated their economic rate of return--that is, for each
dollar that it spent, the dollar benefit the country is likely to
receive. MCC used an economic model that included data and assumptions
about project beneficiaries' expected behaviors. However, the data and
assumptions used in some of MCC's economic analyses may not accurately
reflect the countries' socioeconomic conditions. As a result, MCC
cannot be assured that it has selected projects that will achieve the
compacts' goals. In the two countries we visited, Madagascar and Cape
Verde, country representatives were not closely involved in MCC's
economic analyses. This limited involvement in the process resulted in
countries' having little understanding of the underlying economic
framework on which the compacts were based.
MCC and the countries have made progress in implementing structures for
oversight and management, fiscal accountability, and procurement for
Madagascar, Cape Verde, and Honduras, but some of these structures are
not yet complete. In addition, although MCC has established monitoring
and evaluation frameworks, the frameworks have weaknesses that could
affect MCC's ability to ensure accountability for results.
* Oversight and management. MCC and the countries have established
structures that allow for country management with MCC review. MCC
maintains a small staff in each country, which refers matters requiring
approval to MCC headquarters in Washington, D.C. Country management
structures include the following: a steering committee, a stakeholder
committee, and a management unit. Staffing for all three country
organizations remained incomplete for months after the compacts entered
into force. This incomplete staffing at entry into force limits the
ability of the countries to achieve their compact objectives within the
fixed time period of the compact.
* Fiscal accountability. MCC has established a framework of internal
controls to incorporate fiscal accountability into MCC-funded projects.
In Madagascar, we found that underlying accountability systems,
policies and procedures, and internal controls are less fully developed
than those in Cape Verde, which built on existing government structures
and systems. Because the accountability systems in both countries are
still under development, both countries face risks in their key
financial processes and activities. In January 2006, the MCC Inspector
General (IG) issued guidelines for the countries' financial audits, and
Madagascar and Cape Verde are preparing for these audits.
* Procurement. In each of the three countries, the MCC-approved systems
have characteristics that typify effective procurement systems--for
example, integrity, openness, and accountability. These systems are
still largely untested, and some MCC and country staff and procedures
are not yet in place.
* Monitoring and evaluation. Compact countries are required to prepare
and implement a monitoring and evaluation framework, including plans
that document the necessary data collection, data quality reviews,
analysis, and interim and final reporting of results. MCC has
conditioned some disbursements on the countries' achieving performance
targets. However, MCC's ability to track and account for results may be
limited by weaknesses in the frameworks related to the availability and
quality of baseline data, ensuring linkage between the economic models
and the monitoring and evaluation plans, and accounting for uncertainty
in setting targets and measuring progress. In addition, although MCC
has retained five U.S. research organizations to independently analyze
the results of MCC compacts, MCC has not completed the research designs
needed to use its preferred methodology of randomized controlled trials
prior to project implementation. Not completing the designs prior to
project implementation potentially limits MCC's ability to perform
randomized controlled trials.
We are recommending that the CEO of MCC (1) adopt procedures that
ensure greater involvement of compact country stakeholders in
developing the economic model used to assess projects' likely impact
and (2) to the extent practical and cost-effective, improve MCC's
monitoring and evaluation capabilities by obtaining more accurate and
reliable baseline data, ensuring a clear linkage between MCC's economic
analyses and monitoring and evaluation frameworks, better accounting
for uncertainty in setting targets and achieving outcomes, and ensuring
the timely development of the needed research designs for randomized
controlled trials prior to project implementation.
In commenting on a draft of this report, MCC generally agreed with our
findings, conclusions, and recommendations. The Department of State
commented that it considered some of the findings of our report as
reflecting minor or transitory problems, and that the report should
note the improvements that MCC made during and since the period it
covers. Throughout this report, we have described MCC and compact
progress through May 2006. We have reprinted MCC's and State's
comments, with our responses, in appendixes V and VI. We also
incorporated technical comments from MCC in our report where
appropriate.
Background:
MCC Principles:
In pursuit of its mission to reduce poverty by supporting economic
growth, MCC has identified and defined the following three key
principles to guide its actions:[Footnote 7]
* Reward good policy. "Using objective indicators, countries are
selected to receive assistance based on their performance in governing
justly, investing in their citizens, and encouraging economic freedom."
* Operate in partnership. "Countries that receive MCA assistance are
responsible for identifying the greatest barriers to their own
development, ensuring civil society participation, and developing a
multi-year MCC compact."
* Focus on results. "MCA assistance goes to those countries that have
developed well-designed programs with clear objectives, benchmarks to
measure expected results, procedures to ensure fiscal accountability
for the use of MCA assistance, and a plan for effective monitoring and
objective evaluation of results. Programs are designed so that
recipient countries can sustain progress after the funding under the
compact has ended."
MCC Structure:
MCC is a government corporation that is managed by a CEO appointed by
the President with the advice and consent of the Senate and is overseen
by a Board of Directors (MCC Board). The Secretary of State serves as
board chair, and the Secretary of the Treasury serves as vice-chair.
Other board members are the U.S. Trade Representative, the
Administrator of the U.S. Agency for International Development (USAID),
the CEO of MCC, and up to four Senate-confirmed public members who are
appointed by the President from lists of individuals submitted by
congressional leadership.
Eligibility for MCC Assistance:
The Millennium Challenge Act of 2003 requires MCC to select countries
as eligible for MCA assistance each fiscal year. Countries with per
capita income at or below a set threshold may be selected as eligible
for assistance if they pass MCC indicator criteria and are not
statutorily barred from receiving U.S. assistance. MCC uses 16
indicators divided into three categories: Ruling Justly, Encouraging
Economic Freedom, and Investing in People.[Footnote 8] To be eligible
for MCA assistance, countries must score above the median relative to
their peers on at least half of the indicators in each category and
above the median on the indicator for combating corruption. MCC used
these quantitative indicators, as well as the discretion implicit in
the Millennium Challenge Act, to select 17 countries as eligible to
apply for MCA compact assistance for fiscal years 2004 and 2005. For
fiscal year 2006, MCC identified 23 countries[Footnote 9] as eligible
for assistance--the 17 previously selected and 6 additional countries,
which included lower-middle-income countries eligible for the first
time in fiscal year 2006.[Footnote 10]
MCC's Compact Development:
After MCC selects eligible countries, they may begin a four-phase
process that can lead to the entry into force of compacts (see fig. 1).
Each phase of this process is discussed after figure 1.
Figure 1: Summary of the MCC Compact Development and Implementation
Process and Status of Countries Eligible to Apply for Compacts, as of
June 2006:
[See PDF for image]
Source: GAO analysis of MCC data.
[A] MCC must notify congressional appropriations committees 15 days
prior to obligating funds.
[B] Compact negotiations begin after the MCC investment committee
approves a Consultation Memorandum prepared by the MCC transaction
team. The memorandum is based on the transaction team's determination
that the country proposal has sufficient information to justify
entering into negotiations with the country. MCC must consult with and
report to the appropriate congressional committees 15 days prior to the
start of compact negotiations.
[C] The names of newly eligible countries in fiscal year 2006 are
italicized.
[D] The MCC Board suspended Gambia's eligibility on June 16, 2006.
[End of figure]
1. Country proposal development. Eligible countries are invited to
submit compact proposals, which are to be developed in consultation
with members of civil society, including the private sector and
nongovernmental organizations (NGO). The eligible country also
identifies an "accountable entity" to manage the programs funded by
MCC.[Footnote 11] Eligible countries submitting proposals are not
guaranteed funding; instead, MCC assesses proposals through its due
diligence review. As of May 2006, 14 of the 17 countries selected as
eligible in fiscal year 2004 or 2005, and 1 of the 6 countries selected
as eligible for the first time in fiscal year 2006, had submitted
proposals accepted by MCC for due diligence review.
2. MCC's due diligence review. MCC determines whether the proposal that
an eligible country has submitted meets MCC criteria to ensure that
proposed programs will be effective and funds will be well-used. Due
diligence primarily occurs between MCC's acceptances of an "opportunity
memo" and an "investment memo." MCC assembles a transaction team of MCC
staff, personnel from other U.S. agencies, and consultants to conduct a
preliminary assessment of a country's proposal and reports the team's
findings in an opportunity memo to the MCC investment committee. The
MCC investment committee consists of MCC's CEO, vice presidents, and
other senior officials. If the opportunity memo is approved, the
transaction team launches a detailed due diligence review. The team
assesses the country proposal, reports its findings, and makes
recommendations based on its assessment in an investment memo to the
MCC investment committee. As of May 2006, MCC was conducting due
diligence analyses of seven eligible country proposals.
3. Compact negotiation and MCC Board approval. MCC may enter into
compact negotiations with the eligible country before the investment
memo is completed. If compact negotiations are successful, MCC staff
formally submit the compact for MCC Board approval. Once the board
approves the compact, MCC and the eligible country may sign it. As of
March 2006, MCC had signed compacts with 8 of the 17 countries
determined eligible in fiscal years 2004 and 2005. (See fig. 2.) MCC
commits the full amount of the compact funding at signing but obligates
and begins to disburse funds to implement projects only after the
compact has entered into force. Under the Millennium Challenge Act,
compacts may remain in force no longer than 5 years. The compacts
stipulate that, with limited exceptions, all funds must be spent during
that time.
4. MCC and compact country complete entry-into-force requirements.
MCC's compact with each country identifies the following supplemental
agreements that MCC and the country's accountable entity must complete
before the compact can enter into force.[Footnote 12]
* The disbursement agreement sets out the "conditions
precedent"[Footnote 13] and other requirements for disbursements from
MCC and redisbursements to any person or entity. These conditions
include performance targets for projects outlined in the compact.
* The procurement agreement sets forth guidelines for all procurements
of goods, works, and services financed with MCC funding.
* Compact term sheets for supplemental agreements, which vary by
country, and include documents such as a governance agreement, fiscal
agent agreement, form of implementing entity agreement, and form of
bank agreement.
After compacts enter into force, MCC may begin the disbursement of
funds and countries may begin implementing projects. In the first eight
compacts, approximately 53 percent of funding went to transportation
and other infrastructure projects; 22 percent went to agriculture and
rural development; 13 percent went to other project types; and 12
percent went to program management, monitoring, and evaluation. (See
fig. 2.)
Figure 2: MCC-Eligible Countries with Signed Compacts, as of May 2006:
[See PDF for image]
Source: GAO analysis of MCC data and Map Resources (map).
Note: Numbers may not add due to rounding of source data. See appendix
III for photographs of projects we visited in Cape Verde and
Madagascar.
[A] The Access to Markets Project in Benin is a major construction
project at the Port of Cotonou and includes associated studies and
institutional strengthening.
[B] The Irrigated Agriculture Project in Armenia includes the repair of
irrigation infrastructure.
[C] The Justice Program in Benin includes institutional strengthening
and infrastructure components (construction of new courthouses).
[End of figure]
Compact Signature and Entry-into-Force Time Frames:
The length of time from country eligibility selection to compact
signature has varied, with proposal development and due diligence
generally requiring the most time (see fig. 3). For the six countries
whose compacts had entered into force as of the end of May 2006,
completing the steps necessary for entry into force after compact
signing took approximately 3 to 4 months for Madagascar, Cape Verde,
and Honduras; approximately 7 months for Georgia; 2 months for Vanuatu;
and about 10½ months for Nicaragua. For the two countries whose
compacts had not entered into force as of the end of May 2006, 3 months
had elapsed since compact signature for Benin, and 2 months had elapsed
for Armenia (see fig. 3).
Figure 3: Number of Days Elapsed from Eligibility to Compact Signature
or Entry into Force, as of May 2006:
[See PDF for image]
Source: GAO analysis of MCC data.
[A] Benin and Armenia have not yet entered into force.
[End of figure]
MCC Guidance:
MCC has issued guidance and policies for its compact development
process in several stages. Before publishing its initial guidance in
May 2004, MCC provided countries with preliminary guidance addressing
fiscal accountability and monitoring and evaluation. Figure 4 shows the
evolution of MCC's published guidance relative to the end of the due
diligence process with the investment memo.
Figure 4: Timeline of MCC Guidance Issuance and Investment Memo Dates:
[See PDF for image]
Source: GAO analysis of MCC data.
[A] MCC issued interim Environmental Guidelines on March 4, 2005, and
final Environmental Guidelines on January 23, 2006.
[B] Economic analysis guidelines and compact assessment and approval
guidelines were reissued on January 23, 2006.
[C] Consultative process guidance was reissued on January 6, 2006.
[D] According to MCC officials, MCC provided an initial 2-page guidance
to eligible countries in December 2004, but it did not publish the
guidance on the MCC Web site. MCC issued an updated version of the
guidance on January 6, 2006.
[End of figure]
MCC Assessed Five Key Aspects of Proposals While Developing Guidance,
but Limitations Affected the Accuracy of Economic Analyses:
MCC undertook a wide range of activities in its due diligence of the
Madagascar, Cape Verde, and Honduras proposals, while at the same time
developing guidance on key aspects of the countries'
proposals.[Footnote 14] During due diligence, MCC primarily considered
criteria related to the proposals' consultative process, project
coherence, environmental impact, institutional and financial
sustainability, and economic analyses.[Footnote 15] MCC generally
approved proposals that were based on a consultative process and
returned proposals that lacked adequate consultations; however, MCC did
not publish detailed criteria for the consultative process until 1 year
after selecting the countries as eligible for MCC assistance. In
assessing project coherence, MCC approved projects that were linked to
the overall proposal objectives and rejected projects that were not,
although its assessments used criteria that it had not yet published in
its guidance. Additionally, MCC screened projects for likely
environmental impacts and considered factors important for
institutional and financial sustainability. Finally, MCC conducted
economic analyses to assess the projects' likely impact on economic
growth. However, limitations in assumptions and data may have affected
the analyses' accuracy and led MCC to select projects that would not
achieve its goals. Also, a lack of country involvement in the analyses
does not reflect the MCC principle of working in partnership with
countries and may have limited the countries' understanding of the
process.
MCC Assessed the Consultative Process as Guidance Evolved:
MCC's due diligence for Madagascar, Cape Verde, and Honduras assessed
whether the countries had consulted with public and private sector and
civil society stakeholders during proposal development. MCC officials
told us that before beginning due diligence for the three countries,
they assessed the consultative process in proposal drafts and, on the
basis of this review, returned proposals that were not sufficiently
founded on a consultative process. For example, according to the
officials, MCC did not accept one country's proposal because, although
the country had consulted with stakeholders, its proposal did not
reflect the priorities identified during the consultative process.
MCC documents indicate that during its due diligence for two of the
three countries, MCC obtained the views of government, civil society,
and private sector officials on how the governments conducted the
consultative process. In addition, MCC assessed any previous experience
the country had had with a consultative process[Footnote 16] For all
three countries, MCC also reviewed factors such as the date, frequency,
and locations of the process. MCC's documentation of these assessments
in Madagascar, Cape Verde, and Honduras indicated that key stakeholders
had generally agreed on the proposed compact priorities. However, the
MCC assessments noted some weaknesses in the Madagascar and Honduras
governments' management of the process:
* In Madagascar, some groups expressed concern to MCC that the
government had provided short notice (less than 10 days) for
consultative meetings, and that this might have limited rural groups'
participation. Additionally, the government did not communicate to
participants its rationale for accepting or rejecting projects.
However, despite these shortcomings, MCC noted "widespread agreement
and enthusiasm for the…primary components of [the] proposal" among the
business community, local and international NGOs, civil society, and
donors.
* In Honduras, MCC found, on the basis of discussions with local civil
society organizations and international NGOs, that weaknesses--such as
the large size of meetings--had limited effective participation in the
consultative process for the poverty reduction strategy, which formed
the basis of Honduras's MCC proposal. MCC's assessment also indicated
that the government had not directly asked the consulted groups to
identify obstacles to growth. However, MCC found that these groups
concurred with the priorities identified in the country proposal.
According to MCC officials, they followed up with the Honduran
government to address the weakness noted by MCC in Honduras's
consultative process. As a result, according to MCC documentation, the
Honduran government conducted additional consultative sessions with
civil society organizations and donors.
MCC's due diligence of the countries also noted requirements for
additional consultations during project implementation. For example, in
some cases, countries were to undertake consultations with local
stakeholders to identify project sites and conduct environmental
assessments.
Our discussions with representatives of civil society groups and donors
in Madagascar and Cape Verde indicated that they generally concurred
with the compact proposals.[Footnote 17] In Madagascar, a
representative from a key civil society organization noted weaknesses
in the government's conduct of the consultative process similar to
those recorded by MCC. Nevertheless, stakeholders on the Madagascar
advisory council, which includes various civil society and other
organizations, said that the compact proposal generally accounted for
their views, especially in comparison with other donor programs.
As a relatively new organization, MCC conducted due diligence reviews
while it was developing consultative process guidance:
* Evolving guidance. While developing their proposals, the countries
had access to general criteria in MCC's 2004 guidance; however, in
assessing the proposals, MCC applied the more specific criteria
contained in its detailed 2005 guidance. As figure 5 shows, the 2005
guidance was issued 1 year after MCC announced the eligibility of
Madagascar, Honduras, and Cape Verde. According to MCC's Guidance for
Developing Proposals for MCA Assistance in FY 2004, "…each proposal is
expected to reflect the results of an open consultative process,
integrating governmental interests with those of the private sector and
civil society." The 2004 guidance required that proposals include a
description of the consultative process, such as how the proposal takes
into account local-level perspectives of the country's rural and urban
poor, including women, and of private and voluntary organizations and
the business community. Additionally, the guidance required the country
to list all key participants, such as government and nongovernmental
officials, who played a significant role in developing the proposal.
MCC's 2005 Guidance on the Consultative Process more specifically
requires eligible country governments to involve their citizens in
identifying obstacles to economic growth and developing and
prioritizing the development strategies and programs that will be
included in the compact proposal. The 2005 guidance further states that
an adequate consultative process should be timely, participatory, and
meaningful. MCC's guidance also took into account the country's
experience in using a consultative process to develop other national or
poverty reduction strategies. If the compact was built on these
consultations, MCC required some additional consultations to provide
justification of country priorities in the MCA proposal.
Figure 5: Issuance of Key MCC Criteria for the Consultative Process
Relative to Completion of Due Diligence for Madagascar, Honduras, and
Cape Verde:
[See PDF for image]
Source: GAO analysis of MCC data.
[End of figure]
* Incomplete documentation. MCC's documentation of its due diligence
for the three countries presents summary findings, rather than an
analysis of the extent to which the countries consulted with the rural
and urban poor. For example, MCC's due diligence documentation
indicates that the countries' governments included women's groups and
rural sector groups in their consultative process; however, the
documentation does not indicate the extent to which these groups
represented the poor. Additionally, in Madagascar and Honduras, MCC's
documentation does not indicate how MCC assessed the extent to which
the consulted groups informed compact proposal priorities.
MCC Assessed Project Coherence While Developing Specific Guidance:
In keeping with MCC's emphasis on results, due diligence for the three
countries also assessed whether proposed projects were linked to one or
more of the country's compact goals, and whether the projects addressed
key impediments or constraints to achieving these goals. On the basis
of its due diligence assessment, MCC rejected projects that were not
linked to key constraints in two of the three countries we reviewed.
Specifically, MCC rejected tourism and preschool education projects
proposed by Honduras because they were not linked to the impediments to
growth that emerged from the consultative process.[Footnote 18] MCC
also rejected projects to construct feeder roads, which connect some
watershed areas to the markets, and provide access to electricity in
the rural areas of Cape Verde, because MCC's due diligence did not
indicate that these projects addressed key constraints in Cape
Verde.[Footnote 19]
In reviewing MCC's due diligence for the three countries, we found that
MCC did not issue guidance stating that proposed projects should be
linked to the compact goal until after it had concluded its due
diligence assessments. MCC's 2004 guidelines for proposal development
broadly instruct eligible countries to identify priority areas, such as
health or education, and their expected goals for each priority area
over the term of the proposed compact. The guidance also asks the
countries to show how these strategic goals are related to the economic
growth and poverty reduction of the country. MCC's November 2005 MCC
Compact Assessment and Approval Guidelines more specifically indicates
that MCC will assess how the project addresses compact goals. However,
MCC issued the November 2005 guidance after completing its due
diligence for the three countries.[Footnote 20] (See fig. 6.)
Figure 6: Issuance of Key MCC Criteria for Project Coherence Relative
to Completion of Due Diligence for Madagascar, Honduras, and Cape
Verde:
[See PDF for image]
Source: GAO analysis of MCC data.
[End of figure]
MCC Screened Projects for Environmental and Social Impact, Conditioning
Funding on Detailed Assessment:
MCC's due diligence for the three countries included a review of the
probable environmental and social impact of projects that met its
economic analysis and other criteria.[Footnote 21] For projects that it
deemed likely to cause adverse environmental and social impact, MCC
required impact assessments or environmental analyses, including an
impact management plan.
On the basis of its assessment, MCC assigned each project to category
A, B, or C during due diligence to reflect its likely impact and
assessments required.[Footnote 22] For example, MCC assigned all
projects in the Madagascar proposal to category C, because MCC
determined that these projects were not likely to have adverse
environmental and social impact. In contrast, MCC assigned
infrastructure projects in Cape Verde's and Honduras's proposals to
category A or B. For example, the highway expansion project in Honduras
was assigned category A, because it involves the clearing of rights-of-
way that will require compensation for more than 200 affected people.
The port expansion project in the Cape Verde proposal was also assigned
to category A, because it entails dredging and construction in and
around an existing port.
Additionally, MCC assessed whether environmental impact assessments had
been conducted for category A or B projects. For projects lacking
environmental impact assessments, MCC conditioned project funding on
the completion of such assessments as well as on the development of
mitigation plans in consultation with affected groups. For projects
that other organizations had assessed for environmental impact, MCC
used a U.S. agency or a contractor to evaluate the assessment and
determine its adequacy. For example, in Honduras, another donor had
already conducted the environmental impact assessment for the highway
segments proposed for MCC funding. As part of MCC due diligence, the
U.S. Army Corps of Engineers reviewed these assessments and made
recommendations that were incorporated in the existing assessments.
However, in Cape Verde, the MCC contractor conducting due diligence
found another organization's assessment of the port project's
environmental impact to be inadequate. As a result, MCC required a new
environmental assessment, along with plans to manage adverse impact, as
a precondition for funding the project. MCC has allocated funds for
this analysis in the compact budget for Cape Verde.
MCC's 2004 proposal development guidelines do not address projects'
environmental and social impact. In assessing environmental impact, MCC
applied criteria from the Millennium Challenge Act of 2003, which
prohibits MCC from funding projects that are "likely to cause a
significant environmental, health, or safety hazard." MCC also used
criteria laid out in its March 2005 interim Environmental Guidelines,
which state that MCC will not fund projects that lack the appropriate
screening or analysis for environmental impact.[Footnote 23]The
guidance also states that the country has primary responsibility for
conducting and monitoring environmental assessments. (See fig. 7.)
Figure 7: Issuance of Key MCC Criteria for Environmental Impact
Relative to Completion of Due Diligence for Madagascar, Honduras, and
Cape Verde:
[See PDF for image]
Source: GAO analysis of MCC data.
[End of figure]
MCC Evaluated Country Commitment and Capacity and Estimated Project
Impact to Assess Sustainability:
In keeping with its emphasis on sustainable progress, MCC's due
diligence examined whether the three countries' proposed projects could
be sustained after their compacts expired. In assessing project
sustainability, MCC reviewed each country's policy and regulatory
environment and commitment to reforms and financing of future
maintenance costs; it also reviewed expected results from MCC-funded
projects. In addition, MCC considered the countries' institutional
capacity to sustain proposed projects as well as other donors' roles in
strengthening countries' capacity.
* Country commitment. For specific projects in Madagascar, Honduras,
and Cape Verde, MCC reviewed policy or regulatory reforms, and host
country financial commitment, and required some changes or commitment
as a precondition to funding. For example, MCC determined that
agricultural projects proposed by Cape Verde and Honduras would not be
sustainable without the governments' commitment to policy reform. To
ensure adequate operations and maintenance of proposed irrigation
systems, MCC required that the Cape Verde government implement fees
that reflect the costs of scarce water resources and recommended that
the Honduran government also institute such reforms. In the case of
Madagascar, MCC reviewed policy reforms in land management and
financial sectors that would benefit MCC-funded activities.
* Country capacity. In Madagascar, MCC determined that the government
had limited institutional capacity to achieve the project objectives of
increasing land security and promoting financial intermediation to
increase rural savings and extension of credit. To build the
government's capacity, MCC budgeted funding for (1) staff recruitment
and training at Madagascar's land management department to support the
land security project and (2) finance, management, and production
training for rural producers and microfinance institutions to support
the financial intermediation project. In addition, MCC considered the
role of other donors in strengthening the countries' institutional
capacity. For example, while assessing road projects, MCC considered
the World Bank's road sector initiative, which includes institutional
capacity-building in Cape Verde. In Honduras, MCC considered World Bank-
funded and Inter-American Development Bank-funded programs intended to
develop the transportation ministry's management capacity and
maintenance contracting capacity.
* Project impact. MCC also relied on the assumptions used in its
analysis of projects' economic impact to determine the sustainability
of agricultural sector projects for all three countries we reviewed.
For example, MCC expects that as a result of MCC-funded technical
assistance or credit to farmers and rural entrepreneurs, recipients
will be able to generate enough income to afford these services by
paying fees to providers. In cases where sustainability depends on
achieving MCC's projected impact, the soundness of MCC's economic
analysis, discussed in the next section of this report, will also be an
important factor.
In its sustainability assessments, MCC generally adhered to guidance
issued in 2005, rather than to guidance from 2004. MCC's May 2004
proposal development guidance included a general requirement for a
strategy to sustain progress after the compact's expiration. MCC's
November 2005 MCC Compact Assessment and Approval Guidelines did not
require such a strategy, but the guidelines required identification of
factors contributing to institutional and financial sustainability for
each project. (See fig. 8.)
Figure 8: Issuance of Key MCC Criteria for Project Sustainability
Relative to Completion of Due Diligence for Madagascar, Honduras, and
Cape Verde:
[See PDF for image]
Source: GAO analysis of MCC data.
[End of figure]
Limitations in Assumptions and Data, as Well as Country Involvement,
Affected the Accuracy of Economic Analyses:
During its due diligence reviews for Madagascar, Cape Verde, and
Honduras, MCC analyzed proposed projects' probable impact on the
country's economic growth and poverty reduction.[Footnote 24] This
analysis was intended both to assess whether these projects would
achieve MCC's goals and to provide a basis for monitoring their
progress and evaluating their impact.
To predict each project's impact on economic growth, MCC calculated an
economic rate of return (ERR)--that is, the expected annual average
return to the country's firms, individuals, or sectors for each dollar
that MCC spends on the project.[Footnote 25] In calculating projects'
ERRs,[Footnote 26] MCC used an economic model that includes the
following elements (see fig. 9):
* MCC's annual expenditures for the project,
* the project's annual benefits to the country,
* predicted net benefits[Footnote 27] of the project, and:
* the projected ERR.
Figure 9: Illustration of MCC's Methodology for Calculating an Economic
Rate of Return: Advanced Crop Irrigation Project:
[See PDF for image]
Source: GAO analysis of MCC data.
[End of figure]
ERRs for the proposed projects in Madagascar, Cape Verde, and Honduras
varied considerably, ranging from 116 percent for Madagascar's land
titling project to 10 percent for a watershed management and
agriculture support project in Cape Verde;[Footnote 28] the median ERR
for projects in the three countries was 14 percent. However, because
MCC analyzed the Madagascar and Honduras proposals before publishing
its first set of economic analysis guidelines, the ERRs did not
significantly affect those countries' overall choice of projects. (See
fig. 10.)[Footnote 29] MCC's finding of a low ERR for part of Cape
Verde's watershed management and agriculture support project resulted
in the country's dropping an irrigation activity on one island.
Figure 10: Issuance of Key MCC Criteria for Economic Analysis Relative
to Completion of Due Diligence for Madagascar, Honduras, and Cape
Verde:
[See PDF for image]
Source: GAO analysis of MCC data.
Note: The 2006 guidance states that because of economic analyses'
confirmed tendency to be overly optimistic about project benefits, MCC
prefers that evidence about project impact be based on evaluations of
similar, completed projects. For a discussion of the accuracy of such
economic analyses, see Gerhard Pohl and Dubravko Mihaljek, "Project
Evaluation and Uncertainty in Practice: A Statistical Analysis of Rate-
of-Return Divergences of 1015 World Bank Projects," World Bank Economic
Review, Volume 6 (1992): 255-277. With respect to poverty reduction,
the guidelines state that poverty reduction should be measured by the
expected decrease in the poverty gap, defined simply as the amount of
money that, when transferred to poor people, brings everyone's income
up to the poverty line.
[End of figure]
We found limitations in the assumptions and data that MCC used in its
analyses as well as in the countries' involvement in the analyses.
These limitations may negatively affect the accuracy of the analyses
and the countries' understanding of the analysis process, respectively.
* Assumptions and data. Some of the assumptions and data that MCC used
in its analyses do not fully reflect the countries' socioeconomic
environment. As a result, MCC cannot be assured that the projects it
approved, partly on the basis of these analyses, will achieve the
compacts' goals. For example, in calculating the ERR for the Madagascar
land titling project, MCC assumed that local small farmers would use
newly titled land as collateral for loans, invest the borrowed funds in
agricultural activities, and benefit from the increased income from
those activities We discussed this assumption with focus group
participants in Madagascar, including MCA and U.S. government
officials, senior Madagascar government officials representing
ministries affected by the compact, and bank representatives. Our
discussions suggested that MCC's analysis may have been overly
optimistic in assuming that small farmers would mortgage 40 percent of
their newly titled land in Madagascar's uncertain market, which does
not offer insurance.[Footnote 30] While MCC assumed that 40 percent of
the newly titled land will be collateralized, focus group participants
believed this was overly optimistic. As a result, MCC may have
calculated an unrealistically high ERR for the land titling project.
Furthermore, the project may be more likely to benefit farmers with
large, secure landholdings and investors from outside the farming
community, rather than the local small farmers it was intended to
help.[Footnote 31]
Similarly, MCC's economic model for a part of Madagascar's finance
project, the modernization of the National Savings Institution, may not
accurately reflect the institution's financial condition. MCC's model
for the institution's modernization uses the institution's April 2003
net profits (1.4 billion Malagasy francs) as a baseline for estimating
the benefits of computerizing the bank. However, according to
institution officials, 2003 is not a representative year for the bank,
and it experienced unusually large net losses (13 billion Malagasy
francs) after an economic crisis in 2002. As a result, MCC may have
inaccurately estimated the project's likely impact on the banking
system.
* Country involvement. In the two countries we visited, country
representatives were not closely involved in MCC's economic analyses of
the proposed projects. This constrained the contribution of the
analysis process to enhancing country partnership, and stakeholders'
understanding of MCC's economic analysis--including the data analyzed,
assumptions used in the analysis, and the expected outcomes. According
to country officials in Madagascar and Cape Verde, MCC developed the
economic models and selected data with little assistance from country
representatives. According to the Cape Verde proposal team, they
provided some data but did not participate in the actual analysis and
did not have a clear understanding of the process and the criteria MCC
used to assess proposed projects. We discussed these issues with MCC
officials, who told us that the countries' degree of involvement
depended on the capability and willingness of the countries' proposal
development team to actively participate in the analyses.
MCC-Approved Compact Implementation Structures Are Not Yet Complete;
Weaknesses in Monitoring and Evaluation Framework May Limit Measurement
of Results:
MCC gave Madagascar, Cape Verde, and Honduras authority to propose and
develop the implementation structures that they will use to manage
compacts, although MCC retains authority over a number of management
decisions. To govern their programs, the countries have created
management units under the direction of a steering committee, but they
have had difficulty in filling key positions. The countries also have
established structures for ensuring fiscal accountability and for
managing procurements that appear to be effective; however,
implementation is still at a very early stage, and some required
elements of these structures are not yet in place. Finally, the
countries have established frameworks for monitoring and evaluating the
performance of MCC projects. However, the frameworks have weaknesses
related to the inadequacy of baseline data, linkage of monitoring plans
with economic models, methods of addressing uncertainty in achieving
stated targets, and the timeliness of research designs for randomized
controlled trials. These weaknesses may limit MCC's ability to track
and account for program results.
Countries Manage Projects with MCC Review, but Have Faced Delays in
Staffing Oversight and Management Structures:
MCC has established a limited presence in Madagascar, Cape Verde, and
Honduras and has approved oversight and management structures that
allow the countries to direct their compact programs, subject to MCC
review. However, these structures were not fully staffed until months
after the countries' compacts entered into force. According to MCC
officials, the delay in staffing has shortened the available time in
which to achieve compact goals. MCC has adopted a policy that
implements its authority to provide funding to hire officials prior to
compact signature, which may reduce delays in hiring in the future.
MCC Has Oversight Staff in Each Country, while Countries Retain
Management Responsibility:
Although MCC maintains a limited presence in each country and retains
some review and approval authorities, the countries have authority to
manage and oversee the compact program using MCC funds. In each
country, MCC has directly hired a resident country director and a small
staff. The director serves as MCC's public face and manages its
relationship with the compact country, guiding and overseeing the
country's efforts to complete needed plans and reports, implement
accountability mechanisms, and make appropriate use of resources. The
director also supervises management of the MCC office and staff, site
visits by MCC staff and technical teams, and engagement with MCC
headquarters. MCC retains authority and approval over several key
elements, including quarterly disbursements, plans such as those for
monitoring and evaluation, procurements above certain thresholds, and
the hiring of key employees. According to MCC officials, resident
country directors review matters that require MCC approval, sometimes
making a recommendation, before forwarding the matters to MCC
headquarters.
Each of the three countries has established structures to provide
oversight and management of their compact programs and to ensure
stakeholder input (see fig. 11). Although these structures have most
elements in common, the countries have the flexibility to design these
structures to fit their needs.
Figure 11: Compact Country Oversight and Management Structure:
[See PDF for image]
Source: GAO analysis of MCC data.
Note: This figure represents a composite of the Madagascar, Cape Verde,
and Honduras oversight and management structures. However, in Honduras,
stakeholder input is obtained through representatives on the steering
committee, rather than through a stakeholder committee.
[End of figure]
The three countries have generally included the following three key
oversight and management entities in their structures:
* The steering committee represents the country government and
interfaces directly with MCC. The committee is ultimately responsible
for the oversight and results of the MCC compact, oversees management
unit employees, and approves and signs off on key decisions and
reporting to MCC. Steering committee members vary by country but have
included government executives, such as representatives of the Prime
Minister's office; representatives of key ministries affected by
compact projects; and representatives of stakeholder groups, such as
civil society and the private sector. The MCC resident country director
serves as a nonvoting member of the steering committee. According to
MCC guidelines, MCC must approve the individual members of the steering
committee.
* In Madagascar and Cape Verde, a stakeholder committee meets
periodically to advise and inform the steering committee regarding
compact implementation and to serve as the official liaison between
interested parties and the steering committee. Stakeholder committee
members include representatives of civil society, the private sector,
and NGOs. MCC approves the type of organization that should be
represented on the committee (e.g., civil society) but does not approve
the specific individuals. MCC reserves the right to approve all changes
in the committee's membership.
* The management unit is directed by the steering committee and has
principal responsibility for overall compact management and
implementation.[Footnote 32] The compact lists key management positions
responsible for daily program operations. The steering committee hires
employees for these positions, but MCC retains the authority to approve
the country's choices.[Footnote 33]
Oversight and Management Entities Have Faced Staffing Delays:
In each of the three countries, key compact management positions have
remained unfilled after the compacts have entered into force.
Madagascar and Cape Verde did not hire key officials until several
months after their compacts' entry into force in July and October 2005,
respectively. As of April 2006, Honduras's hiring of key officials was
not yet complete, although the compact entered into force in September
2005. (See table 1.) This incomplete staffing at entry into force
limits the ability of the countries to achieve their compact objectives
within the fixed time period of the compacts.
Table 1: Hiring Dates for Key Positions Identified in Madagascar, Cape
Verde, and Honduras Compacts, as of April 2006:
Key position: Managing director:
Responsible for the overall operation of the management unit, including
all communications with third-parties and the public, including other
donors. Certifies all quarterly financial and performance reports,
budgets, and any other documents presented to the steering committee;
Hiring date: Madagascar: (entry into force: July 27, 2005): Incumbent
participated in compact development and served as interim director
prior to being confirmed on November 22, 2005;
Hiring date: Cape Verde[A]: (entry into force: Oct. 17, 2005): January
1, 2006;
Hiring date: Honduras: (entry into force: Sept. 29, 2005): The first
person to hold this position, from November 2005 to March 2006, left
after the change in ruling party from the January elections. The
current managing director was hired March 6, 2006.
Key position: Manager of administration and finance:
Responsible for ensuring that reporting is prepared, and that all
accounting records are maintained in a form acceptable to MCC;
Hiring date: Madagascar: (entry into force: July 27, 2005): November
22, 2005;
Hiring date: Cape Verde[A]: (entry into force: Oct. 17, 2005): March 1,
2006;
Hiring date: Honduras: (entry into force: Sept. 29, 2005): April 10,
2006.
Key position: Manager of procurement:[ B]; Serves as a liaison with the
procurement agent and for the preparation of periodic reporting to the
steering committee regarding procurement activities;
Hiring date: Madagascar: (entry into force: July 27, 2005): November
22, 2005[C];
Hiring date: Cape Verde[A]: (entry into force: Oct. 17, 2005): January
1, 2006;
Hiring date: Honduras: (entry into force: Sept. 29, 2005): Selection in
progress.
Key position: Manager of monitoring and evaluation:
Responsible for setting up the data collection, analysis, and reporting
systems for the overall program, and, in turn, for training and
assisting project managers in their implementation project monitoring
systems for each project;
Hiring date: Madagascar: (entry into force: July 27, 2005): January 19,
2006;
Hiring date: Cape Verde[A]: (entry into force: Oct. 17, 2005): January
1, 2006;
Hiring date: Honduras: (entry into force: Sept. 29, 2005): Selection in
progress.
Key position: Manager of environmental and social assessment:
Ensures that environmental and social mitigation measures are followed
for all activities of the program, in accordance with the provisions
set in the compact and other documents;
Hiring date: Madagascar: (entry into force: July 27, 2005): Position
does not exist at MCA-Madagascar;
Hiring date: Cape Verde[A]: (entry into force: Oct. 17, 2005): March 1,
2006;
Hiring date: Honduras: (entry into force: Sept. 29, 2005): April 1,
2006.
Key position: Managers of individual compact projects:
Project managers should have sectoral expertise relevant for their
respective projects. Project managers are responsible for the day-to-
day oversight and management of implementing entities.[];
Hiring date: Madagascar: (entry into force: July 27, 2005): Land
Tenure: Incumbent participated in compact development and served as
interim manager prior to being confirmed on November 22, 2005.
Finance: November 22, 2005.
Agricultural Business Investment: November 22, 2005; A new manager took
over this position on January 19, 2006; Hiring date: Cape Verde[A]:
(entry into force: Oct. 17, 2005);
Hiring date: Cape Verde[A]: (entry into force: Oct. 17, 2005):
Watershed Management and Agricultural Support: January 1, 2006.
Infrastructure: January 1, 2006.
Private Sector Development: January 1, 2006; Hiring date: Honduras:
(entry into force: Sept. 29, 2005);
Hiring date: Honduras: (entry into force: Sept. 29, 2005):
Rural Development: Selection in progress:
Transportation: April 1, 2006.
Source: MCC compacts and hiring data.
[A] According to MCC, in Cape Verde, some of the key staff "became
engaged" shortly after Compact signature--though the hiring process was
not complete. These positions, including the senior economist, project
managers, and monitoring and evaluation manager, were retained through
a provisional employment contractual arrangement made with the Ministry
of Finance.
[B] This position is referred to as the "procurement specialist" in
Honduras.
[C] Incumbent serves as manager of procurement and of administration
and finance.
[D] The Cape Verde compact, for example, defines an "implementing
entity" as a government affiliate, nongovernmental organization, or
other public or private sector entity or persons to which MCA-Cape
Verde may provide funding, directly or indirectly, to implement or
carry out the projects or any other activities in furtherance of the
compact.
[End of table]
The apparent reasons for these difficulties in hiring key personnel
vary by country. For example:
* In Madagascar, according to U.S., MCC, and country officials, finding
technically qualified candidates for key positions has presented a
challenge compounded by competition for their availability. According
to the Managing Director of MCA-Madagascar, hiring her MCA team was
difficult because "all the good people in Madagascar are taken."
* In Honduras, a postelection change in government has delayed staffing
the program management unit. After a recent national election, the
director of the management unit resigned to allow the new
administration to appoint a director. According to MCC, all other
management unit staff appointed under the previous government have been
retained.
* In Honduras, the monitoring and evaluation director position and one
of the project director (Rural Development) positions have been
difficult to fill. MCA-Honduras recompeted both of these positions
because the first attempt did not produce suitable candidates.
In October 2005, after the signature of its first six compacts, MCC
adopted a policy implementing the authority given it by section 609(g)
of the Millennium Challenge Act of 2003 to make grants to facilitate
the development and implementation of compacts. MCC's policy includes a
provision where, if certain conditions are met, it may fund an eligible
country's request for "management support payments" for salaries, rent,
and equipment for the country's MCA technical team prior to compact
signature. MCC has used this authority in four countries. Experienced
officials may therefore be hired and in place prior to compact
signature and the delays in hiring may be reduced in the future.
MCC Has Made Progress in Ensuring Fiscal Accountability, Although
Countries' Systems Are Still Being Developed:
MCC has established a framework of internal controls to incorporate
fiscal accountability into compacts between MCC and recipient
countries. The MCC fiscal accountability framework includes an
accountable entity (MCA), which is responsible for internal control and
at least an annual audit of the program's financial and procurement
transactions, with semiannual audits initially expected for countries
starting implementation.[Footnote 34] MCA-Madagascar and MCA-Cape Verde
both have made significant progress in implementing the framework's
internal controls.[Footnote 35] However, MCA-Madagascar's newly formed
system is somewhat less mature than that established by MCA-Cape Verde.
While MCA-Madagascar has established internal control processes, some
processes that are routinely practiced are not formally documented, and
it has not yet converted to a fully automated financial management
system. MCA-Cape Verde was established partly within the country's
Ministry of Finance, and it continues to develop internal controls, put
in place key finance and administration personnel, and make use of
existing information system platforms with the characteristics of
advanced and integrated systems. Neither country has yet had an MCA
unit in operation long enough to require an annual audit, but officials
at both MCAs told us that they expected their first audits to be
completed within the period established by the MCC IG.
MCC Has Made Progress in Fiscal Accountability at the Compact Level:
MCC has made progress in defining the mechanisms and processes of
internal control that provide reasonable assurance of fiscal
accountability at the compact level. MCC incorporated these internal
controls into a fiscal accountability framework for the compacts
between MCC and recipient countries. To help countries implement this
framework, MCC also developed guidance[Footnote 36] and a series of
charts that identify the characteristics of each element of the
framework, responsibilities of key personnel, and examples of
alternative methods of achieving fiscal accountability. Figure 12,
which is based on these charts, provides an overall view of the
framework.
Figure 12: Elements of the MCC Fiscal Accountability Framework:
[See PDF for image]
Source: GAO analysis of MCC data.
[End of figure]
Although MCC identifies the country's government as ultimately
responsible for meeting its fiscal accountability requirements, the
country's accountable entity is responsible for ensuring that basic
internal control functions--such as funds control and documentation,
cash management, disbursement controls, and timely and meaningful
reporting--are established as specified in the country's
compact.[Footnote 37] The fiscal accountability framework also requires
that the accountable entity obtain at least an annual audit of
financial and procurement transactions.
Each country's compact requires the accountable entity to develop its
own fiscal accountability plan (FAP), which typically includes a
transparent process for ensuring that open, fair, and competitive
procedures will be used to administer grants or cooperative agreements
and procurement. Required elements of the FAP include funds control,
internal controls, accounting standards and systems, reporting, public
availability of financial information, cash management, procurement and
contracting, the role of independent auditors, and the roles of fiscal
and procurement agents.
MCA-Madagascar and MCA-Cape Verde Have Implemented Internal Controls
for Fiscal Accountability, but Their Systems Are Not Yet Fully
Developed:
MCA-Madagascar and MCA-Cape Verde have made significant progress in
implementing internal controls for fiscal accountability, although
their systems are not yet mature or fully developed. In line with their
compacts and supplemental agreements with MCC, both countries have put
in place such elements of the fiscal accountability framework as bank
agreements, fiscal agents, procurement agents, and financial reporting
mechanisms; however, controls called for in the framework, such as
proper and consistent accounting and authorization of transactions, are
still in development. Because the controls are not yet fully
operational, we focused on those that have been put in place in
relation to an internal control maturity model that provides a scale of
development.[Footnote 38] We found that both MCAs differ in the
maturity of their internal controls. Madagascar is using a newly formed
entity; therefore, the maturity of MCA-Madagascar's internal control is
still at a developing stage. MCA-Cape Verde used an existing government
institution with established processes and systems as the accountable
entity and, therefore, represents a higher level of internal control
maturity. Neither country has yet been required to provide any
financial statement audits, but officials at both MCAs told us that
they expected their audits to be completed within the period specified
in their compacts.
MCA-Madagascar:
Although MCA-Madagascar has established important internal control
processes, some have not been formally documented, and processes in
several areas are in the early stages of development. (See table 2.)
The government of Madagascar proposed, and MCC agreed, to establish MCA-
Madagascar as a new unit of government with the sole purpose of
managing implementation of the MCC compact. The MCA has established
finance and administrative operations based on the MCC fiscal
accountability framework, with segregation of duties[Footnote 39] over
such functions as work-order approval, vendor selection, and
authorization to disburse funds. Standard operating procedures for
these processes are outlined in the MCA's FAP, which was approved by
MCC in January 2006. However, the FAP includes limited detail on asset
management, and sections in the management of tax exemption,
information technology management, communications, and audits serve
mainly as placeholders with information "to follow." Although MCA-
Madagascar acquired and is testing an automated accounting system,
current use of a manually based system leaves operations vulnerable to
human omission and error. The FAP makes reference to MCC's procurement
agreement guidelines but does not formally incorporate procurement
processes.
Table 2: Status of MCA-Madagascar Internal Controls, as of February
2006:
Internal control: Key capability attributes;
Status:
* Overall, a fiscal accountability framework is in place and
functioning;
* The FAP was approved by MCC in January 2006;
* Duties are segregated, with oversight of fiscal and procurement agent
activities performed by MCA- Madagascar officials;
* Bank agreements have been established;
* Expenditure process is in place;
* MCC conducts quarterly visits to review fiscal activities;
* Staff has been hired.
Internal control: Control efforts under way;
Status:
* Formal documentation that covers the internal control structure for
all significant transactions and events (i.e., standard operating
procedures) is being developed on the basis of existing documents and
practice;
* Procurement has not yet been formally incorporated into the FAP. At
the time of our review, the procurement manual was considered to be an
annex to the FAP.
Internal control: Potential residual risks;
Status:
* Accounting system carries the inherent risks of a manually based
system;
* The MCC IG risk assessment of the MCA-Madagascar's financial
operations identified high vulnerabilities in several critical areas--
procurement, cash management, and disbursements--that may adversely
impact financial operations;
* Lack of a disaster recovery or back-up plan could result in the loss
of critical data;
* Inadequately addressed data security could result in fraud, waste,
and abuse.
Source: GAO analysis of MCA-Madagascar systems and documents.
[End of table]
Although internal controls with and without formal documentation were
in operation in MCA-Madagascar at the time of our review in February
2006, a risk assessment performed by the MCC IG in December 2005 found
vulnerabilities in high-risk areas that could affect the MCA's
financial operations. The assessment was performed during MCA-
Madagascar's implementation phase to detect early-on any
vulnerabilities that could prevent the MCA from establishing effective
financial operations. The IG reported that the MCA had high
vulnerabilities in several critical areas--including procurement, cash
management, and disbursements--that may adversely impact its financial
operations, and that it had not implemented all of the key controls
described in its compact with MCC.[Footnote 40] The IG also reported
that the MCA's FAP was interim and not comprehensive in establishing
the internal controls needed to effectively manage the funds provided
by MCC.
MCA-Madagascar's internal controls are in development. It is putting in
place a control framework defined in MCC's fiscal accountability
framework, but the MCA currently has basic internal control policies
and processes that are not fully documented or implemented. Due to the
developmental stage of MCA-Madagascar's internal controls, residual
risks currently exist in its key financial processes and activities.
MCA-Cape Verde:
MCA-Cape Verde is developing its internal controls on the basis of the
Cape Verde government's existing structure and information system
platforms, which have the characteristics of advanced and integrated
systems (see table 3). The MCA was established in the Ministry of
Finance, and management is focused on implementing its interim FAP,
which was approved by MCC in December 2005. The MCA is using ministry
staff, and most key finance and administration personnel are in place.
To support the financial processing requirements set out in Cape
Verde's compact, the MCA is also using the ministry's financial
management information system and the basic accounting system that the
Cape Verde government has used for several years. Cape Verde's fiscal
agent, which is responsible for the MCA's financial processing,
consists of Ministry of Finance staff with experience in the budget,
treasury, and financial processes and in the use of the accounting
system. The MCA's expense transaction flows are documented, and key
duties and responsibilities are segregated.
Table 3: Status of MCA-Cape Verde Internal Controls, as of February
2006:
Internal control: Key capabilities;
Status:
* Fiscal agent and key finance and administration personnel are mostly
in place;
* Basic accounting system appears to be fully operational;
* The interim FAP is in place;
* Transparency of operations is enhanced by Cape Verde's Web site and
management oversight;
* Information system platform was preexisting in the Ministry of
Finance and is fully operational;
* Expenditure flows for MCA office transactions are documented;
* Procurement and payment authorizations are segregated.
Internal control: Control efforts under way;
Status:
* Currency conversion capability for accounting system is being
implemented;
* Conversion to commitment-based budgeting system is under way;
* Accounting policies and procedures manuals are being formalized;
* Outside audit entity is yet to be contracted;
* The FAP is being updated and finalized;
* Enhanced transparency initiative is to be completed, which includes
management tools.
Internal control: Potential residual risks;
Status:
* Currency conversion calculations and some conversion schedules are
being done manually, which could lead to potential errors or omissions;
* Complications may result from converting to an enterprisewide and
commitment-based budgeting system;
* Potential conflicts of interest could arise due to the different
ministries that currently have operational responsibilities and may
also participate in the oversight of the effectiveness and efficiency
of MCA-Cape Verde's operations.[A];
* Lack of a disaster recovery plan could result in the loss of critical
data.
Source: GAO analysis of MCA-Cape Verde systems and documents.
[A] MCC officials have stated that they believe this risk is mitigated
through clearly defined roles and responsibilities, autonomy of the
ministries, and specific controls over procurement approvals.
[End of table]
MCA-Cape Verde had internal controls across the entity, documented
transaction flows, sources of risk identified, and risk-mitigating
control processes that were mostly documented at the time of our
review. MCA-Cape Verde also had several internal control efforts under
way, including the development of policies and procedures manuals. As
these efforts continue, the MCA continues to have some residual risks
that are related to specific processes.
Financial Audits for MCA-Madagascar and MCA-Cape Verde:
MCA-Madagascar and MCA-Cape Verde are preparing for financial statement
audits as required in their compacts. In January 2006, the MCC IG
issued guidelines for MCA financial audits and a standard statement of
work for the MCAs to use in their financial audits and posted lists of
MCC-approved audit firms for Madagascar and Cape Verde.[Footnote 41]
According to MCC IG revised guidance issued in January 2006, compacts
are to have an audit completed no later than 90 days after the first
anniversary of the entry into force of the compact or such other
periods as parties may agree in writing. The MCC IG recently informed
us that the audits for MCA-Madagascar and MCA-Cape Verde are to be
completed by September 30, 2006, for the period from entry into force
until June 30, 2006. The MCC IG also told us that it waived the first
audit to be completed for MCA-Madagascar as stipulated in the compact
documents because the MCA did not have any financial activity for the
period to be audited.
Although we were able to assess the countries' progress in establishing
a system of internal controls to provide the financial accountability
required by MCC, it is too early to assess the effectiveness of those
controls over the funding provided to them. Each country has made
progress, but they are at different stages of maturity in their
underlying accountability systems, policies and procedures, and
internal controls. Oversight mechanisms that support program
effectiveness as compact projects disburse funds throughout multiple
countries and promote the segregation of procurement and payment
processes will be critical for the continual monitoring of progress,
the development of key accountability mechanisms that function as
intended, and the mitigation of risks to acceptable levels.
MCC-Approved Country Procurement Systems Have Effective
Characteristics, but Are Early in Their Implementation:
Consistent with MCC guidance, each of the three countries--Madagascar,
Cape Verde, and Honduras--proposed its own procurement system, subject
to MCC review and approval during due diligence. Although the approved
systems have characteristics that we have found typical of effective
procurement systems, their effectiveness has not yet been tested by
many procurements. In addition, some of the staff and procedures needed
to implement the systems are not yet in place at MCC headquarters or in
compact countries.
MCC Allowed Countries to Propose Procurement Agents and Systems, but
Retained Approval Rights:
To capitalize on existing country knowledge and experience, MCC has
given the countries flexibility to choose their procurement agents and
standards, with the choice subject to MCC approval during due
diligence. Madagascar and Cape Verde are using private and public
sector procurement agents, respectively, and Honduras is using a
combination of public and private sector agents.[Footnote 42] (See app.
IV.) In contrast to donors such as the World Bank, MCC also allowed the
countries to propose their own standards for managing
procurements.[Footnote 43] (See app. IV for details of the countries'
procurement agents and standards.) MCC required the countries to adhere
to "procurement principles" that include equal access to procurements,
competition for awards, and transparency of the process.[Footnote 44]
Madagascar, Cape Verde, and Honduras have used either existing World
Bank standards or their own laws to govern MCC procurements. In its
procurement agreement with each country, MCC included modifications to
the country's selected standards to reconcile them with U.S. law and
MCC principles. For example, MCC required that countries not include
preferences for domestic suppliers in solicitations paid for with MCC
funds.
Although MCC's agreements with the three countries give the countries a
number of authorities over procurements, MCC retains certain approval
rights. MCC's fiscal accountability framework describes procurement as
one of the highest-risk areas of fiscal accountability. MCC's compact,
disbursement agreement, and procurement agreement with each of the
countries describe the relationship, roles, and authority of MCC, the
procurement agent, and the compact country (see app. IV for details).
Although these agreements have some common elements, each agreement is
unique to the individual country. To determine when MCC approval of
individual actions is appropriate, MCC included review thresholds in
the procurement agreements keyed to procurement size and methods. Above
cost thresholds, which vary among countries, MCC must approve items
such as the procurement method, terms of reference, and selection.
Below thresholds, the compact country may conduct procurements in
keeping with the procurement plan without additional MCC oversight.
These thresholds are a risk management tool that maximizes MCC control
for larger transactions but leaves discretion to the compact country
for smaller transactions.[Footnote 45]
Countries' Procurement Systems Have Strengths, but MCC and Country
Systems Are Not Fully Established:
Although the three countries' procurement systems vary, each has
characteristics that we have previously identified as typical of
effective international procurement systems.[Footnote 46] These
characteristics are similar to the principles, such as equal access,
competition, and transparency, that MCC applies in its review of the
systems during due diligence. However, MCA officials in Madagascar and
Cape Verde told us that they had completed few procurements that would
test the systems in practice. Furthermore, MCC headquarters has not yet
finished hiring its procurement staff, and procurement systems in
Madagascar and Cape Verde are not yet fully established.
* Incomplete staffing. As of May 2006, MCC headquarters had hired its
senior director of procurement but had not yet hired five director-
level procurement staff in its Accountability Department. According to
MCC officials, the function of these vacant positions is currently
being filled by six intermittent personal service contractors, and MCC
has offered one of these contractors a full-time position.[Footnote 47]
During our site visits in January and February 2006, staff in
Madagascar and Cape Verde reported that the time frames for MCC review
of procurements had been satisfactory. However, if MCC staffing does
not increase as the countries submit more procurement decisions for
approval, MCC may have difficulty in conducting timely reviews of these
decisions.
* Incomplete systems. During our site visits, we found that some
elements of the procurement systems documented in MCC's agreements with
Madagascar and Cape Verde were not yet in place or had not functioned
smoothly.
* Neither country had yet established the procurement bid protest body
required by MCC or put in place automated systems for procurement
tracking and management, although both reported plans to do so in 2006.
Madagascar also had not established a process for reviewing contractor
change order requests.
* During our visit to Madagascar, a senior MCC official discovered that
the country's procurement plan for months 2 through 4 did not reflect
current project work plans. As a result, the procurement agent was
preparing for procurements that were no longer needed. Members of the
Madagascar management team told us that they would establish new
procedures to coordinate future work plan changes with the procurement
agent.[Footnote 48]
* In Cape Verde, the procurement review commission lacked office space
and was concerned about being able to handle the number of reviews
required of it once procurements begin in earnest. The commission is
comprised of members who are fully employed elsewhere and are not
permitted to delegate their work. The members have worked nights and
weekends or have negotiated with their supervisors to be released from
their other jobs to perform the commission's work. According to MCC,
subsequent to our site visit, Cape Verde has taken actions to mitigate
the risks to the efficient operation of the commission.[Footnote 49]
* Both Madagascar and Cape Verde reported difficulties with an MCC
requirement that documents be in both English and the local language.
Cape Verdean procurement review commission members are not required to
speak English but are expected to review documents prepared in English.
Cape Verde was considering hiring a full-time translator to address
this need.[Footnote 50] In Madagascar, the translation of a French
document prepared by the implementer of the finance project and
requested by MCC delayed the approval of needed project procurement.
MCC Has Established Country Monitoring and Evaluation Frameworks, but
Weaknesses May Limit Measurement of Results:
Each of the three countries' (Madagascar, Cape Verde, and Honduras)
programs includes a monitoring and evaluation framework that includes
plans for data collection, data quality reviews, analysis, and interim
and final reporting of results. We found several weaknesses in the
monitoring and evaluation frameworks that could affect MCC's ability to
track and account for program results.
Countries' Programs Include Monitoring and Evaluation Frameworks:
Each of the three countries' programs includes detailed plans for
monitoring and evaluating program results. MCC approved Madagascar's
and Cape Verde's plans in November 2005 and April 2006, respectively,
while Honduras's plan existed in April as a detailed draft but had not
yet been approved. According to MCC officials, Honduras's plan is not
final pending the staffing of the monitoring and evaluation director
position to ensure country understanding and buy-in. In accordance with
MCC guidance, the countries' plans include separate components for
monitoring and evaluation:[Footnote 51]
* The monitoring component includes, among other things, key indicators
that are linked as closely as possible to the variables identified in
the economic analysis of the country's proposed projects. These
indicators are to be used throughout implementation to assess whether
the program is likely to achieve the desired results. The monitoring
component also identifies baseline and target values for each indicator
and includes plans for periodic performance reports and data quality
reviews. MCC guidelines state that monitoring can be on select
indicators to minimize reporting requirements.[Footnote 52]
* The evaluation component identifies, among other things, the
methodology that will be used to assess the program's impact, such as
randomized controlled trials, and describes plans for collecting
baseline, interim, and final data on program results.
Countries' monitoring and evaluation plans use, to varying degrees, the
economic analysis of the proposed projects to identify indicators for
monitoring progress toward project objectives, calculate targets for
each indicator, and evaluate the achievement of compact goals. The
economic relationships specified in the models, such as the
relationship between improved infrastructure and farm output, provide a
basis for tracking project success. The monitoring framework also
includes setting target values for indicators. Figure 13 illustrates
indicators at various levels for a rural development project in
Honduras.
Figure 13: Indicator Structure for Honduras's Rural Development
Project:
[See PDF for image]
Source: GAO presentation of MCC data.
[End of figure]
Although the specifics of the countries' plans vary, the monitoring
component of each plan calls for the country's MCA to submit periodic
performance reports and data quality assessments to MCC. Performance
reports will include quarterly assessments to alert the countries and
MCC to any problems, periodic audits that analyze performance at all
compact levels, and annual reports that consolidate the quarterly
reports and recommend adjustments. Most performance data will be
gathered by project implementers, either country government employees
or contractors; the plans allow for contracting with other entities to
prepare the reports. In addition, each plan calls for third-party data
quality reviews over the course of the compact. For example, in
Madagascar, data quality assessments are planned to occur every 6
months during the first year and annually thereafter.[Footnote 53]
Weaknesses in Monitoring and Evaluation Frameworks May Limit MCC's
Ability to Measure Results:
In reviewing the frameworks for monitoring and evaluation in the three
countries, we identified several challenges that MCC faces in ensuring
accountability for results. These challenges include (1) ensuring the
availability and quality of baseline data, (2) establishing clear links
between the economic model and the monitoring and evaluation framework,
(3) accounting for the degree of uncertainty in expected outcomes, and
(4) using randomized controlled trials in compact countries.
Ensuring Baseline Data Availability and Quality:
Baseline data are essential to measuring the results of MCC-funded
compact projects. Although MCC has taken steps to collect baseline data
for monitoring and evaluation, problems with data availability and
quality may lead to challenges in measuring the progress and impact of
the countries' projects over time. MCC officials told us that they
worked with their country counterparts to set up a Management
Information System that can meet the requirements for collecting
performance data. In addition, MCC evaluated the technical capabilities
of the country staff and the information system the country proposes to
use for data management purposes. Finally, MCC budgeted funding for
surveys in Madagascar and Cape Verde to collect baseline data when it
was not available. However, we found that some of the baseline data in
the countries' monitoring plans were not complete, and that some of the
data MCC collected were not reliable.
* Baseline data availability. In some instances, the countries'
monitoring and evaluation plans lack complete baseline data against
which to measure progress. For example, two activity and final project
indicators in Madagascar's plan--"volume of production covered by
warehouse receipts in zones" and "volume of microfinance institution
lending in the zones"--currently lack baselines because the
intervention zones have not yet been selected. Moreover, although the
collection of performance data is closely linked to project
implementation, Madagascar's plan contains no intermediate outcome
indicators and target values, thereby making it difficult to
effectively track project progress. (See fig. 13 for an example of the
role of intermediate outcome indicators in the monitoring structure of
Honduras.)
* Baseline data quality. MCC may face challenges in ensuring the
quality of the baseline data that it uses to monitor and evaluate
program impact and, as a result, may have difficulty in accurately
measuring the impact of compact projects. MCC officials told us that it
has been difficult to obtain accurate and reliable baseline data
against which to measure program results. In some countries, MCC has
funded surveys to obtain the needed baseline data. However, even with
the additional resources provided by MCC, obtaining baseline data has
been a challenge. For example, we found significant data quality
problems associated with one of three surveys that MCC funded to
collect baseline data in Madagascar. Our interviews with Madagascar and
USAID officials who oversaw the survey revealed that the survey
results, which were used to estimate average land values, are flawed in
that they do not reflect recent significant changes in Madagascar's
currency.[Footnote 54] Madagascar's compact goal is to increase
household income, as measured by the percentage of increase in land
values.[Footnote 55] Because of the survey error, the land value
estimates may not be sufficiently reliable to evaluate the project
impact and the compact as a whole.[Footnote 56]
Linking Monitoring Plans to Economic Analyses:
Linking the indicators used to monitor and evaluate progress to the
data and assumptions used in MCC's due diligence economic analyses will
also present a challenge. In reviewing the draft plan for Honduras, we
found consistent linkages between the indicators for monitoring and
evaluation and the variables and assumptions used in the economic
model. However, in the plans for Madagascar and Cape Verde--the first
two plans that MCC approved--we found instances where MCC did not
sufficiently link the monitoring plans to the economic models, which
may hamper its ability to effectively measure project results. For
example:
* After signing its compact with Cape Verde, MCC changed the interim
targets for seven indicators. In two cases, neither MCC nor Cape Verde
was able to identify the methodology used to select the indicators in
the monitoring and evaluation plan.[Footnote 57] According to MCC
officials, they decided that the assumptions in the economic analysis
were a poor basis for constructing the monitoring indicators and,
therefore, chose other indicators and estimated the targets. According
to MCC officials, the inability to identify the methodology, in
conjunction with updated baselines and revised work plans in the
country resulted in MCC's and Cape Verde's agreeing to reduce the
interim targets that had been established.
* MCC's economic analysis for Madagascar's Land Tenure Project, which
is approximately one-third of the compact budget, did not identify the
expected benefits from the separate project activities. Therefore, we
could not track the linkage between the activities in the model to
those in the monitoring and evaluation plan.
* MCC's approved monitoring and evaluation plan does not include
tracking the results of two finance project activities--modernizing the
National Savings Institution and opening bank branches--although they
were included in economic analysis calculations during due diligence.
According to an MCC official, these two finance project activities will
be tracked at a higher level of aggregation--the finance project level-
-and monitored by tracking the number and value of new accounts.
However, this approach may not adequately capture the outputs and
benefits from the institution's modernization and could potentially
confound the effect of one activity (modernizing the institution) with
that of other activities. For example, while an increase in the number
and value of new accounts could result from the two finance project
activities, it could also result from an overall increase in savings if
customers invest in government savings bonds, issued recently and prior
to the start of the MCC compact.
Accounting for Uncertainty in Achieving Target Values:
Although the countries' monitoring and evaluation plans acknowledge the
uncertainty of achieving indicator target values, MCC project
monitoring does not adequately address (1) the effect of potential
variations in uncertainty on the range of acceptable target values or
(2) the plausibility of target values. As a result, some targets
specified in the countries' monitoring plans may not be achieved.
MCC disbursement agreements include as a condition precedent that, if
an indicator value observed during compact implementation does not fall
within 10 percent of the agreed-on target values, MCC may withhold
disbursements. MCC applies this 10 percent margin to all projects,
regardless of type (e.g., agriculture or infrastructure). However, our
analysis suggests that several factors could cause indicator values for
many projects to fall outside the 10 percent range.
* External factors. Uncertainty associated with external factors varies
by country and project. For example, according to previous GAO work,
external factors that could affect project implementation might include
political instability, the lack of commitment of political leaders to
necessary reforms, the magnitude of assistance from other bilateral and
multilateral donors, weather conditions that affect crop yields, and
the instability of international markets.[Footnote 58] These factors
could cause indicator values to fall outside the 10 percent range used
across all countries and projects.[Footnote 59]
* Time factor. The 10 percent range also does not account for the
increase in the uncertainty of targets over time--for example, target
values for year 5 of a compact are likely to be less precise than those
for year 1.
Therefore, to the extent that MCC bases its disbursement decisions on
results falling within a common range, it may not fully account for
variations in uncertainty across projects and over time.
According to MCC guidelines, the economic analyses and monitoring and
evaluation plans should, as much as possible, be clearly linked.
However, limitations in (1) the economic analyses due to problems with
data quality, assumptions, and lack of country involvement and (2) the
consistency between the economic analyses and the monitoring and
evaluation plans constrain MCC's ability to set plausible targets. If
targets are overly optimistic, countries may fail to reach them and MCC
may not be justified in halting disbursements because the countries
failed due to unattainable targets. Conversely, setting too
conservative a target may not prompt the country to fully utilize MCC
resources. A lack of plausible targets may lead to MCC's making ad hoc
decisions regarding the consequences of missing targets and applying
judgment subjectively and inconsistently in setting or modifying
targets. MCC officials told us that they would use a missed target as a
cue to discuss with the country's MCA its reasons for missing the
target, and, on the basis of those discussions, they would determine
whether to use their authority to withhold funding. According to MCC
officials, senior management approval would be needed to significantly
modify targets; however, MCC currently has no policy or documentation
that defines a "significant modification" of targets, especially for
targets used as conditions precedent to disbursements.
Using Randomized Controlled Trials in Compact Countries:
MCC has retained research organizations to help the countries evaluate
program impact using, as appropriate, randomized controlled trials, but
MCC's involvement of these organizations after project implementation
begins may limit their ability to evaluate impact accurately.[Footnote
60] These organizations' scope of work may include training MCC and
compact country staff; designing the trials and data collection; and
proposing appropriate methodologies for, and analyzing results from,
impact evaluations. According to MCC monitoring and evaluation
officials, MCC has begun designing impact evaluations by identifying
those program components that can and cannot be evaluated using
randomized controlled trials, which MCC has indicated are its preferred
method of impact evaluation.[Footnote 61] According to MCC officials,
MCC considers evaluations using randomized controlled trials as
"rigorous" and evaluations using other methodologies as "standard."
MCC has hired an independent consultant experienced in impact
evaluations to work with compact countries to assess the
appropriateness of using randomized trials to evaluate MCC's projects.
When these assessments are completed, the five research organizations
will be invited to compete to conduct randomized trials after compact
implementation begins.[Footnote 62] However, at that point, the
organizations will not have had an opportunity to assess the design of
the countries' evaluation strategy, including the adequacy and
reliability of the baseline data. Without the involvement of these
organizations before implementation of the relevant project(s) begins,
MCC may not be able to ensure that they have the necessary data and
have established appropriate research designs for their work. MCC
officials told us that they thus far had not involved the five
organizations because rigorous evaluations were turning out not to be
feasible in some cases, or because the tasks were not large enough to
warrant the use of the five research organizations.[Footnote 63]
Conclusions:
MCC continues to mature and evolve as an institution, taking on the
ambitious task of creating new, country-managed organizations while
developing processes to oversee what are expected to be relatively
large amounts of foreign assistance. Toward that end, MCC has taken
positive steps with regard to establishing policies and procedures for
MCA organizations. However, it has taken time to complete the numerous
agreements necessary for compacts to enter into force. This could
continue to present challenges, given that MCC is working
simultaneously with a number of nations to develop and implement
compacts.
As MCC moves forward, partnering with countries to develop well-founded
economic assumptions will be crucial to establishing a foundation for
the work of MCC and its partners. Furthermore, holding countries
accountable for results requires, to the extent practical and cost-
effective: collecting reliable and accurate baseline data, linking
economic analyses to monitoring plans, addressing the uncertainty
associated with program results, and ensuring the timely development of
the research design for randomized controlled trials.
Recommendations:
Because of the central role of reliable economic analyses and the
importance of partnering with countries in achieving MCC goals and
ensuring accountability for MCC programs, we recommend that the Chief
Executive Officer of MCC take the following two steps:
* Ensure that MCC officials, in partnership with country
representatives, perform economic analyses that more fully reflect the
countries' socioeconomic environment and are better understood by
country public and private sector representatives.
* To the extent practical and cost-effective, improve MCC's monitoring
and evaluation capabilities by:
* obtaining more accurate and reliable baseline data needed to permit
tracking progress during compact implementation;
* ensuring a clear linkage between MCC's economic analyses and
monitoring and evaluation frameworks;
* developing policies, procedures, and criteria for establishing
targets and for adjusting those targets if unforeseen events affect
outcomes; and:
* taking steps to ensure the timely development of the needed research
design for randomized controlled trials, if they are undertaken, prior
to project implementation.
Agency Comments and Our Evaluation:
We received written comments on a draft of this report from MCC and the
Department of State. In commenting on a draft of this report, MCC
generally agreed with our findings, conclusions, and recommendations.
MCC noted that our discussion of the evolving guidance provided to
eligible countries in 2004 and 2005 was a result of the complex process
of engaging with eligible countries while simultaneously developing
policies and procedures. MCC stated that this criticism should not be
valid beyond the initial instances covered in this report. We recognize
that MCC was simultaneously addressing a number of issues during this
period, but we felt it was important to discuss the evolving nature of
guidance to provide a balanced perspective regarding the process
eligible countries had to follow to sign compacts.
MCC also commented (1) that our characterization of data quality issues
in Madagascar cited only a single survey and (2) that it differed with
us on the appropriate level of aggregation in linking economic models
and monitoring and evaluation plans. We note that there was one other
instance of poor data quality in Madagascar, but that we focused on the
Agricultural Productivity Survey because of the importance of
accurately tracking land values to monitor results. We agree that
disaggregation may not always be feasible, but note that aggregation
poses some challenges that could limit the effectiveness of monitoring
and evaluation. We reprinted MCC's comments, with our responses, in
appendix V. We also incorporated technical comments from MCC in our
report where appropriate.
State commented that some of our findings reflect minor or transitory
problems and provided specific observations regarding MCC's evolving
guidance, the assumptions used in MCC's economic models, country
participation, staffing delays, fiscal accountability structures, and
the use of randomized controlled trials.
State noted that MCC's guidance could be expected to evolve, given the
newness of the organization, and that informal guidance from MCC was
always available to eligible countries. We agree that guidance could be
expected to evolve, but we sought to provide a balanced perspective by
noting instances where MCC's verbal guidance may not have been
sufficient to assist countries in submitting proposals that met MCC's
criteria.
State questioned the findings from our Madagascar focus groups and our
finding that MCC conducted economic analyses with limited country
involvement. Our focus groups resulted in discussions of the
assumptions used in the Madagascar economic analysis with a broad range
of country stakeholders representing U.S. and Malagasy agencies and
organizations involved in implementing the compact. Furthermore, MCC
agreed in its comments that, in some cases, the level of country
engagement on economic analysis could be improved. MCC outlined
specific steps that it had taken to increase country involvement in
economic analyses.
Regarding staffing delays and fiscal accountability structures, State
commented that we offered no suggestions to MCC for things they could
have done differently. We added additional material to the report to
elaborate on the steps MCC has taken to reduce delays in staffing key
positions. In regard to fiscal accountability, we agree that different
countries will differ in their maturity of internal control. However,
evaluating maturity is key to properly assessing risk and establishing
effective oversight mechanisms.
Finally, State disagrees with what it terms our "reliance" on
randomized controlled trials to measure success. This comment
misconstrues our findings. We did not rely on or advocate this
methodology, but rather are commenting on MCC's use of randomized
controlled trials as its preferred method of impact evaluation.
We have reprinted State's comments, with our responses, in appendix VI.
We are sending copies of this report to interested congressional
committees as well as the Secretary of State, the Secretary of the
Treasury, the CEO of MCC, and the Administrator of USAID. We will also
make copies available to others upon request. In addition, this report
will be available at no charge on the GAO Web site at [Hyperlink,
http://www.gao.gov].
If you or your staff has any questions about this report, please
contact David Gootnick at (202) 512-3149 or gootnickd@gao.gov. Contact
points for our Offices of Congressional Relations and Public Affairs
may be found on the last page of this report. GAO staff who made major
contributions to this report are listed in appendix VII.
Sincerely yours,
Signed by:
David Gootnick:
Director:
International Affairs and Trade:
Signed by:
Jeanette Franzel:
Director:
Financial Management and Assurance:
[End of section]
Appendix I: MCC Actions in Response to April 2005 GAO Recommendations:
The Millennium Challenge Corporation (MCC) has taken a number of steps
to address our April 2005 recommendations regarding its strategic
planning, internal controls, and human capital and governance policies,
although some aspects of its organizational structure are not yet
complete. MCC has prepared a strategic plan for fiscal years 2006
through 2011, and an annual performance plan for fiscal year 2006. MCC
also has strengthened its internal controls and taken steps to
implement some fiscal accountability mechanisms, established an audit
committee within its governing board, and completed several required
audits and reviews. However, MCC has not documented all financial
control activities and continues to face risks from the poor
interfacing of its systems with those of the Department of the
Interior's National Business Center (NBC). MCC has reassessed its
staffing model, developed a plan for recruitment, and implemented an
improved performance management system; but, it does not systematically
track the use of staff resources to verify its human capital model.
Finally, MCC has approved a corporate governance policy and taken steps
to improve board involvement in planning, management, and
communication; but it has not yet fully addressed risk management for
the corporation.
MCC Has Developed a Strategic Plan to Enhance Accountability and
Completed a 2006 Performance Plan:
Consistent with our recommendation to enhance corporate accountability,
MCC completed a strategic plan, approved by the Office of Management
and Budget (OMB) in November 2005. In April 2006, MCC completed an
annual performance plan that provides goals and benchmarks for
assessing its performance in fiscal year 2006. This annual performance
plan will enable MCC to report in the future on its progress in meeting
its goals.[Footnote 64] As part of the annual performance plan process,
MCC also has developed goals and benchmarks for its individual
departments that support the attainment of the corporate goals
identified in the strategic plan. (See table 4.)
Table 4: MCC Response to GAO Recommendations for Corporatewide
Accountability:
Summary of GAO recommendation: Implement a strategic plan;
MCC actions completed: Completed strategic plan, November 2005;
MCC actions not completed or in progress: N/A.
Summary of GAO recommendation: Establish annual performance plans and
goals;
MCC actions completed: Annual performance plan completed in April 2006;
MCC actions not completed or in progress: N/A.
Summary of GAO recommendation: Use performance measures to monitor
progress in meeting both strategic and annual performance goals;
MCC actions completed: Annual performance plan completed in April 2006;
MCC actions not completed or in progress: N/A.
Summary of GAO recommendation: Report internally and externally on
progress in meeting strategic and annual performance goals;
MCC actions completed: [Empty];
MCC actions not completed or in progress: Completed annual performance
plan provides benchmarks against which to measure MCC's fiscal year
2006 performance.
Sources: GAO interviews and analysis of MCC documents.
[End of table]
MCC Has Made Significant Progress in Establishing Internal Controls:
In response to our April 2005 recommendations, MCC has made significant
progress in establishing internal controls over program and
administrative operations at both the MCC headquarters and compact
levels.[Footnote 65] MCC has made progress in each of the five
components of internal controls discussed in our April 2005
testimony:[Footnote 66] (1) control environment, (2) risk assessment,
(3) control activities, (4) monitoring, and (5) reporting.
* Internal control environment. MCC has initiated several measures to
establish a positive internal control environment, including
documenting its organizational structure for the Administration and
Finance and Accountability structures. The Fiscal Accountability area,
which is a component of the Accountability structure, currently
consists of a managing director and several directors; however, other
positions to support the fiscal oversight of MCC compacts have yet to
be filled. A formal ethics program has also been established for MCC
headquarters.
* Risk assessment. MCC is developing a process for assessing risks
facing the corporation and its programs. To this end, MCC has hired a
third-party consulting firm to support the implementation of processes,
based on OMB A-123, Management's Responsibility for Internal Control,
December 2004, criteria, to address risks associated with domestic and
foreign operations.
* Control activities. MCC has instituted several control activities to
reduce risk. MCC has submitted its Strategic Plan under the Government
Performance and Results Act (GPRA) of 1993 and has completed required
audits and reviews, such as those required under the Federal Managers
Financial Integrity Act (FMFIA) of 1982 and the Federal Information
Security Management Act (FISMA) of 2002.[Footnote 67] MCC is in the
process of addressing the various material weaknesses, reportable
conditions, findings, and recommendations identified in the audits. For
example, MCC continues to address the formal documentation of control
activities for the financial reporting process at MCC headquarters. MCC
also contracted with a third-party service-provider, NBC, to maintain
its accounting system and the recommended Statement on Auditing
Standards No. 70 (SAS 70) review[Footnote 68] was performed; however,
MCC is still addressing issues reported as a result of the review, such
as the need to address manual processes that present inherent risks
associated with accounting systems and related processes. For example,
MCC forwards daily to NBC a package containing source documentation
that is used by NBC to record transactions to the general ledger.
Similarly, information for MCC's travel expenses is prepared at MCC
headquarters office and daily communicated to NBC through manual
processes. MCC is aware of the need to address these issues and is
working with NBC to mitigate or eliminate the manual processes.
* Monitoring. MCC has taken steps to ensure ongoing monitoring and
periodic testing of control activities. MCC's investment committee
embodies functions of monitoring and testing and operating as an
integral part of MCC's internal control program by overseeing the
program units' compliance with both the procedural and substantive
elements required by its internal processes. Also, MCC conducted its
first comprehensive survey of internal controls, performed by outside
consultants, in conjunction with its annual audit. In addition, MCC has
formed formal review panels to monitor the progress of addressing
findings from internal and external reviews. For example, consistent
with OMB guidance, MCC formed an FMFIA Management Review Panel to
assess the results of the internal control survey along with the
findings of MCC's independent financial auditors. Similarly, MCC
implemented a specific procedure to address recommendations from
reviews and audits performed by its Inspector General (IG).
* Reporting. MCC has made progress in establishing a process for
assessing and reporting on the operating effectiveness of its internal
controls. MCC established a formal board-level audit committee whose
responsibilities include (1) financial controls; (2) the integrity of
the reporting process; and (3) performance of the independent audit
process. In addition, MCC's FMFIA Management Review Panel assessed the
results of the internal control survey, along with the findings of
MCC's independent financial auditors. The panel identified four
material weaknesses and actions that MCC will be taking in future
months to resolve them, which the acting Chief Executive Officer
certified on November 7, 2005.
MCC officials told us that the development of new internal control
policies and procedures and the revision of those already in place is a
continuing process as MCC continues to mature as an organization. Table
5 summarizes MCC's progress in addressing our April 2005
recommendations.
Table 5: MCC Response to GAO Recommendations for Internal Controls over
Program and Administrative Operations:
Summary of GAO recommendation: Complete the development and
implementation of overall plans and related time frames for actions
needed to establish the following:
Summary of GAO recommendation: A positive and supportive internal
control environment;
MCC actions completed: Documented the organizational structure;
Documented a corporate strategy;
Implemented a procedure for addressing IG audit findings;
Established formal ethics program, including training for all MCC
employees;
Conducted audits required under FISMA and FMFIA;
Submitted Strategic Plan under GPRA;
MCC actions not completed or in progress:
* Positions within the accountability area remain vacant;
* Continued efforts to address audit findings of material weaknesses
and reportable conditions.
Summary of GAO recommendation: A process for ongoing risk assessment;
MCC actions completed: Identified 14 priority internal control areas
relating to administrative and program risks;
MCC actions not completed or in progress:
* Risk assessment process is still being developed;
* Outside consulting firm to support MCC risk analysis based partly on
OMB A-123 criteria.
Summary of GAO recommendation: Control activities and procedures for
reducing risk, such as measures to mitigate risk associated with
contracted operational and administrative services;
MCC actions completed: Internal Controls Strategy Group identified 14
key internal control areas for MCC to focus efforts;
MCC actions not completed or in progress:
* MCC continues to work on implementing and formally documenting some
procedures and control activities.
Summary of GAO recommendation: Ongoing monitoring and periodic testing
of control activities;
MCC actions completed: Established formal review panels to monitor
progress of efforts to address findings from internal and external
reviews and formally documented a process for addressing MCC IG
recommendations;
MCC actions not completed or in progress: N/A.
Summary of GAO recommendation: A process for assessing and reporting on
the effectiveness of internal controls and addressing any weaknesses
identified;
MCC actions completed: Created a board-level audit committee to monitor
MCC's financial controls, integrity of the financial reporting process,
and performance of the audit process;
MCC actions not completed or in progress: * MCC continues efforts to
establish a comprehensive database to track recommendations.
Sources: GAO interviews and analysis of MCC documents.
[End of table]
MCC Has Taken Steps to Develop Human Capital, but Does Not Track
Allocation of Human Capital to Key Activities:
MCC has taken steps and is continuing to further develop its human
capital systems by (1) assessing its staffing needs; (2) improving its
recruitment, development, and retention systems; and (3) implementing a
performance management system linking compensation to individual
contributions toward corporate goals. However, despite having plans to
increase its staff by an additional 38 percent between May and
September 2006, MCC does not systematically assess its staffing needs,
has not developed a human capital plan, and has not yet fully
implemented its improved performance management system, as follows:
* Staffing. Although MCC has completed an assessment of its human
capital needs since our April 2005 recommendation, it does not
systematically track the use of staff time on an ongoing basis. MCC's
updated assessment of its human capital needs shows that it plans to
increase its staffing from 218 staff, as of May 2006, to approximately
300 staff, as of September 2006.[Footnote 69] MCC also has created an
organization chart that includes the specific approved positions for
each department under the new 300-person staffing model, and it is
hiring for many of these positions using limited-term appointments to
provide greater flexibility in filling future needs.[Footnote 70]
According to MCC officials, MCC made its case to OMB for increasing
staffing to 300 persons on the basis of an analysis of MCC staffs'
recollection of the amount of time they spent developing the Georgia
compact, which was thought to have been the most complex compact
development process to date. However, MCC officials felt that this
analysis may not have fully captured the amount of time spent by some
departments in developing the compact. Retrospective analysis was
necessary because MCC does not track employees' time on mission-related
projects on an ongoing basis. Without such data, MCC management is not
able to systematically assess the staffing requirements needed to carry
out MCC's mission and consistently align its human capital with its
changing needs.
* Recruitment and retention. MCC has identified priorities and
committed resources for recruitment and is developing a human capital
plan to address retention and training. To support its effort to hire
approximately 82 additional staff between May and September 2006, MCC
has retained an outside consultant firm to work on-site and help with
recruitment and has identified positions as first-or second-tier hiring
priorities. MCC officials also told us that they are developing an
overall human capital plan that will include planned activities and a
time frame for identifying critical skills and competencies for MCC's
key positions. The officials stated that the human capital plan also
will include a strategy for staff retention and will address staff
training. Currently, MCC has developed procedures for providing
employees with outside training. MCC intends to develop a comprehensive
training plan following the completion of the human capital plan, a
draft of which it expects to circulate to MCC senior staff for their
review and comment by September 30, 2006.
* Performance management. In keeping with our recommendation, MCC has
established a performance-based compensation framework. MCC provided us
with documentation of its employee ratings process, showing that
employee expectations and performance reviews were keyed to
organizational goals. However, according to MCC officials, MCC did not
incorporate the departmental performance plans for the year into the
performance framework for 2006 until March 2006, as the annual
performance plan neared completion. MCC is shifting from a calendar
year to a fiscal year performance evaluation schedule to better align
employee compensation with its annual corporate goals. MCC anticipates
that its strategic plan, annual performance plan, department plans, and
individual performance goals will be fully synchronized beginning in
fiscal year 2007. (See table 6.)
Table 6: MCC Response to GAO Recommendations for Human Capital
Infrastructure:
Summary of GAO recommendation: Establish an effective human capital
infrastructure, including:.
Summary of GAO recommendation: A thorough and systematic assessment of
the staffing requirements and critical skills needed to carry out MCC's
mission;
MCC actions completed: MCC has reassessed its human capital needs at
approximately 300 staff, and OMB has approved this model; MCC has
completed an updated organization chart identifying the positions to be
filled under the new human capital model;
MCC actions not completed or in progress: * MCC currently does not
track the amount of time its employees spend on specific tasks. Without
this information, MCC cannot systematically assess its staffing needs
on an ongoing basis.
Summary of GAO recommendation: A plan to acquire, develop, and retain
talent that is aligned with the corporation's strategic goals;
MCC actions completed: MCC has identified priorities for hiring based
on its 300-staff model; MCC has retained an outside consultant firm in
support of its effort to increase staff from 218 in May 2006 to
approximately 300 at the end of September 2006; MCC has developed a
procedure for employee training;
MCC actions not completed or in progress: * MCC has not completed an
overall human capital plan but plans to circulate a draft to MCC senior
staff for review and comment by September 30, 2006;
* MCC plans to include a retention strategy and address training in the
human capital plan;
* MCC also plans to develop a comprehensive training plan following the
completion of the human capital plan.
Summary of GAO recommendation: A performance management system linking
compensation to employee contributions toward the achievement of MCC's
mission and goals;
MCC actions completed: MCC has established a performance framework
that links compensation to employee expectations and performance
reviews to MCC goals;
MCC actions not completed or in progress:
* MCC is shifting from a calendar year schedule to a fiscal year
schedule of performance evaluation to better align employee
compensation with its annual corporate goals;
* MCC anticipates that its strategic plan, annual performance plan,
department plans, and individual performance goals will be fully
synchronized beginning in fiscal year 2007.
Sources: GAO interviews and analysis of MCC documents.
[End of table]
MCC Has Taken Steps to Define Corporate Governance:
The MCC Board of Directors (MCC Board) has taken steps to define the
scope of its corporate governance and oversight.[Footnote 71] The MCC
Board has approved a corporate governance policy developed by MCC with
the involvement of staff from MCC Board agencies.[Footnote 72]
According to MCC officials, the policy incorporates guidance on
governance matters provided by board members at their previous
meetings, and the board participated in formulating MCC's strategic
plan before approving it in December 2005. The board has established a
board-level audit committee and a charter for that committee. According
to MCC officials, to address risk, MCC is recruiting for the position
of risk specialist and has used a contractor to support MCC risk
analysis. Finally, to improve communication with stakeholders in
eligible countries, MCC has published and distributed updated
guidelines for compact development and eligibility. MCC also has
developed a series of open forums where input is sought from groups
with an interest in MCC. (See table 7.)
Table 7: MCC Response to GAO Recommendations for Corporate Governance:
Summary of GAO recommendation: Recommended that the Secretary of State
ensure that the MCC Board considers and defines the scope of its
responsibilities with respect to corporate governance and oversight of
MCC…including oversight of the following:
Summary of GAO recommendation: Executive management;
MCC actions completed: The MCC Board has approved a corporate
governance policy;
MCC actions not completed or in progress: N/A.
Summary of GAO recommendation: The formulation and execution of
corporate strategies;
MCC actions completed: The board participated in formulating MCC's
strategic plan and approved the plan;
MCC actions not completed or in progress: N/A.
Summary of GAO recommendation: Risk management and audit and assurance
processes;
MCC actions completed: The MCC Board has formed an audit committee and
approved a charter for that committee;
MCC actions not completed or in progress: MCC has not yet hired a risk
specialist for the corporation but has used an outside consulting firm
to support MCC risk analysis.
Summary of GAO recommendation: Communication and coordination with
corporate stakeholders;
MCC actions completed: MCC has published and distributed updated
guidance on compact development and implementation; MCC has developed
an outreach program of open forums;
MCC actions not completed or in progress: N/A.
Sources: GAO interviews and analysis of MCC documents.
[End of table]
[End of section]
Appendix II: Scope and Methodology:
At the request of the Senate Committee on Foreign Relations, we
examined the structures and procedures MCC has developed in
consultation with compact countries to manage compacts. Specifically,
our work focused on (1) the key areas that MCC examined in its due
diligence assessments of proposals for Madagascar, Cape Verde, and
Honduras, and the criteria that MCC used in these assessments, and (2)
the form and adequacy of the implementation structures that MCC and
compact countries have put in place for governance, procurement, fiscal
accountability, and monitoring and evaluation. In addition, we reviewed
MCC's progress in responding to our April 2005 recommendations on its
corporate management and accountability structures (see app. I).
To accomplish our objectives, we reviewed MCC's documentation of its
processes and agreements, supplemented by interviews with MCC
officials. We focused our review for objectives 1 and 2 primarily on
the first three countries with signed compacts--Madagascar, Cape Verde,
and Honduras. These countries' compacts were the first to enter into
force. To further our analysis for objectives 1 and 2, we also visited
Cape Verde and Madagascar in January and February 2006. We selected
Cape Verde and Madagascar for our site visits because they had advanced
further than Honduras in filling key positions and beginning compact
implementation. While in Cape Verde and Madagascar, we interviewed a
number of MCC, Millennium Challenge Account (MCA), and government
officials and visited project sites.
To identify MCC's evaluation criteria and process for evaluating
eligible country proposals in due diligence, we reviewed MCC guidance
and the record of MCC analysis contained in (1) MCC's "due diligence
books," which are its internal records of how it assessed proposals
submitted by Madagascar, Cape Verde, and Honduras, and (2) investment
memos, which are MCC's analyses based on due diligence and internal
recommendations to its investment committee. These documents are
restricted from public dissemination due to their sensitive nature, but
MCC made them available to us for analysis. We have coordinated with
MCC on describing the information from these books in general terms
without disclosing sensitive information. We used MCC's definition of
the due diligence process as beginning with MCC's opportunity memo and
ending with the acceptance of the investment memo by MCC's investment
committee. Our review, therefore, may not capture some changes and
decisions made by MCC or eligible countries during proposal development
and compact negotiations. To evaluate MCC's assessments of proposals'
consultative process, project coherence, environmental and social
impact, and institutional and financial sustainability, we relied
primarily on MCC's data and analysis contained in the due diligence
books and, to some extent, in the investment memos. We compared MCC's
analysis in these documents with criteria outlined in MCC's guidance.
We were able to perform only limited independent verification of the
use and adequacy of these criteria during our site visits. With regard
to its economic analyses, MCC also made available to us the spreadsheet
models it used to develop the economic rate of return calculations that
formed the basis for its evaluations of the suitability of country-
proposed projects for MCC funding. We independently analyzed these
spreadsheets and validated their logic and conclusions on the basis of
a review of economic literature and practices. In addition, in
Madagascar, we conducted a series of focus groups with country
officials to assess the data and logic used by MCC in developing their
economic analysis.
To assess MCC's compact implementation structures, we reviewed the
compacts with Madagascar, Cape Verde, and Honduras and the supplemental
agreements required for those compacts to enter into force. We
supplemented this review with our site visits to Madagascar and Cape
Verde. In all cases, our ability to analyze the adequacy of these
structures was limited by their relative newness and limited use in
actual implementation. We addressed the following four areas of MCC's
implementation structures:
* To determine the form of governance structures and key positions in
these three countries, we reviewed the requirements of MCC compacts and
supplemental agreements. We determined the progress of the country
organizations in Madagascar, Cape Verde, and Honduras in filling these
positions and establishing these structures by analyzing MCC's reported
staffing and status information. We independently assessed MCC's
progress in our site visits to Madagascar and Cape Verde. We discussed
factors affecting the filling of these positions through discussions
with MCC and compact country officials.
* To assess the adequacy of the countries' fiscal accountability
structures, we reviewed MCC's overall fiscal accountability framework
and the operations in place in Madagascar and Cape Verde. We assessed
the adequacy of these structures according to the criteria contained in
GAO's Standards for Internal Control in the Federal
Government.[Footnote 73] We assessed MCC's and the countries'
implementation of these structures by using criteria in the Internal
Controls Capability Maturity Continuum developed by the independent
risk consulting company, Protiviti, Inc. We independently verified the
existence of the structures described in the plan and discussed its
strengths and weaknesses during the site visits.
* To assess MCC's procurement structures, we reviewed the MCC fiscal
accountability framework, and the implementing procurement documents
for the first three compact countries. We then assessed the adequacy of
MCC's framework, using criteria identified in previous GAO reports on
international procurement. To determine the status and factors
affecting the implementation of this framework in Cape Verde and
Madagascar, we interviewed compact country officials and obtained
documentation of procurement procedures.
* To determine the form of monitoring and evaluation structures in the
three countries with entry into force, we reviewed the requirements of
MCC compacts and supplemental agreements. We assessed the status of the
country organizations in Madagascar, Cape Verde, and Honduras to
establish these structures by analyzing the staffing and status
information provided to us by MCC. We independently assessed MCC's
progress in our site visits to Madagascar and Cape Verde. Additionally,
we reviewed the scope of work of the independent U.S. evaluation
contractors retained by MCC, and closely reviewed the monitoring and
evaluation plan for Madagascar--the only plan approved by MCC prior to
April 2006. We also reviewed the plan for Cape Verde and the draft plan
for Honduras. We assessed the adequacy of the Madagascar plan against
the criteria of data quality, and consistency with the economic model
and logic identified in MCC's due diligence review of projects. We also
applied general principles of economic logic, such as treatment of
uncertainty in data, to assess how uncertainty was incorporated into
MCC's monitoring and evaluation framework.
To review MCC's progress in responding to GAO's April 2005
recommendations (see app. I), we examined MCC documents, such as its
strategic plan, planning documents, policies, procedures, and human
capital documents. In December 2005, MCC provided us with a letter
outlining the steps that the corporation had taken in response to our
recommendations. Using this as a basis for discussion, we held
additional meetings with MCC officials and received additional
documentation of MCC's responses. We also reviewed the findings of the
MCC IG analysis of the functions of the Corporation and met with the IG
to determine the steps that MCC had taken in response to IG findings
related to our recommendations.
We conducted our review from June 2005 through May 2006 in accordance
with generally accepted government auditing standards.
[End of section]
Appendix III: MCC Projects in Madagascar and Cape Verde:
Figure 14: Jatropha Plantation, outside Antsirabe, Madagascar, Targeted
for Assistance by the MCA-Madagascar Agricultural Business Investment
Project:
[See PDF for image]
Source: GAO.
[End of figure]
Figure 15: Antananarivo Land Records Storage Room, Programme National
Foncier Offices:
[See PDF for image]
Source: GAO.
Note: Digitization of these land records is being supported by MCA
funds.
[End of figure]
Figure 16: Port of Praia, Cape Verde:
[See PDF for image]
Source: GAO.
Note: The quay of the current port can only accommodate one large
international vessel. Under the MCA program, the quay will be expanded
to almost double its current length, enabling three ships to dock at a
time.
[End of figure]
Figure 17: The Road from Assomada to Rincao, Cape Verde:
[See PDF for image]
Source: GAO.
Note: Using MCA funds, about half of the road will be converted from
cobblestone to asphalt. The cobblestones from the first half will be
used to pave the second section of the road leading up to Rincao.
[End of figure]
[End of section]
Appendix IV: Summary of MCC Procurement Agents, Standards, and
Provisions in Madagascar, Cape Verde, and Honduras:
Table 8: Procurement Agents and Standards in Madagascar, Cape Verde,
and Honduras:
Country: Madagascar;
Type of procurement agent: Nongovernmental;
Procurement agent: GTZ[A];
Procurement standards: Madagascar Law Number 2004-009 Dated July 26,
2004 on Public Contracts subject to listed exceptions in Procurement
Agreement.
Country: Cape Verde;
Type of procurement agent: Governmental;
Procurement agent: Cape Verde accountable entity;
Procurement standards: World Bank Guidelines: Procurement Under
International Bank for Reconstruction and Development Loans and
International Development Association Credits, May 2004, as modified
for nonconsultant services, works and goods.
World Bank Guidelines: Selection and Employment of Consultants by World
Bank Borrowers, May 2004, as modified, for consultants.
Ministry of Infrastructure and Transportation rules apply to
procurements below $55,000 for transportation projects.
Country: Honduras;
Type of procurement agent: Hybrid governmental and nongovernmental;
Procurement agent: An outside project manager for the transportation
project, and a government agency, the Fondo Nacional de Desarollo Rural
Sostenible, for the Rural Development Project.[B];
Procurement standards: World Bank Guidelines: Procurement Under
International Bank for Reconstruction and Development Loans and
International Development Association Credits, May 2004, as modified
for nonconsultant services, works, and goods.
World Bank Guidelines: Selection and Employment of Consultants by World
Bank Borrowers, May 2004, as modified, for consultants.
Source: GAO analysis of MCC procurement documents.
[A] Deutsche Gesellschaft für Technische Zusammenarbeit is a private
company owned by the German federal government.
[B] National Fund for Sustainable Rural Development.
[End of table]
Table 9: Procurement Provisions, Roles, and Responsibilities Stated in
MCC Agreements with Madagascar, Cape Verde, and Honduras:
Procurement-related provisions: Document;
Compact: Outlines MCC's approval rights and documents the
responsibility of compact country governments to ensure oversight of
procurement;
Disbursement agreement: Includes conditions precedent to disbursements
of funds from MCC to the compact country that MCC can use to enforce
the requirements in the procurement agreement and compact;
Procurement agreement: Lists specific rules and procedures that the
procurement agent will follow in managing procurement for the compact
country.
Procurement-related provisions:
Compact country roles and responsibilities; Compact:
* Provide oversight, ensure the removal or disclosure of any conflicts
of interest, and conduct audits on at least an annual basis;
* Ensure transparency by establishing a Web site for the program and
posting procurement documents online.[A];
* Create procurement review bodies in both Cape Verde and Honduras;
Disbursement agreement:
* Submit a report comparing actual procurement activity with
procurement plans for the second most recent disbursement period as a
condition precedent to disbursements;
* In Madagascar, provide a semiannual review report on compliance with
the procurement guidelines for MCC review;
* In Cape Verde, establish a procurement review commission and its
charter, prior to receiving MCC disbursements.[B];
* MCA-Honduras must create and approve a bid challenge system
acceptable to MCC prior to receiving funds for the second quarter of
operations;
Procurement agreement:
* Submit reports; and obtain MCC approvals before undertaking
procurements above thresholds;
* Report deviations from approved procurement plan to MCC. Cape Verde
and Madagascar to report deviations valued at more than $10,000.
Honduras to report all deviations;
* Provide a procurement plan semiannually.
Procurement-related provisions: MCC role and approval authority;
Compact:
* Approve the members of Cape Verde's procurement review commission and
Honduras's procurement supervisor agreement.[C];
* Authority to terminate the compact if the compact country government
or another party materially breaches the compact or any supplemental
agreement, such as a procurement agreement;
* Requires the signing of a procurement agreement prior to entry into
force;
Disbursement agreement:
* Review and approve periodic plans and reports;
Procurement agreement:
* Approve procurements above thresholds;
* Review and approve any deviations from approved plan above certain
thresholds;
* Review and approve the compact country's procurement plan, including
proposed procurements and methods of procurement.
Source: GAO analysis of MCC documents.
[A] In Cape Verde, MCC compact funding is also supporting the creation
of an electronic procurement system. First, MCC is funding the
establishment and implementation of a public e-procurement system for
use in compact procurements. Second, it is funding the expansion of
this system to all other units of the government.
[B] As conditions precedent to any disbursements for procurement
management, MCC has required that the government of Cape Verde will (1)
request, as a matter of national interest that requires urgent
resolution of the National Assembly, that the passage of a harmonized
public procurement law, with a supporting regulatory framework, be
placed on the legislative agenda and (2) establish a long-term program,
with support in the national budget, for the establishment of local
procurement training capacity (facilities and personnel), which will be
sustainable without donor funding within a period of 5 years.
[C] In Honduras, the compact also requires MCC approval of a
procurement operations manual.
[End of table]
[End of section]
Appendix V: Comments from the Millennium Challenge Corporation:
Millennium Challenge Corporation:
Reducing Poverty Through Growth:
July 7, 2006:
Mr. David B. Gootnick, Director:
International Affairs and Trade:
Government Accountability Office:
441 G Street NW:
Washington, DC 20548:
Dear Mr. Gootnick:
We appreciate the opportunity to review and comment on your draft
report, "Millennium Challenge Corporation Compact Implementation
Structures Are Being Established, Framework for Measuring Results Needs
Improvement."
The enclosed Millennium Challenge Corporation views are provided for
incorporation with this letter as an appendix to the final report.
Thank you again for your willingness to accommodate our concerns and
for the consistent professionalism of the GAO staff in their dealings
with MCC.
If you have any questions concerning this response, please contact Rick
Stilgenbauer, Program Officer, Department of Congressional and Public
Affairs, at (202) 521-3629.
Sincerely,
Signed by:
Frances C. McNaught:
Vice President:
Congressional and Public Affairs:
Millennium Challenge Corporation Comments on GAO Draft Report:
Millennium Challenge Corporation Compact Implementation Structures Are
Bein Established, Framework for Measuring Results Needs Improvement:
General:
The Millennium Challenge Corporation (MCC) would like to thank the
Government Accountability Office for their thorough work in producing
this audit. MCC appreciates the acknowledgement by GAO of the
considerable effort and progress that we have made to develop
structures, processes, and procedures at the same time that initial
Compacts were being developed with eligible countries. There does
however, appear to be some misunderstanding about the full complexity
of engaging eligible countries and developing and writing policies and
procedures for how we would engage countries at the same time, as
exemplified in statements such as: ".in assessing the proposals, MCC
used criteria contained in guidance issued after or shortly before the
countries submitted their proposals" or ".although its assessments used
criteria that it had not yet published in its guidance." MCC's mandate
to engage directly with eligible countries shortly after its creation
did not allow for the possibility of developing and vetting policies
and procedures in advance of such engagement. We expect this criticism
will not be valid beyond the early instances pointed out in this audit.
We also appreciate GAO's identification of several issues of concern to
the Corporation.
Economic Analyses:
MCC remains devoted to one of its founding principles: that economic
analyses reflect country conditions and involve country participation.
Indeed, this is one of the primary goals in developing Compacts.
Whenever feasible, any MCC economic analysis uses data and makes
assumptions based on information obtained from each country in order to
accurately reflect the country's socioeconomic conditions. For example,
in Madagascar, an important part of the analysis was based on a study
using agricultural data from rural areas in Madagascar. In Honduras,
the analysis was based on data that an agricultural consulting firm had
collected from local activities. We agree that in some cases the level
of country engagement on economic analysis could be improved. To
address this issue, MCC revised its economic guidelines and other
guidance materials to better prepare country staff to play a larger
role in the analysis. MCC now asks each eligible country to engage an
economist on its core team early in the proposal process. In addition,
MCC economists make early technical guidance trips to work with the
core team and the core team economist in the country. Finally, the new
economic guidelines provide specific examples of economic rate of
return ("ERR") analysis for development projects in order to reduce
confusion about the analysis required for investment decisions.
Baseline Data:
Obtaining more accurate and reliable baseline data is a priority for
MCC. For example, in the case of Madagascar, MCC completed three large
national surveys which were subsequently reviewed by the U.S. Census
Bureau. These surveys collected information from hundreds of questions,
most of which produced satisfactory data. Since baseline data
collection has not been completed in Cape Verde or Honduras, it is not
yet possible to assess data quality in those cases. (The decision to
undertake and structure baseline data surveys must be conditioned by
reasonable cost and likely value of additional precision.)
GAO's concern over unreliable baselines appears to stem mainly from the
poor data on land value collected through Madagascar's Agricultural
Productivity Survey. MCC and the Malagasy had hoped that a similar
question in the Household Survey would fill the gap but unfortunately,
this survey question has also proven to be unreliable; MCC and MCA-
Madagascar are now discussing other methods of estimating targets for
the high-level investment and income goal indicators. These will be
outlined in a revised M&E plan to be completed in July.
As a more general point on baseline data, MCC, and all other donors,
will continue to face data quality issues in the collection of baseline
data in developing countries. MCC will continue to apply careful
attention to baseline survey design, data collection, the review of
data quality (as exemplified by our requirement for frequent data
quality reviews), and data quality issues in a timely and transparent
fashion. MCC remains committed to the value of these efforts and agrees
with GAO in stressing their importance.
Linkage of M&E Plans to Economic Analyses:
MCC agrees that there should be clear linkage of Monitoring and
Evaluation plans to economic analyses. There has been notable
improvement in this linkage since the development of the first
Compacts. At the same time, MCC has a slight difference of opinion with
GAO in terms of the appropriate level of aggregation in linking the
two. MCC contends that not every estimated benefit from an ERR
spreadsheet can be isolated and measured during implementation. MCC
believes that the primary and most significant purpose of the M&E plan
is to develop metrics that capture overall project results. In some
cases, disaggregating each individual benefit stream from a project may
not be possible or may prove to be too costly to be a feasible option.
Policies and Procedures for Establishing Targets:
As a rule, targets on performance indicators are set in accordance with
the economic analysis justifying the program. At the same time, MCC
agrees that more defined policies and procedures for establishing and
modifying targets are needed. We will develop these further as
additional Compacts are implemented.
Timely Design of Impact Evaluations:
MCC appreciates the emphasis by GAO on impact evaluation and its timely
design. MCC remains committed to credible and transparent impact
evaluation which is another of our principles. In fact, M&E plans and
disbursement agreements require evaluation plans to be substantially
complete and baseline data gathered prior to disbursement on activities
that might skew results measurement. Compacts subsequent to those
discussed in this report have benefited from earlier development of
evaluation designs.
Randomized Controlled Trials:
MCC would like to clarify that randomized controlled trial (RCT) is one
of several methods that will be used to conduct rigorous impact
evaluations. While most experts agree that evaluations using random
assignment are best able to estimate and attribute the impact of
specific interventions, MCC anticipates that, given the requirements of
the design, RCT will be feasible and cost-effective for only a small
number of projects to be evaluated in this manner. In all cases,
metrics need to be developed before project or activity implementation
begins. For example, Madagascar's evaluation of agricultural business
centers cannot be designed until investment studies for the Compact
regions are completed. These studies should be finished by December
2006 with evaluators targeted to begin work with country staff in
September.
Summary:
Overall, MCC is satisfied with the draft report GAO has produced from
our nine-month audit and the constructive criticism within. However,
GAO provided comments on a number of matters, but did not provide
specific or tangible recommendations within these comments.
As always, MCC appreciates the efforts of the professionals at GAO and
their willingness to work with the MCC to help us improve processes
that will help carry out our mission of reducing poverty through
sustainable economic growth.
The following are GAO's comments on the Millennium Challenge
Corporation letter dated July 7, 2006.
GAO Comments:
1. MCC stated that "MCC's mandate to engage directly with eligible
countries shortly after its creation did not allow for the possibility
of developing and vetting policies and procedures in advance of such
engagement." We recognize that MCC was simultaneously addressing a
number of issues during its initial years of operation--selecting
eligible countries, setting up MCC as an organization, and developing
guidance and policies while also working with countries to finalize
compacts. In the context of MCC as an evolving organization, we felt it
was important to discuss the evolving nature of guidance to provide a
balanced perspective regarding the process that eligible countries had
to follow to sign compacts.
2. MCC stated that our concern regarding baseline data quality stems
from one survey in Madagascar. However, we found issues with data
reliability in Madagascar with both the Agricultural Productivity
Survey and Household Survey. In the second survey, the error was
smaller; therefore, we focused on the Agricultural Productivity Survey
to illustrate the more significant example.
The measurement of land values is very important for Madagascar's
monitoring and evaluation plan. Both the compact-level goal of
increasing household income and the program objective of increasing
investment in rural Madagascar are measured in terms of land values. In
each of the zones, the expected increase in household income is
estimated at 5 percent of the average land value, and the expected
increase in investment is estimated at 27 percent of the average land
value. With a large error in baseline data, it will be difficult to
accurately track progress toward the compact- level goal and program
objective.
3. MCC noted that not every benefit can be isolated and measured during
implementation, and that disaggregating may not be feasible. We
recognize that not all outcomes of the economic analysis can be
directly tracked with indicators at different levels of monitoring or
for impact evaluation. There are trade-offs between cost and level of
detail. However, as we note in this report, aggregation poses some
challenges that could limit the effectiveness of monitoring and
evaluation.
[End of section]
Appendix VI: Comments from the Department of State:
United States Department of State:
Assistant Secretary for Resource Management and Chief Financial
Officer:
Washington, D.C. 20520:
JUL 10 2006
Ms. Jacquelyn Williams-Bridgers:
Managing Director:
International Affairs and Trade:
Government Accountability Office:
441 G Street, N.W.
Washington, D.C. 20548-0001:
Dear Ms. Williams-Bridgers:
We appreciate the opportunity to review your draft report, "Millennium
Challenge: Compact Implementation Structures are Being Established,
Framework for Measuring Results Needs Improvement," GAO Job Code
320370.
The enclosed Department of State comments are provided for
incorporation with this letter as an appendix to the final report.
If you have any questions concerning this response, please contact
Lawrence Connell, Financial Economist, Bureau of Economic and Business
Affairs, at (202) 647-9462.
Sincerely,
Signed by:
Bradford R. Higgins:
cc: GAO - Michael Simon:
EB - Daniel Sullivan:
State/OIG - Mark Duda:
Department of State Comments on GAO Draft Report "Millennium Challenge
Corporation: Compact Implementation Structures are Being Established,
Framework for Measuring Results Needs Improvement" (GAO-06-805, GAO
Code 320370):
Overview:
Thank you for the opportunity to comment on GAO's draft report
"Millennium Challenge Corporation: Compact Implementation Structures
are Being Established, Framework for Measuring Results Needs
Improvement."
The GAO report is well researched, but has a tendency to make minor or
transitory problems sound like major criticisms. The report should note
the improvements that MCC made during and since the period it covers.
The report makes six specific criticisms of the MCC that reflect its
interviews, research, and country site visits. These criticisms are
presented and addressed below.
1) "MCC based its assessments on an evolving set of criteria: early,
general guidance to the countries followed by later, more specific
guidance. "
GAO criticizes MCC for assessing country proposals based on 2005
guidance that was updated from the 2004 guidance countries used when
drafting proposals. GAO claims that MCC's initial guidance did not
inform countries that their proposals should be linked to compact
goals, but the notion that countries would be wholly unaware of the
need to design their compact programs to specifically target compact
goals in order to receive compact funding is erroneous. It is to be
expected that MCC would update formal guidance as it grew, evolved, and
built institutional knowledge. Countries requesting informal guidance
from MCC could have received it at any time, and the notion that
compact funding would be linked to compact goals was clearly implied.
These mitigating factors should be acknowledged in the report.
2) "MCC's analyses of projects 'economic impact were limited in that
some of the assumptions and data used may not reflect country
conditions, and as a result, the projects selected on the basis of the
analyses may not be the most likely to achieve compact goals. "
MCC is criticized for making assumptions on the Economic Rate of Return
(ERR) of projects which were not substantiated by GAO's in-country
"focus groups." GAO fails to detail, however, the (1) methodology used
by these groups, (2) group size, (3) gender, age, or racial status of
participants, or (4) whether participants were urban or rural, etc. It
is unrealistic to assume that GAO's focus group study should exactly
reproduce the results of MCC country team's economic analysis.
MCC is criticized for analyzing proposals from Madagascar and Honduras
before publishing formal economic analysis guidelines. One could argue
that this was nonetheless the right decision, given the need for MCC to
demonstrate progress by approving/signing compacts 15-18 months into
its existence, rather than holding them up for finalization of such
guidelines. The compacts themselves have not been criticized for their
economic content.
3) "MCC conducted the analyses with limited country participation, with
the result that the countries had little understanding of the process."
GAO reports that representatives in the three compact countries
analyzed, Cape Verde, Honduras, and Madagascar, reported receiving
little assistance from MCC officials in developing the economic models
and data used for analysis. GAO mentions that MCC officials told them
that "countries' degree of involvement depended on the capability and
willingness of the countries' proposal development team to actively
participate in the analysis[Footnote 74]." GAO does not rebut this
point, which seems to explain fairly the limits on compact country
participation.
4) "Oversight structures for country management with MCC review. were
not fully staffed for months after the compacts entered into force. "
GAO criticizes MCC for the amount of time it took to hire in-country
staff--two to six months to fill critical positions. This timeline,
however, compares favorably to general government hiring. GAO
acknowledges that in the case of Honduras, two jobs had to be re-
competed because there were initially no suitable candidates. GAO
offers MCC no recommended alternative to its taking the time to find
good, or at least fully qualified, candidates for positions of trust.
5) "Fiscal accountability structures for MCC funded projects, and
established procurement structures with effective characteristics .are
still largely untested and some are still under development. "
GAO praises MCC for making "progress in defining the mechanisms and
processes of internal control that provide reasonable assurance of
fiscal accountability at the compact level[Footnote 75]." GAO notes
that "each country's compact requires the accountable entity to develop
its own fiscal accountability plan,[Footnote 76]" but criticizes MCC
because the various compact agreements "differ in the maturity of their
internal control"[Footnote 77] systems. This is to be expected, given
that some countries have more historically well-developed systems from
which to draw on; GAO is not clear as to which if any of MCC's choices
it would have made differently. GAO notes that MCC will be conducting
at least yearly audits, and consequently the processes and procedures
in place will ultimately become even better developed through
additional time and experience.
6) "Limitations in the baseline data collected, linkage to economic
analyses, addressing uncertainty associated with program results, and
the timely design of randomized controlled trials may constrain MCC's
ability to monitor and evaluate programs.
The Department concurs with GAO that baseline data is vital for
measuring results, and that a lack of quantifiable baseline data could
be problematic for assessing project results. However, we believe that
GAO's reliance on project evaluation criteria that make use of
randomized controlled trials to measure success, while appropriate in
judging scientific experiments, is not likely feasible for most
infrastructure and social development projects. Use of cost-benefit
analysis that measures projected and actual outcomes with and without
the projects is a better model, and could be usefully adopted by MCC if
it has not done so already.
The Department notes that in the footnotes on Page 8, "country credit
rating" is listed as one of the Encouraging Economic Freedom
indicators, when the report should list this indicator as "the cost of
starting a business," to reflect the change made for FY 2006. We would
also like to suggest that on Page 1, the sentence "MCC expects to raise
incomes and lift thousands out of poverty in countries receiving MCC
assistance," be changed to "MCC expects to permanently raise incomes
and lift tens of thousands out of poverty in countries receiving MCC
assistance," to underscore MCC's transformational goal.
The following are GAO's comments on the Department of State letter
dated July 10, 2006.
GAO Comments:
1. In referring to our discussion about MCC's guidance on project
coherence, State Department commented that it would be erroneous to
indicate that countries were wholly unaware of the need to design their
compact programs to target compact goals, and that informal guidance
from MCC could have been received at any time. We agree that countries
could have requested informal guidance from MCC. According to MCC, they
did provide verbal guidance to countries regarding the need to link
projects to compact goals.
However, MCC rejected tourism and preschool education projects proposed
by Honduras because they were not linked to the impediments to growth
that emerged from the consultative process. MCC also rejected projects
in Cape Verde because MCC's due diligence did not indicate that these
projects addressed key constraints in Cape Verde.Given that these two
countries submitted projects that did not meet project coherence
criteria, MCC's verbal guidance may not have been sufficient to direct
countries to submit proposals meeting these criteria.
We recognize that MCC was simultaneously addressing a number of issues
during its initial years of operation--selecting eligible countries,
setting up MCC as an organization, and developing guidance and policies
while also working with countries to finalize compacts. In the context
of MCC as an evolving organization, we felt it was important to discuss
the evolving nature of guidance to provide a balanced perspective
regarding the process that eligible countries had to follow to sign
compacts.
2. State noted that we did not detail the methodology and composition
of our Madagascar focus groups. We summarized the composition of the
focus groups in the letter of this report. We met with these 29
individuals, including representatives of MCA-Madagascar and Madagascar
ministries, during the course of our week in Madagascar and reviewed
with them the assumptions used in MCC's economic model for its
projects. As previously discussed in this report, we found that these
officials were not closely involved in MCC's economic analysis.
According to MCC and Madagascar officials, an MCC economist met with
four Madagascar representatives in Paris, France, during compact
negotiations and reviewed the model with them at that time. (MCC
officials did travel to Madagascar during due diligence to assess
proposed projects.) When we discussed this with MCC officials, they
stated that not traveling to Madagascar to develop the model was a
mistake, and one that has not been repeated.
3. State commented that we do not rebut MCC's point that countries'
degree of involvement depended on their capability and willingness to
participate. (See comment 2.)
In its letter, which is reprinted in appendix V, MCC agreed that in
some cases, the level of country engagement on economic analysis could
be improved. MCC also now requires that countries engage an economist
on the core country team, provides examples of the economic rate of
return analysis, and has MCC economists make early technical guidance
trips to work with the core country team and the core country team
economist. These procedures were not in place when MCC conducted the
economic analysis for Madagascar, Cape Verde, or Honduras.
By not thoroughly involving the countries in this analysis, MCC risks
not meeting its stated principle of focusing on results. In MCC's
results-based framework, the economic model is the basis for
determining performance targets against which funding is conditioned.
MCA-Madagascar may find that the targets are unrealistic if the targets
have not been thoroughly discussed with, and understood by, Madagascar
officials.
4. State noted that the time for the in-country MCA organizations to
hire staff compares favorably with general government hiring.
Management unit officials are hired by a compact country's MCA program
steering committee, not the U.S. government. As such, it is not
"general government hiring" and should not be compared with it.
5. State correctly observed that we did not offer MCC a recommended
alternative to its taking the time to find good candidates for
positions. We did not make a recommendation for addressing this issue
because MCC has adopted a policy implementing the authority given it by
section 609(g) of the Millennium Challenge Act of 2003 to make grants
to facilitate compact development and implementation. MCC's current
policy includes a provision where, if certain conditions are met, it
may fund an eligible country's request for "management support
payments" for salaries, rent, and equipment for the country's team
prior to compact signature. We have added a discussion of this policy
in the letter of this report to clarify this point.
6. In referring to our discussion about the different maturity of
internal control between Madagascar and Cape Verde, State pointed out
that differences in the maturity of internal control between the
countries are to be expected, given that some countries have
historically more well-developed systems. We agree that different
countries will differ in their maturity of internal control due to
historical differences in the underlying systems and accountability
infrastructures. The current maturity of internal control in each
country, however, is a key consideration in assessing risk and
establishing effective oversight mechanisms to deal with the unique
risks in the key financial processes and activities of each country.
7. State referred to "GAO's reliance on project evaluation criteria
that make use of randomized controlled trials to measure success." This
comment misconstrues our findings on the use of randomized controlled
trials. We do not rely on these criteria. We are commenting on MCC's
use of randomized controlled trials, which MCC has indicated are its
preferred method of impact evaluation, and on the potential challenges
to the use of this approach. State's comments regarding cost-benefit
analysis would be more appropriately addressed to MCC.
8. State correctly noted that we did not capture the change to one
indicator for fiscal year 2006. We have updated the list of MCC
indicators in footnote 8.
9. State suggested rewording MCC's expected results to "MCC expects to
permanently raise incomes.…" We did not use the word "permanently"
because in the three compacts that we focused on for this report, it
was not used to describe MCC's expected results at the country level.
[End of section]
Appendix VII: GAO Contacts and Staff Acknowledgments:
GAO Contact:
David B. Gootnick, Director, 202-512-3149:
Staff Acknowledgments:
In addition to the person named above, Phil Herr (Assistant Director),
Claude Adrien, Cara Bauer, Gergana Danailova-Trainor, Jeanette Franzel,
Keith Kronin, Reid Lowe, Norma Samuel, Mona Sehgal, and Michael Simon
made key contributions to this report. Also, Tim DiNapoli, David
Dornisch, Etana Finkler, Ernie Jackson, Debra Johnson, Bruce Kutnick,
Janice Latimer, Charlotte Moore, and Celia Thomas provided technical
assistance.
(320370):
FOOTNOTES
[1] Millennium Challenge Act of 2003, Public Law 108-199, Division D,
Title VI of the Consolidated Appropriations Act, 2004. Title II,
Division D of this act established MCA for MCC appropriations.
[2] About $400 million has been set aside for MCC's administrative
expenses, due diligence, monitoring and evaluation, threshold country
program--which provides funding to candidate countries to help them
become eligible for MCA assistance, and other costs. An MCC compact is
an agreement between the U.S. government, acting through MCC, and the
government of a country eligible for MCC assistance. Compacts have a
maximum duration of 5 years.
[3] GAO, Millennium Challenge Corporation: Progress Made on Key
Challenges in First Year of Operations, GAO-05-455T (Washington, D.C.:
Apr. 26, 2005) and GAO-05-625T (Washington, D.C.: Apr. 27, 2005).
[4] A compact proposal outlines, among other things, program
objectives, related projects, project funding, and a program
implementation framework.
[5] Between April 2005 and May 2006, MCC signed compacts with Honduras,
Cape Verde, Nicaragua, Georgia, Benin, Vanuatu, and Armenia.
[6] MCC's compacts with Georgia, Vanuatu, and Nicaragua entered into
force, respectively, on April 7, May 11, and May 26, 2006. However,
because these compacts entered into force late in our review, we did
not focus our analysis on these countries.
[7] MCC, 2005 Annual Report.
[8] These indicators are as follows: Ruling Justly: (1) political
rights, (2) civil liberties, (3) voice and accountability, (4)
government effectiveness, (5) rule of law, and (6) control of
corruption. Encouraging Economic Freedom: (1) cost of starting a
business, (2) 1-year consumer price inflation, (3) fiscal policy, (4)
trade policy, (5) regulatory quality, and (6) days to start a business.
Investing in People: (1) public expenditures on health as a percentage
of the gross domestic product (GDP), (2) immunization rates for DPT3
and measles, (3) public primary education spending as a percentage of
the GDP, and (4) girls' primary education completion rate. See MCC,
Report on Criteria and Methodology for Determining the Eligibility of
Candidate Countries for Millennium Challenge Account Assistance in FY
06 (Sept. 8, 2005).
[9] The MCC Board suspended Gambia's eligibility for assistance on June
16, 2006, citing a pattern of actions inconsistent with MCC's selection
criteria.
[10] The annual per capita income threshold for MCA assistance was
$1,415 in fiscal year 2004 and $1,465 in 2005. In fiscal year 2006, the
threshold was $1,575 for low-income countries and $1,575 to $3,255 for
lower-middle-income countries. In fiscal year 2006, MCC selected Cape
Verde, El Salvador, and Namibia from its list of lower-middle-income
countries.
[11] MCC refers to the accountable entity by combining "MCA" and the
country's name--for example, "MCA-Madagascar."
[12] In addition, MCC and the compact country must certify the
completion of these requirements for entry into force and provide
statements of the incumbency of their representatives with specimen
signatures.
[13] "Conditions precedent" are specific steps that must be completed
by the compact country prior to MCC's providing a disbursement.
[14] In January 2006, MCC issued new guidance to better prepare
countries at the start of the proposal development process by providing
estimates of personnel and funds required for proposal development.
According to MCC officials, MCC is engaging with countries earlier in
the compact development process to ensure that countries develop better
proposals.
[15] In addition to assessing aspects related to project selection, MCC
also assessed program oversight and management, fiscal accountability,
procurement, and monitoring and evaluation structures to ensure the
proper oversight and use of MCC funding. These structures are discussed
later in this report in the context of compact implementation.
[16] For example, Madagascar and Honduras participated in a World Bank-
initiated poverty reduction strategy initiative that required countries
to involve stakeholders in identifying national development priorities
through a consultative process.
[17] We did not conduct in-depth interviews into the nature of these
representatives' participation in the consultative process. Also, due
to the limited number of people interviewed, this result is not
representative of the entire set of consultative process participants
in these countries.
[18] MCC also had various concerns about these projects, including who
would likely benefit from them. MCC considered these projects
appropriate for private sector funding. MCC also stated that some
Honduran government officials had conflicts of interest related to
these projects. For example, the then President of Honduras was the
founder of a preschool foundation, and a cabinet minister had financial
interests in the Tela Bay project.
[19] Additionally, according to MCC officials, MCC chose not to work on
electricity in Cape Verde because of problems stemming from the
privatization effort that had taken place in the electricity sector.
MCC had asked the Cape Verde government to address these problems
related to the privatization.
[20] MCC's April 2005 Guidelines for Economic Analysis states MCC's
intention to "assess a project's logic and potential impact on economic
growth and poverty reduction [and] to test claims of the necessity and
sufficiency of project components in achieving the overall objective
and to examine critical linkages and dependencies." However, this
guidance was issued after MCC finished due diligence for Madagascar and
Honduras and 1 month before it completed due diligence for Cape Verde.
[21] In its interim Environmental Guidelines, issued in March 2005, MCC
defines "environmental impact" as the effects of a project on the
surrounding natural environment and on the humans reliant on that
environment, including the effects on cultural property, indigenous
peoples, and involuntary resettlement, as well as impact on human
health and safety. In its January 2006 Environmental Guidelines, MCC
provides the same definition for "environmental and social impact." MCC
added that environmental impact may also include significant induced,
indirect, and cumulative impact and reasonably foreseeable effects that
may be associated with, or ancillary to the project.
[22] MCC's 2006 Environmental Guidelines assign proposed projects to
category D if they involve an intermediate facility, such as a grant
fund, that will be used later to finance activities that may have
adverse environmental and social impact.
[23] The final Environmental Guidelines were issued in January 2006.
[24] MCC regards the promotion of economic growth as being closely
linked to the reduction of poverty, its other key objective. MCC cites,
as one source that investigates the two objectives' relation, Pro-Poor
Growth in the 1990s: Lessons and Insights from 14 Countries, a study
coauthored by Agence Française de Développement, Bundesministerium für
Wirtschaftliche Zusammenarbeit und Enwicklung, U.K. Department for
International Development and the World Bank (available at http://
siteresources.worldbank.org/INTPGI/Resources/342674-1119450037681/Pro-
poor_growth_in_the_1990s.pdf).
[25] The following is a simplified example of how an ERR is calculated:
If MCC spent $100,000 on a project in year 1 and expected that the
project would yield net benefits of $120,000 in year 2, the project's
ERR for year 1 would be (120,000-100,000)/100,000= 0.2, or 20 percent.
Technically, the ERR of a project is the discount rate (interest rate)
at which the present value of the cost stream is equal to the present
value of the benefit stream. If ERR computations are not appropriate
for a project--such as policy reform at the national level--MCC
guidelines describe other methods that can be used to determine project
impact. For example, impact on growth can be estimated with statistical
analysis using data from other countries or with simulations.
[26] In the case of Madagascar, MCC performed an economic analysis for
two out of the three projects, but calculated only a compact-level ERR.
[27] Net benefits include not only net income but also imputed
benefits, such as spillovers.
[28] Economic analyses of similar projects in two countries may have
both common and country-specific elements. For instance, the analyses
of transportation projects in Cape Verde and Honduras share some common
elements--reduction in vehicle operating costs and increase in annual
daily traffic. However, because the Cape Verde project was also aimed
at alleviating bottlenecks caused by the absence of all-weather
crossings, income loss from the transportation blockage, which would be
eliminated with the new bridges, was included in a broader measure of
net benefits.
[29] According to MCC officials, the ERR for all activities that were
assessed during due diligence was above the threshold set in the
guidelines.
[30] In Madagascar, where prevailing land tenure practices do not
provide the titles needed for mortgage transactions, 80 percent of the
households own 20 percent of rural cultivated land, and 48 percent of
the population is illiterate. In 1999, the poverty rate for individuals
owning 0.001 to 0.09 hectares of land was 91.7 percent. The richest
families own 3.7 times more agricultural rural land than the poorer
families. Furthermore, interest rates are high; for example, in
December 2004, the Central Bank charged a base rate of 14 percent to 18
percent.
[31] Several studies suggest that, given a real threat of foreclosure
and land loss and an uncertain environment without insurance markets,
the impact of land titles on individual investment incentives and
productivity is likely to be greater for the wealthy farmers whose
landholdings and other assets will enable them to benefit from capital
markets. In addition, customary land tenure most often provides
security of tenure. However, major threats to such security often come
from outside investors seeking land with the backing of the state.
Thus, land titling, if misused, may become a tool for land grabbing by
official elites. See Jolyne Melmed-Sanjak and Susana Lastarria-
Cornhiel, "Land access, off-farm income and capital access in relation
to the reduction of rural poverty," Land Reform, Volume 1 (1998): 4-18
(available at http://www.fao.org/sd/LTdirect/LTan0023.htm); M. Carter,
K. Weibe, and B. Blarel, "Tenure security for whom? Differential
impacts of land policy in Kenya," Research Paper No. 106 (Madison, WI:
University of Wisconsin (1991)); and John W. Bruce and Shem Migot-
Adholla, eds. Searching for Land Tenure Security in Africa (Dubuque,
IA: Kendal/Hunt Publishing Company (1994)).
[32] Several papers written by members of the development community
have identified limitations to capacity development resulting from the
use of project implementation units (PIU), which are similar to MCC's
management units. While PIUs seem to benefit from short-term efficiency
in service delivery, the model does not fare well when it comes to
building long-term capacity in developing countries. Critics view PIUs
as parallel and insular structures that do not promote skills transfers
to host governments. The skills and abilities that staff develop on one
PIU-managed project tend to be lost to the host government when the
staff accept positions with the next donor-funded project. See John
Lawrence, Sarah Renner, and Jan Vandemoortele, The PIU Dilemma: How to
Address Project Implementation Units, UN Development Program (September
2003); UN Development Program, Background Note on Preparation of
Guidelines on Strengthening Country Capacity for Financing,
Implementing, and Managing Development Programs (Feb. 7, 2005); and
World Bank, Guidance Note for Project Management: Strengthening
Institutional Capacity during Project Implementation (October 2005).
[33] To date, MCC has objected only once to a country's choice, during
the hiring of an environmental and social assessment manager for one of
the countries.
[34] As defined for MCC programs, "fiscal accountability" is the
assurance that funds are managed properly and procurements are
undertaken in a fair, open, and transparent manner. "Internal control"
is an integral component of an organization's management that provides
reasonable assurance that the following objectives are being achieved:
effectiveness and efficiency of operations, reliability of financial
reporting, and compliance with laws and regulations. See GAO, Standards
for Internal Control in the Federal Government, GAO/AIMD-00-21.3.1
(Washington, D.C.: November 1999).
[35] We did not evaluate Honduras's implementation of fiscal
accountability.
[36] MCC, MCC Fiscal Accountability Guidelines (June 14, 2005).
[37] MCC Fiscal Accountability Guidelines.
[38] The internal control maturity continuum developed by the
independent risk consulting company, Protiviti, Inc., provides a scale
for evaluating the sufficiency of an entity's internal controls. The
scale describes the attributes of controls at five levels: initial,
repeatable, defined, managed, and optimizing. See Guide to the Sarbanes-
Oxley Act: Internal Control Reporting Requirements, 3RD ed. (2004): 84
(www.protiviti.com).
[39] Key duties and responsibilities of financial operations need to be
divided, or segregated, among different members of the staff to reduce
the risk of error or fraud. This should include separating the
responsibilities for authorizing transactions, processing and recording
them, reviewing the transactions, and handling any related aspects. No
one individual should control all key aspects of a transaction or
event. See GAO/AIMD-00-21.3.1.
[40] USAID, Office of Inspector General for the Millennium Challenge
Corporation, Risk Assessment of Millennium Challenge Corporation's MCA
Madagascar Financial Operations, Audit Report No. M-000-06-003-F
(Washington, D.C.: Mar. 28, 2006).
[41] MCC IG, Guidelines for Financial Audits Contracted by MCA (revised
January 2006); "Standard Statement of Work for Financial Audits of
Accountable Entities" (revised January 2006); and "Standard Statement
of Work for Financial Audits of Covered Providers" (revised January
2006).
[42] Within the MCC fiscal accountability framework, the procurement
agent is responsible for impartially administering and/or certifying a
process for procurement, up to the point of selection, that adheres to
a defined set of procurement standards.
[43] Procurement standards define technical language, provide
instructions to participating vendors, and communicate the terms and
conditions for vendor selection and process performance.
[44] The Millennium Challenge Act of 2003 requires MCC to ensure that
open, fair, and competitive procedures are used in a transparent manner
in the procurement of goods and services under the compact.
[45] According to MCC officials, Madagascar's procurement law did not
identify review thresholds. They stated that, because of the lack of
thresholds in Malagasy law, MCC developed them in consultation with
Madagascar representatives prior to compact signing. Although MCC
officials stated that the thresholds were based on the best available
knowledge of local practices, the thresholds had not yet been tested in
practice prior to their adoption by MCC.
[46] These characteristics include integrity, openness, accountability,
professional workforce, competition, and value. See GAO, United
Nations: Progress of Procurement Reforms, GAO-NSIAD-99-71 (Washington,
D.C.: Apr. 15, 1999); and United Nations: Preliminary Observations on
Internal Oversight and Procurement Practices, GAO-06- 226T (Washington,
D.C.: Oct. 31, 2005).
[47] MCC officials also stated that they have entered into blanket
purchase agreements with other firms to supplement their procurement
staff.
[48] The MCC IG identified additional weaknesses in management
practices in its Risk Assessment of Millennium Challenge Corporation's
MCA-Madagascar Financial Operations (Washington, D.C.: Mar. 28, 2006).
The MCC IG found that MCA-Madagascar had not implemented procedures for
validating the receipt of goods and services, recording goods in an
inventory system, and tracking time and attendance. Furthermore, the
MCC IG found that MCA-Madagascar was in the process of purchasing
vehicles for MCA employees' work-related and personal use. MCA-
Madagascar canceled the procurement after this finding. MCC noted in
its response to the audit that the provision of vehicles for personal
use was an accepted practice in many parts of the world; but MCC
adopted a policy reinforcing the compact provision against the personal
use of MCC assets. On the basis of its findings, the MCC IG recommended
that MCC establish and distribute a set of policies and procedures for
managing assets purchased with MCC funds. MCC agreed with this
recommendation.
[49] These actions include working with the Ministry of Finance to
provide adequate permanent facilities, changing the review process to
reduce the number of actions required of the commission, developing a
training plan for the members of the commission, and receiving
additional technical assistance from MCC.
[50] According to MCC officials, MCA-Cape Verde subsequently hired a
full-time translator to serve the procurement review commission and MCA-
Cape Verde.
[51] According to MCC's January 2006 Guidelines for Monitoring and
Evaluation Plans, each plan is to include separate components for
monitoring and evaluation as well as summarize the compact program and
objectives, identify program beneficiaries, document assumptions and
risks, identify country staff responsibilities for managing and
implementing the plan, and provide a detailed annual monitoring and
evaluation budget. Compact Disbursement Agreements also condition MCC's
periodic disbursement of compact funds on a country's demonstrating
that it is meeting the performance requirements outlined in its plan.
Prior to January 2006, MCC provided very limited guidance to eligible
countries for developing monitoring and evaluation plans. For instance,
its Guidance for Developing Proposals for MCA Assistance in FY 2004
described the need for measurable goals and accountability for results.
In addition, an attachment to the 2004 guidance required that a
country's proposal describe a plan, including progress indicators and
baseline data, for monitoring progress toward compact goals. This
guidance, however, provided few specifications.
[52] The Millennium Challenge Act of 2003 requires MCC to include
regular benchmarks in its compacts to measure progress toward achieving
compact objectives.
[53] The Madagascar, Cape Verde, and Honduras compacts show that each
country will use $3 million to $5 million in MCC funding for these
monitoring and evaluation efforts.
[54] In January 2005, the Central Bank of Madagascar introduced
monetary reform and a new currency with one-fifth the value of the
prior currency. When the Agricultural Productivity Survey was conducted
in April 2005, survey enumerators and respondents used both currencies
when discussing land prices. However, in conducting the survey,
enumerators did not clearly identify the currency in which they
recorded land values. Madagascar's Statistical Institute, which
conducted the survey, found that 30 percent of the surveys contained
errors.
[55] Household income is expected to increase by 5 percent of the
average land values in targeted project zones. In addition, the impact
of a land tenure project will be measured by the amount of total new
investment in the zones, with a target of increasing the average land
value by an estimated 27 percent by the end of the 4-year compact.
[56] MCC officials stated that they were aware of the survey problem,
and that data from a household survey could be useful as an additional
measure of land values.
[57] MCC also reduced five interim indicators for the Watershed
Management and Agriculture Support Project because of a delay in
implementing the project.
[58] GAO, Foreign Assistance: USAID's Reengineering at Overseas
Missions, GAO/NSIAD-97-194 (Washington, D.C.: Sept. 12, 1997).
[59] The approved monitoring and evaluation plans for Madagascar and
Cape Verde acknowledge that external factors, many of them beyond the
control of program management, may directly affect progress toward
project objectives. The countries' plans and the draft monitoring and
evaluation plan for Honduras each include a discussion that elaborates
on the assumptions in the economic analyses and the uncertainties
associated with those assumptions. Although MCC performs a limited
sensitivity analysis that analyzes the response of the variables used
in the economic model to changes in some of the external factors that
could affect it, MCC does not perform this evaluation systematically
for all countries and projects.
[60] As defined by the Office of Management and Budget (OMB), a
randomized controlled trial is "a study that measures an intervention's
effect by randomly assigning individuals or other units into an
intervention group, which receives the intervention, and into a control
group, which does not. At some point following intervention,
measurements are taken to establish the difference between the
intervention group and the control group." However, randomized
controlled trials are not suitable for every program and generally can
be employed only under special circumstances. See OMB, "What
Constitutes Strong Evidence of a Program's Effectiveness?"
(http://www.whitehouse.gov/omb/part/2004_program_eval.pdf).
[61] According to OMB, randomized controlled trials provide the highest-
quality unbiased evaluation to demonstrate actual program impact.
[62] According to MCC officials, at that point, MCC also plans to
invite other groups and individuals to compete for conducting
randomized trials.
[63] As of May 2006, MCC had not awarded task orders for this work.
[64] As we noted in our April 2005 testimonies, strategic and
performance planning and reporting processes that establish, measure,
and report an organization's progress in fulfilling its mission and
meeting its goals are important for organizational accountability. See
GAO, Millennium Challenge Corporation: Progress Made on Key Challenges
in First Year of Operations, GAO-05-455T (Washington, D.C.: Apr. 26,
2005) and GAO-05-625T (Washington, D.C.: Apr. 27, 2005).
[65] Internal controls provide reasonable assurance that key management
objectives--that is, efficiency and effectiveness of operations,
reliability of financial reporting, and compliance with applicable laws
and regulations--are being achieved.
[66] GAO-05-455T and GAO-05-625T.
[67] GPRA, Public Law 103-62; FMFIA, Public Law 97-255; and FISMA,
Public Law 107-347.
[68] SAS 70, "Reports on the Processing of Transactions by Service
Organizations," covers situations in which an organization has
outsourced its financial operations. It is intended to provide a
reasonable assurance that a service provider has adequate internal
controls.
[69] In April 2005, we reported that MCC had 107 employees in place
toward a target of no more than 200 employees at the end of December
2005.
[70] Several development experts have stated that MCC's proposed
staffing level (300) is very lean for an organization planning to
disburse $2 billion or more per year.
[71] Corporate governance can be viewed as the formation and execution
of collective policies and oversight mechanisms to establish and
maintain a sustainable and accountable organization, while achieving
its mission and demonstrating stewardship over its resources.
[72] Four of the nine board positions established in the Millennium
Challenge Act of 2003 are to be filled based on nominations from the
House and Senate majority and minority leaders. As of May 2005, two of
these positions had not yet been filled.
[73] GAO, Standards for Internal Control in the Federal Government,
AIMD-00-21.3.1 (Washington, D.C.: November 1999).
[74] Page 30.
[75] Page 37.
[76] Page 38.
[77] Page 39.
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548: