Designing Multi-Country Evaluations: Approaches, Challenges and Opportunities

Session Number: 2090
Track: International and Cross Cultural Evaluation
Session Type: Panel
Tags: cluster, multi-site and multi-level eval, multi-country evaluation, regional evaluation
Session Chair: Jennifer Gauck [Senior M&E Specialist - Social Solutions International, Inc.]
Discussant: Andrew Green [Technical Director - MSI]
Presenter 1: Elizabeth Ruedy [Director, Office of Monitoring & Evaluation - International Republican Institute]
Presenter 2: Jennifer Gauck [Senior M&E Specialist - Social Solutions International, Inc.]
Presenter 3: Krystin Krause [Assistant Professor - Emory & Henry College]
Time: Oct 28, 2016 (08:00 AM - 09:30 AM)
Room: L508

Abstract 1 Title: One Evaluation in Seven Parts: IRI’s Experience Evaluating a Multi-Country International Exchange Program
Presentation Abstract 1:

In addition to in-country programs, the International Republican Institute (IRI) implements global programs, such as exchanges, to help democratic actors share ideas and experiences in democratization and good governance. A particular challenge to evaluating global exchange programs is that while program context is narrowly defined - a short-term intervention targeting a small group of direct beneficiaries – the context in which results occur is far more complex, involving networks of indirect beneficiaries in multiple countries and emerging over the course of years. IRI recently completed a retrospective evaluation of a parliamentary exchange program that captured outcomes across seven countries, with some exchanges dating back nearly a decade. This presentation will focus on the design challenges for this evaluation, and how techniques such as outcome mapping allowed for a flexible framework for data collection, analysis and synthesis across countries.

Abstract 2 Title: They're Out Catching Bad Guys: Lessons from a Regional Evaluation of the CBSI Program
Presentation Abstract 2:

In an effort to promote citizen security and safety in the region, the State Department's Bureau of International Narcotics and Law Enforcement Affairs (INL) provides capacity-building assistance to law enforcement institutions in the Caribbean through the Caribbean Basin Security Initiative (CBSI). In addition to partnering with multiple law enforcement institutions in each of the 13 countries it covers, the CBSI also aids regional security bodies and organizations. This multi-level, multi-country focus posed several challenges for DevTech, which in 2015 completed a two-year, mixed-methods performance evaluation of the CBSI program. This presentation will discuss how DevTech designed the evaluation, collected and analyzed data, and organized the evaluation report in a way that wove together project-, country-, and regional-level findings and conclusions.

Abstract 3 Title: But Will That Work in Honduras?: Multi-Country Survey Design in Central America
Presentation Abstract 3:

The multi-country assistance provided by the State Department's Bureau of International Narcotics and Law Enforcement Affairs (INL) for security sector capacity-building assistance in Central America through the Central American Regional Security Initiative (CARSI) offers several challenges for evaluators. Given the broad scope of INL assistance across the region, as well as the limited availability of quantitative outcome data, evaluators used a mixed-methods design to gather data on output and outcome results, and to identify lessons learned and best practices. This paper discusses the evaluators’ efforts to overcome obstacles restricting their ability to statistically assess changes in knowledge, attitudes, and behavior among partner government participants in CARSI activities. In particular, it outlines the design and implementation of a survey instrument intended to quantify outputs and outcomes within a broad range of institutions and across the diverse set of countries that are members of CARSI.

Audience Level: All Audiences

Session Abstract: 

Evaluations of global or regional programs that are implemented in multiple countries pose an additional set of challenges beyond those encompassed in single-country evaluations. This session will discuss the approaches the panelists used in designing three multi-country evaluations that addressed these challenges. The presentations include a retrospective evaluation of a parliamentary exchange program implemented in seven countries by the International Republication Institute; a performance evaluation of a U.S. government-funded law enforcement reform program that covers 13 countries as well as regional security bodies; and a mixed methods evaluation of the State Department's Central American Regional Security Initiative (CARSI), which provides citizen security and counternarcotics assistance to seven countries with significantly different profiles, including Belize and Honduras. The panelists will draw together the lessons learned from these evaluations and suggest opportunities for future evaluation designs that cover multi-country programs.