Lessons learned from designing and executing ex-post evaluations of international development programs

Session Number: 3262
Track: Mixed Methods Evaluation
Session Type: Panel
Tags: cross-sectoral, evaluation use, methods
Session Chair: Annette Fay [M&E Specialist - Water CKM Project, Social Impact]
Presenter 1: Jean-Camille Kollmorgen [Deputy Director, Performance Evaluation - Social Impact, Inc.]
Presenter 2: Juan Carlos Guzman [University of Notre Dame]
Presenter 3: Beatrice Rogers [Friedman School at Tufts University]
Presenter 4: Justin Fugle [Senior Advisor for Policy and Program Outreach - Plan International USA]
Presenter 5: Annette Fay [M&E Specialist - Water CKM Project, Social Impact]
Time: Nov 10, 2017 (08:00 AM - 09:30 AM)
Room: Taylor

Abstract 1 Title: Lessons learned from an Ex-post Evaluation of Sapan 'Bridge' Program in Thailand
Presentation Abstract 1:

Social Impact conducted an ex-post evaluation of the USAID funded Sapan ("Bridge") program in Thailand. The program aimed to strengthen engagement between civil society and the Thai government, mainly through capacity building activities for local CSOs, media outlets, and women led organizations. This ex-post evaluation, conducted 1-year after the program ended, looks at the drivers for sustaining organizational and technical capacity of civil society actors in a politically volatile environment.

Abstract 2 Title: Lessons from an Ex-Post Evaluation of a Maternity and Child Health Care Program in Indonesia
Presentation Abstract 2:

This paper presents the results of an evaluation of the Child Health Opportunities Integrated with Community Empowerment program, designed to improve children’s health and nutrition and implemented in 30 Indonesian villages between 2003 and 2007. We collected quantitative data from mother-infant pairs in the villages and compared these with data collected seven years earlier when the program ended. For further quantitative evaluation, we collected 2014 data from villages in the same administrative region that did not receive the intervention. We also conducted focus groups and in-depth interviews with community members. We conclude with recommendations that a rigorous evaluation design, if included in the planning and implementation of similar projects, would facilitate and enhance the value of PSS—and therefore of the projects themselves—in the future.

Abstract 3 Title: A Synthesis of Results from a Four-Country Study of Sustainability and Exit Strategies among Development Food Assistance Projects
Presentation Abstract 3:

To be effective, development projects must result in changes that last beyond the duration of the projects themselves, without the continued provision of external resources to sustain benefits. In 2006, the U.S. Agency for International Development (USAID) Office of Food for Peace (FFP) began requiring that all development food assistance projects include explicit explanations of how projects intend to ensure the sustainability of activities and benefits after each project’s end. From 2009–2015, FFP, through the USAID funded Food and Nutrition Technical Assistance Project (FANTA), provided support to Tufts University to conduct a multi-country study to assess the effectiveness of FFP-supported projects’ sustainability plans and exit strategies. The objective of the study was to determine what factors enhanced the likelihood of sustained project activities and benefits in order to provide guidance to future development projects (whether food-assisted or not) on how to ensure sustainability.

Abstract 4 Title: Ex-Post Evaluation of a Millennium Challenge Corporation-funded Girls’ Education Program in Burkina Faso

Presentation Abstract 4:

The evaluators found a high-level of lasting impact four years after project close-out and attributed this to the program’s approach. The study found that girls’ enrollment, test scores in math and French and graduation rates all continued to be significantly better. This ex-post evaluation utilized the quasi-experimental regression discontinuity design, comparing hundreds of schools in the program with hundreds not participating. When major cuts to US foreign aid are being contemplated, the policy implications of ex-post evaluations are clear. By empowering Agencies to identify their most effective and lasting contributions, ex-post evaluations provide rigorous evidence to bolster and improve US development assistance.

Abstract 5 Title: Lessons Learned from USAID's Water Communications and Knowledge Management Sustainability Evaluation Series
Presentation Abstract 5:

The Water CKM Project is conducting a series of ex-post performance evaluations of closed USAID-funded water and sanitation projects to further USAID’s understanding of why its completed WASH activities have been sustained or not. Evaluations have been conducted in Madagascar, Indonesia and Ethiopia. Although conducted under the same mechanism, each evaluation has experienced unique challenges to obtaining documentation needed from former implementers and to being able to access endline datasets and methods to be able to replicate, with differing impact on results.   

Audience Level: All Audiences

Session Abstract (150 words): 

This session presents five examples of how implementers and evaluators are examining the long-term impact of international development projects through ex-post evaluations. This is a cross-sectoral panel, suggesting that there is value in ex-post evaluations regardless of the sector. The first example looks at the drivers for sustaining organizational and technical capacity of civil society actors in a politically volatile environment. The second example presents the results of an ex-post evaluation of a program designed to improve children’s health and nutrition. The third example shares results from a multi-country study conducted between 2009-15 that assessed the effectiveness of food assistance projects’ sustainability plans and exit strategies. The fourth example was conducted four years after project close, with comparison studies conducted three and seven years into the project’s intervention. The fifth example discusses lessons learned from executing a series of ex-post evaluations of closed water and sanitation projects.