Evaluation 2015: Exemplary Evaluations in a Multicultural World

View Printable Version

Designing Real World Evaluations That Are Fit for Purpose

Session Number: 2416
Track: Human Services Evaluation
Session Type: Panel
Tags: International and Cross Cultural Evaluat
Session Chair: Heidi Reynolds [MEASURE Evaluation]
Discussant: Heidi Reynolds [MEASURE Evaluation]
Presenter 1: Jen Chapman [Futures Group]
Presenter 2: Molly Cannon [Futures Group]
Presenter 3: Lisa Parker [Futures Group]
Presentation 1 Additional Author: Robert Mswia [Senior Research Advisor - Futures Group]
Presentation 1 Additional Author: Mwila Kanema [Study Coordinator - Futures Group]
Presentation 2 Additional Author: Nena do Nascimento [M&E Advisor - Futures Group]
Presentation 2 Additional Author: Zulfiya Charyeva [Senior Research Analyst - Futures Group]
Presentation 2 Additional Author: Karen Foreit [Senior Fellow - Futures Group]
Presentation 3 Additional Author: Kednel Francois [Study Coordinator - Independent Consultant]
Presentation 3 Additional Author: Olbeg Desinor [Child Survival and Community Health Advisor - USAID]
Presentation 3 Additional Author: Toni Cela [Country Coordinator - Interuniversity Institute for Research and Development]
Time: Nov 14, 2015 (08:00 AM - 09:30 AM)
Room: Skyway 273

Abstract 1 Title: Impact Evaluation of Savings and Internal Lending Communities (SILC), a Community Savings Group Intervention, on Child and Household Well-being in Zambia
Presentation Abstract 1: Dr. Jenifer Chapman is a Senior Evaluation Advisor at Futures Group with 15 years of experience in global health research and evaluation, and the Principal Investigator on a USAID-funded Implementation Science Research study evaluating the impact of savings groups on child wellbeing in Zambia.
Savings and internal lending communities (SILC), a community savings group model, have been rolled out in nearly 40 countries. The U.S. Government is supporting the establishment of SILC groups through orphans and vulnerable children programs, with the assumption that SILC group membership impacts household economic status (HES), and that improved HES leads to improvements in child well-being. However, this hypothesis was untested. USAID was seeking information on the impact of SILC group membership on participant, household and child well-being, in order to inform programming globally. To obtain generalizable, impact level information of this intervention for which participants self-select, we conducted a 3-year quasi-experimental trial in Zambia.

Presentation 1 Other Authors: Patricia Funjika, funjipat@yahoo.com, Research Fellow, Institute of Economic and Social Research, University of Zambia
Zulfiya Chariyeva, ZCharyeva@futuresgroup.com, Senior Research Analyst, Futures Group
Mathew Ngunga, mngunga@futuresgroup.com, Senior M&E Advisor, Futures Group
Jane Chege, Jane_Chege@wvi.org, Senior Director, Monitoring & Evaluation; Evidence & Learning
World Vision International
Jolly Kamwanga, jollykamwanga@yahoo.com, Deputy Director, Institute of Economic and Social Research, University of Zambia

Abstract 2 Title: Mozambique Program Assessment—Community Care for Vulnerable Children in an Integrated Vulnerable Children and Home-Based Care Program
Presentation Abstract 2: Molly Cannon has been working in public health programming, research and evaluation in international and domestic settings for 20 years. She is a Senior M&E Specialist with Futures Group/MEASURE Evaluation where she has led several studies related to Orphan and Vulnerable Children (OVC) and Home Based Care (HBC).

In Mozambique, community-based organizations (CBOs) traditionally deploy single-purpose community home visitors. Some of the volunteers serve only OVC and others serve only HBC clients. The USAID funded Community Care Project (PCC) changed its care model by training dual-purpose community home visitors who serve both OVC and HBC clients. USAID/Mozambique and the Mozambique government were interested in understanding the impact of this model on OVC outcomes. After determining that an impact evaluation with comparison group was not feasible, we conducted a mixed methods study in two provinces aimed at understanding how the program was implemented and what factors influenced service provision to OVC.

Abstract 3 Title: Retrospective Qualitative Assessment of a Savings and Internal Lending Community (SILC) Activity in Haiti
Presentation Abstract 3: Dr. Lisa Parker is a Monitoring and Evaluation Technical Advisor with Futures Group with 15 years’ experience in public health, research, and evaluation. She is a Principal Investigator of the USAID-funded HIVCore project assessment of a Savings and Internal Lending Community (SILC) activity in Haiti.
Various savings and lending models have been implemented in Haiti, but few have evaluated their impact on vulnerable populations. USAID/Haiti requested an impact evaluation of a recently concluded SILC activity. We determined that the most appropriate and feasible study design was a retrospective qualitative assessment focusing on program sustainability, participant perceptions, and lessons learned. In-depth interviews and focus group discussions were conducted with SILC members, community mobilizers, and selected staff of community-based organizations and the implementing partner. The study examined SILC impact on member needs; individual, child and household well-being; and factors that fostered/hindered sustainability of SILC groups after donor support ended.

Presentation 3 Other Authors: Karen Foreit
Senior Fellow
Futures Group

Audience Level: None

Session Abstract: 

Evaluations are typically initiated with a request by a donor or funding agency. These requests are often for impact level information with a rigorous impact evaluation study design in mind.  However, impact evaluations, and certainly randomized controlled trials, are not always appropriate or feasible –due to methodological concerns or cost or time limitations, nor are they necessarily the appropriate design to obtain the information needed to make decisions. However, many impact evaluations are implemented at tremendous cost, yielding vast amounts of data that are not used. We will present experiences from three different evaluations for which we were asked to gather impact-level information on child and household well-being. We will describe the approach we used to revise those designs after taking a more “real world” approach to identify the appropriate research questions and determine the best-fit study design so that the right information was available to aid in timely decision-making. Our goal as “real world” evaluators should be to facilitate the choice of appropriate study design based on the questions that need to be answered, within what is feasible given the context and resources.

For questions or concerns about your event registration, please contact registration@eval.org or 202-367-1173.

For questions about your account, membership status, or help logging in, please contact info@eval.org.

Cancellation Policy: Refunds less a $50 fee will be granted for requests received in writing prior to 11:59 PM EDT October 6, 2015. Email cancellation requests to registration@eval.org. Fax request to (202) 367-2173. All refunds are processed after the meeting. After October 6, 2015 all sales are final.