Evaluation 2015: Exemplary Evaluations in a Multicultural World

View Printable Version

Impact Evaluations for Large-Scale Public Health Programs

Session Number: 2736
Track: Health Evaluation
Session Type: Multipaper
Tags: Impact evaluation, International Health Evaluation
Session Chair: Sian Louise Curtis [UNC]
Discussant: Sian Louise Curtis [UNC]
Presenter 1: Tory Taylor [Tulane University]
Presenter 2: Jessica Fehringer [UNC-Chapel Hill]
Presenter 3: Martha Priedeman Skiles [Carolina Population Center, UNC-Chapel Hill]
Presenter 4: Peter M Lance [UNC-Chapel Hill]
Presentation 1 Additional Author: Gustavo Angeles [Research Assistant Professor - Carolina Population Center / University of North Carolina at Chapel Hill]
Presentation 2 Additional Author: Lara Lorenzetti [Doctoral Student - UNC-Chapel Hill]
Presentation 3 Additional Author: Stephanie Mullen [Senior Evaluation Advisor - John Snow, Inc.]
Presentation 3 Additional Author: Sian Louise Curtis [Research Associate Professor - UNC]
Presentation 4 Additional Author: Ilene Speizer [Research Professor - UNC-Chapel Hill]
Presentation 4 Additional Author: Lisa Calhoun [Research Associate - UNC-Chapel Hill]
Presentation 4 Additional Author: David Guilkey [Distinguished Professor - UNC-Chapel Hill]
Time: Nov 13, 2015 (01:45 PM - 03:15 PM)
Room: McCormick

Abstract 1 Title: Implementing the Western Highlands Integrated Program Evaluation Baseline Survey in Guatemala—Lessons Learned
Presentation Abstract 1: The Western Highlands Integrated Program evaluation baseline survey in Guatemala, conducted in 2013, was the first stage in a five year study of the program’s impact on poverty and malnutrition in an area home to 1.5 million people. With support from USAID Guatemala, MEASURE Evaluation formed a team to plan and implement a complex design measuring overall impact plus the effect of adding targeted agricultural support to the health and nutrition program. Residents from more than 6,300 households, leaders in 309 communities, and staff at 266 health facilities participated in the baseline. Recognizing that this experience held broad relevance for others interested in bringing large, rigorous evaluations to fruition, 21 key informants from across the implementation team were recruited to discuss lessons learned. Findings address investment expectations, effective program and research partner coordination, preparation for extended fieldwork in remote low-resource settings, establishing community support in politically sensitive environments, and more.
Abstract 2 Title: Nepal – Impact Evaluation of Approaches to Strengthen Health Facility Operation and Management Committees (HFOMCs)
Presentation Abstract 2: In 2014, MEASURE Evaluation planned and led baseline data collection for an impact evaluation of the Nepal Strengthening HFOMCs through a Community Engagement Approach project. The evaluation aim is to understand the added value of integrating gender and social inclusion (GESI) and community engagement interventions into government guidelines, processes, and training for HFOMCs on household- and community-level health outcomes, and on the utilization of healthcare by women and children under age two. With USAID support, we collected quantitative data from over 3,750 households and individuals and 325 community leader groups. We also used qualitative methods, including patient exit interviews, in-depth interviews with HFOMC members, key informant interviews with district stakeholders, and parent focus group discussions. The evaluation illustrates an approach to measuring long causal pathways and scale-up of an intervention through mixed methods and use of project monitoring data. It also provides a rare example of GESI integration evaluation.
Abstract 3 Title: Maximizing Retrospective Data for a Facility-Based Tuberculosis Impact Evaluation in Ukraine
Presentation Abstract 3: Tuberculosis (TB) treatment default contributes to a growing problem of drug-resistant TB in Ukraine. Social support provided to patients receiving continuation treatment is posited to improve adherence rates; however evidence is inconclusive. Employing a retrospective quasi-experimental design, MEASURE Evaluation assessed program targeting and measured program impact on the probability of treatment default, controlling for facility fixed effects. A break in social service provision offered a unique opportunity to collect retrospective data from patients unexposed in 2011 and exposed in 2012 to the program. Exploiting this break, we randomly sampled the high-risk intervention group and selected comparison groups from the same facilities matching on risk status, time, and sex. This retrospective matching allowed unique comparisons across five groups (N=1892), representing different risk status, treatment year, and intervention group. Initial results found successful program targeting of patients at risk for defaulting and significant program effects on likelihood of treatment default.
Abstract 4 Title: A Multi-Country Evaluation of an Urban Reproductive Health Program
Presentation Abstract 4: With the goal of increasing family planning (FP) use in urban areas of Kenya, Nigeria, Senegal and Uttar Pradesh, India, the Bill and Melinda Gates Foundation began the Urban Reproductive Health Initiative (URHI) in 2009. The Measurement, Learning & Evaluation project (MLE) was tasked with rigorous independent evaluation of the URHI in all four countries. The MLE design includes longitudinal samples of urban women as well as a health facility survey in four countries with baseline (2010/2011) and endline (2014/2015) data collection. Longitudinal samples representative of women at the city level at baseline were selected with oversampling of the poor where feasible. The longitudinal data supports a within/fixed effects approach to identification of program impact on changes in modern FP use. Initial impact evaluation results for India show significant effects of exposure to some demand and supply side program components. Similar analyses are planned for Kenya, Nigeria, and Senegal.
Presentation 4 Other Authors: Meghan Corroon, Research Associate, UNC-Chapel Hill
Audience Level: Advanced

Session Abstract: 

The opportunity to uncover essential information for program planning and resource allocation is a strong motivation for impact evaluations in public health.  Findings from large-scale impact evaluations can be instrumental in key policy and program decisions, yet they are not without costs.  This session will draw on practical experiences from the field to showcase successful exemplary evaluation work, illustrate real world challenges for large-scale impact evaluations, and highlight some of the solutions employed to design and conduct impact evaluations in low and middle income countries.  Evaluators, implementers and funders can share in these learnings as we move forward with expanding our understanding of the costs and benefits for rigorous evaluations.

For questions or concerns about your event registration, please contact registration@eval.org or 202-367-1173.

For questions about your account, membership status, or help logging in, please contact info@eval.org.

Cancellation Policy: Refunds less a $50 fee will be granted for requests received in writing prior to 11:59 PM EDT October 6, 2015. Email cancellation requests to registration@eval.org. Fax request to (202) 367-2173. All refunds are processed after the meeting. After October 6, 2015 all sales are final.