Evaluating Integration: Are We Asking the Right Questions?

Session Number: 2406
Track: Mixed Methods Evaluation
Session Type: Panel
Tags: Integration
Session Chair: Andrew I Epstein [Technical Director - Social Impact, Inc.]
Presenter 1: Salman Ayaz Jaffer [Senior Program Director - Social Impact]
Presenter 2: Bindza Ginindza, MERL officer [MERL officer]
Presenter 3: Andrew I Epstein [Technical Director - Social Impact, Inc.]
Presenter 4: Alison Smith [Program Associate, Impact Evaluation - Social Impact]
Presentation 2 Additional Author: Daisy Nathan Kisyombe [Monitoring and Evaluations Technical Advisor - Pact Inc. ]
Presentation 2 Additional Author: Nicole Miller [Country Director - Pact Swaziland]
Presentation 2 Additional Author: Erica Kuhlik [Senior Advisor, Results and Measurement - Pact]
Time: Nov 01, 2018 (03:45 PM - 04:45 PM)
Room: CC - 1

Abstract 1 Title: Defining and deconstructing the concept of integration
Presentation Abstract 1:

The first step in effectively evaluating integration is defining it. We will discuss different ways to define integration, focusing on a framework used by USAID and evaluated in Social Impact’s impact evaluation of Malawi’s Country Development Cooperation Strategy (CDCS): the “3C” approach of collaboration, coordination and co-location. We will also discuss the merits of different types of measurement in an integrated programming context, including ongoing monitoring and evaluative methods.

Abstract 2 Title: Using unique identification codes to improve measurement of integrated programming and maximize beneficiary outcomes
Presentation Abstract 2:

The USAID-funded Umliba Loya Embili project in Swaziland scales up integrated, evidence-based health and social services- including HIV, sexual reproductive health, education, economic strengthening, psychosocial, parenting, and child protection- to adolescent girls and young women (AGYW).

At enrolment, the project assigned beneficiaries unique identification codes used across project partners to track their uptake of activities across the range of services, and over time. Individual tracking has enabled routine analysis and measurement of service layering among beneficiaries. The latest analysis found that 81% (44,857) of AGYW received six layered services, and that improved program outcomes are associated with participation in integrated interventions.

The unique identifier has enabled the project to analyse data on efficient combinations of services that are likely to yield better outcomes for beneficiaries. Future innovations may include replacing the project ID with the National ID to analyse service integration across Government and other donor-funded providers. 

Abstract 3 Title: Using integration domains and categorization to provide structure to a mixed methods evaluation
Presentation Abstract 3:

Social Impact's evaluation of the Malawi Country Development Cooperation Strategy includes an annual qualitative stakeholder analysis (SHA) that functions as a process evaluation to assess integration in practice. The 2017 SHA was a case study of integration of USAID programs within the agriculture sector. Social Impact designed a survey to assess integration practices on the following domains based on the 3C approach: governance; administration; autonomy; mutuality; proximity; and performance and impact. We will discuss these domains and the way we translated them to a survey instrument, along with the process used to categorize integration relationships into value chain, co-equivalent, and cross-sector types.

Abstract 4 Title: The beneficiary experience of integration: Experience with Malawi CDCS
Presentation Abstract 4:

Social Impact's evaluation of the Malawi Country Development Cooperation Strategy included a focus group discussion (FGD) component to assess the effects of integration on beneficiaries beyond the level of quantitatively measured health, agricultural, economic and wellness outcomes. Analysis of FGD data uncovered the following unanticipated benefits of integration that we will discuss: reduced time burden, increased message consistency, reduction of duplication of services, increased program quality and increased community unity. These unanticipated effects of integration on beneficiaries prompt us to ask the following questions of integrated programs: is measuring outcomes enough? are we collecting data from the right places or people? who currently has the power to make decisions based on these data, and where does the power need to shift to further an integrated development agenda? 

Audience Level: All Audiences

Session Abstract (150 words): 

When measuring integrated programming, it can be difficult to separate the effects of integration from the effects of individual programs or services. We as evaluators of integration have taken a step back to assess whether we are asking the right questions about integration, and how we should be measuring it. This panel explores these questions through the experiences of both integration of services within a program, and integration of programs across a funder’s portfolio.  

The first part of the panel will focus on defining integration and discussing the merits of different types of measurement. The next piece will describe the USAID-funded Umliba Loya Embili project in Swaziland, that uses unique identification codes to track the uptake of integrated services and measure the effects of service layering among beneficiaries. The final part of the panel will highlight mixed methods used to evaluate the integration component of Malawi’s Country Development Cooperation Strategy.