Design strategies for impact evaluations: A new tool to get the evidence r

Session Number: 2363
Track: Quantitative Methods: Theory and Design
Session Type: Panel
Tags: assessment, Community Engagement, evaluability assessment, evaluation preparation, Impact evaluation, utilization-focused evaluation
Session Chair: Jean Paul Petraud, PhD [MECap Fellow, Impact Evaluations - USAID/PPL/LER]
Presenter 2: Jean Paul Petraud, PhD [MECap Fellow, Impact Evaluations - USAID/PPL/LER]
Presenter 3: Molly Hageboeck [Senior Advisor for Research and Evaluation - MANAGEMENT SYSTEMS INTERNATIONAL]
Presenter 4: Mateusz Pucilowski [Director - Impact Evaluation - Social Impact]
Time: Nov 03, 2018 (11:15 AM - 12:00 PM)
Room: CC - 20

Abstract 1 Title: The value of Evaluability Assessment for Impact Evaluations at USAID

Presentation Abstract 1:

The panel will consist in a free flowing conversation from experts around the main items in the tool. In particular, the expert discussion will focus on two themes:

  • In light of experience, how may each task help overcome common IE pitfalls?
  • What might be challenges in conducting an Evaluability Assessments at USAID and what might solutions look like?

Abstract 2 Title: Evaluability Assessment: A systematic approach fitted to IEs at USAID

Presentation Abstract 2:

Jean Paul Pétraud, a MECap (Social Solutions International, Expanding Monitoring and Evaluation Capacities Task Order) Fellow with PPL (Bureau of Policy, Planning and Learning)  at USAID has led the effort to design of the “EA for IE” tool. A team of USAID researchers has summarized an EA for IEs in four iterative and overlapping components:

 

  1. Understanding the theory of change and where are knowledge gaps with regards to impact.
  2. Understanding the usefulness of a potential impact evaluation.
  3. Understanding the stakeholders of the activity/project and how they would be amenable to IE designs and the determination of treatment and comparison groups.
  4. Exploring and assessing the feasibility of an impact evaluation.
    • Given the activity/project implementation timeline, is is possible to embed an impact evaluation in the implementation roll-out?
    • What are Minimum Detectable Effect requirements?

Abstract 3 Title: Evidence for the need for EAs for IEs at USAID

Presentation Abstract 3:

A co-author of the recent report “Opportunities for Enhancing Returns on E3 Investments in IEs”, Molly Hageboeck, Senior Technical Advisor at Management Systems International, brings a historical and practical perspective on impact evaluations at USAID. She will show how, while Evaluability Assessment is not new to the agency, this recent report calls for a new impetus for early systematic assessment and planning of impact evaluations.


Abstract 4 Title: Designing and managing IEs for USAID: How EAs mitigates risks of failure

Presentation Abstract 4:

A significant portion of IEs planned by USAID personnel never get off the drawing board. Many more are terminated or reframed as performance evaluations before they are completed. While each case is unique, there are a number of important commonalities across these cases. In this session, Mateusz Pucilowski (Director of Impact Evaluation at Social Impact), will highlight the most common causes of failure and present practical strategies for enhancing the prospects for successful IEs. He will explain how an EA, through early and methodical scrutiny of the intervention theory of change and implementation plan, can help USAID personnel and evaluators mitigate such risks.

 


Audience Level: All Audiences

Session Abstract (150 words): 

This panel presentation from impact evaluation experts from USAID, Social Solutions International, Social Impact and MSI will discuss a new tool produced by USAID, that provides recommendations for conducting an Evaluability Assessment (EA) for impact evaluations (IEs) with the agency. The “EA for IE” tool describes a series of steps and tasks recommended to conduct an EA when a technical team thinks they want to do an impact evaluation of an activity or a component of an activity. A copy of the new tool will be shared with the public for the first time at this AEA session.

 

This panel presentation is relevant both to internal and external actors because the tool stresses evaluation methodology (the truth) and evaluation stakeholder participation (the power).  Designed for international donors, stakeholders and partners, this presentation will fosters a structured conversation about prerequisites to setting up IEs that produce robust and useful evidence