Strengthening Program Evaluation when RCT is not Feasible: Propensity Score Matching in Practice

Session Number: 2410
Track: Nonprofit and Foundations
Session Type: Panel
Tags: Education, evidence, propensity score matching, RCT
Session Chair: Kevin Rafter [Director of Client Engagement - Harder+Company Community Research]
Presenter 1: Michael Scuello [Senior Associate - Metis Associates, Inc.]
Presenter 2: Joelle Katherine Greene, PhD [Senior Research Associate - Harder+Company Community Research]
Presenter 3: Miranda Yates [Assistant Executive Director for Strategy, Evaluation, and Learning - Good Shepherd Services]
Presentation 1 Additional Author: Donna Tapper [Managing Senior Associate - Metis Associates, Inc.]
Time: Oct 28, 2016 (04:30 PM - 05:15 PM)
Room: A601

Abstract 1 Title: Design Considerations in Propensity Score Matching: Benefits and Challenges
Presentation Abstract 1:

Propensity score matching (PSM) is growing in popularity as an effective technique for estimating intervention and program effects in observational and quasi-experimental evaluations. We overview the propensity score matching approach, and detail its implications for impact evaluations. The utility of PSM for impact evaluations is affected by study design considerations. These include attaining the appropriate comparison sample and sample sizes, limitations and opportunities in measurement design, and challenges in novel solutions to identifying appropriate comparison groups. We discuss these and other implementation considerations for optimizing the benefits of the PSM design, with a focus on PSM implementation challenges and solutions arising in studies conducted in educational and health settings.

Abstract 2 Title: Making the Case for Propensity Score Matching in Program Evaluation Design
Presentation Abstract 2:

It is important for evaluators to have a clear rationale for using PSM to estimate program effects. At a minimum this entails communicating with funding, scientific, and program stakeholders around the goals and limitations of PSM for the evaluation. PSM framing and justification must reflect considerations of the type of program, sources of selection bias, study reproducibility, comparison and treatment group differences, and standards of evidence in causal interpretations. We explore considerations for framing and justifying PSM within proposed evaluation designs. We provide practical examples from the proposal development, evaluation design, and early implementation phases of Genesys Works Bay Area.

Abstract 3 Title: Partnering in the Use Propensity Score Matching: A Nonprofit’s Perspective
Presentation Abstract 3:

In the absence of being able to conduct a Randomized Control Trail (RCT), Propensity Score Matching provides an attractive design option to nonprofits looking to build rigorous evidence of program effectiveness. This statistical technique, however, is more complex than many of the evaluation designs typically used, and requires strong and ongoing collaboration among key stakeholders. Drawing upon a recently completed impact evaluation of the Good Shepherd Services Transfer School model, we share lessons learned partnering with an external evaluator and funders at each stage of the evaluation process. We emphasize the unique place of collaboration on PSM and evaluation design considerations related to data sources, subgroup analyses, and communicating and interpreting findings.

Audience Level: Intermediate

Session Abstract: 

As more funders encourage or require grantees to evaluate program effectiveness, there is growing interest in alternatives to the Randomized Control Trial (RCT) approach, which serves as the gold standard in the field. Propensity Score Matching (PSM) is a useful alternative in many situations. However, there are important challenges and complications for program and evaluation stakeholders to understand when designing an evaluation using PSM. This panel will use examples from evaluations recently completed and in progress to highlight the issues related to meeting evidence standards, gathering sufficient data, and communicating the evaluation design and findings to clients and stakeholders.