Conducting Propensity Score Matching and Survival Analysis to Predict Recidivism for a Home Visitation Program Evaluation: Case Study, and Applying Results

Session Number: 2040
Track: Quantitative Methods: Theory and Design
Session Type: Panel
Tags: propensity scores, quantitative methods, quasi-experimental design
Session Chair: Peter Lovegrove [Research Associate - JBS International]
Presenter 1: Stacey Houston
Presenter 2: Rebecca S Frazier [Research Associate II - JBS International]
Presenter 3: Shannon Williams
Time: Oct 26, 2016 (04:30 PM - 06:00 PM)
Room: A703

Abstract 1 Title: Constructing a Propensity Score Weights and a Propensity Score Matched Sample.
Presentation Abstract 1:

Matching using propensity scores is a technique that can be used to improve the validity of a quasi-experimental comparison study. They can equalize non-randomized treatment and comparison groups more effectively than other forms of matching (e.g. univariate), and produce less biased estimates compared to regression techniques that include statistical controls for baseline differences. The presenters will detail the process for constructing propensity scores in SPSS, providing step-by-step guidance intended for a lay-audience. They will then provide instructions for using the scores as weights or matching construct to adjust the analysis to minimize baseline non-equivalence prior to conducting an impact analysis. The audience will receive handouts including sample SPSS syntax for the procedures.

Abstract 2 Title: An Introduction to Survival Analysis and Applications to Predicting Recidivism.
Presentation Abstract 2:

Survival analysis is a commonly used technique that predicts the likelihood of dichotomous outcomes over time, such as recidivism or Child Protective Services referrals. This methodology is especially useful when participants are entering the program at various times-- as it controls for the length of time that individuals have been in the program and can include participants with widely varying lengths of program participation in a single model. In this presentation, we will present an introduction to survival analyses and the Cox regression method. We also will walk through an example with SPSS syntax and graphs demonstrating how this procedure was used to predict the likelihood of having a subsequent child protective services referral among participants in a child abuse prevention program and a propensity score matched comparison group. Finally, we will discuss other potential applications of this methodology and the strengths and limitations of this approach.

Abstract 3 Title: Strategies for Communicating Quasi-experimental Results and Facilitating Program Learning.
Presentation Abstract 3:

Non-profit organizations are increasingly being expected to demonstrate the rigorous impact of their programs, requiring sophisticated statistical models in many cases. This quasi-experimental comparison study was initiated as an AmeriCorps funding requirement for those grantees such as the Child Abuse Prevention Center of Sacramento, which has received nearly $1million/year for 90 B&B members, and so it was necessary to communicate to a variety of stakeholders. The B&B program was eager to share the positive outcomes of their quasi-experimental evaluation with all its partners, but explaining the relevance and significance of the study outcomes and their overall relevance to program implementation was challenging. This presentation will discuss best practices learned during the project for communicating technical evaluation findings to diverse stakeholders who were accustomed to observation data (e.g., county child welfare agency, county government, and local community-based organizations).

Audience Level: All Audiences

Session Abstract: 

Conducting a quasi-experimental evaluation requires a series of key technical analytical capabilities, especially when using secondary data to construct the comparison group. This was true of LPCs’ and JBS’ study of the Birth and Beyond program for child abuse and neglect prevention. The impact of the program was assessed using Child Protective Services data. The research team used propensity scores to adjust the impact analysis sample to reduce threats to validity. They also used survival analysis to allow for impact analyses to be conducted on a large sample whose intake or eligibility for intake was established on a rolling basis. In a series of presentations, the panel will detail the process for: 1) constructing and using propensity scores for a matched sample, 2) conducting a survival analysis, and (3) using the results to facilitate program learning. The team will provide handouts detailing the syntax and schedule for the SPSS procedure.