Comprehensive Designs for Cost-Inclusive Evaluations: the Resource → activity → process → outcome analysis (RAPOA) Model

Session Number: 1069
Track: Costs, Effectiveness, Benefits, and Economics
Session Type: Panel
Tags: collaborative evaluation, community based organization, cost-inclusive evaluation, multi-site evaluation
Session Chair: Brian T Yates, Ph.D. [Professor - American University]
Discussant: Brian T Yates, Ph.D. [Professor - American University]
Presenter 1: Lana Wald [American University]
Presenter 2: Katheryn Ryan
Presenter 3: Sarah Hornack [Psychologist - Children's National Health System]
Presenter 4: Alexis French
Time: Oct 27, 2016 (03:00 PM - 04:30 PM)
Room: International North A

Abstract 1 Title: Cost-Benefit and Cost-Utility Analyses in a Randomized Clinical Trial for Psychological versus Medical Treatments for Depression
Presentation Abstract 1:

Implementing a cost-inclusive evaluation within a randomized clinical trial (RCT) often provides the “best-case scenario” for designing and implementing evaluations. However, when an evaluation is added on to an RCT and not directly funded as such, it becomes crucial to design the evaluation properly to maintain the fidelity and accuracy of potential outcomes. The objective of this talk is to provide information regarding design methods used to effectively implement a cost-inclusive evaluation as part of an RCT and ways to amend inconsistencies when a previous design had already been put in place. An RCT that compared two different therapeutic modalities for the treatment of seasonal affective disorder (Rohan et al., Grant Number 1 R01 MH0789 82-01A2) will be used as an example.


Abstract 2 Title: What to measure and how to assess programs when a comparison group is unethical: A cost and QALY evaluation of Suicide Hotlines
Presentation Abstract 2:

Real-world challenges to conducting cost-inclusive RCTs, including time, costs, and ethics, indicate that more flexible research design methods are needed. This panelist will describe an alternative evaluation method, probabilistic sensitivity analysis (PSA), which is used to generate a comparison mathematical model using known risk factors as a population-based control. Using suicide hotlines as an example, this talk will focus on PSA as an alternative design method when comparison groups would be unethical. It will also address how to select program outcomes in terms of cost and benefits or cost utility in Quality Adjusted Life Years (QALYs). The objective is to provide audience members with an overview of flexile research design methods in order to more effectively design studies when real-world constraints, stakeholders’ preference, and ethical challenges limit options for program evaluation.


Abstract 3 Title: Quasi-Experimental Longitudinal Large-N Design for Evaluation of Benefit-Cost Relationships Moderated by Parenthood, Gender, and Gender-Sensitivity of Program
Presentation Abstract 3:

We used a longitudinal naturalistic design to evaluate costs, cost-savings, employment income, and benefit/cost of intensive inpatient programs (IIPs) for substance abuse treatment. Benefits quantified from state databases yielded 48 months of monthly individual-level data on employment income, income support, and health and criminal justice services for each of 14,497 women and men who abused drugs. Natural variations in gender-sensitivity of treatment between 13 sites in the state were quantified by colleagues using Rasch analyses. Use and costs of services increased and income decreased after program participation. More gender-sensitive programs were less costly and somewhat more effective. Mothers receiving more gender-sensitive treatment showed less increase in service use and service cost following treatment. Using actual data on service use instead of patient self-reports provided a picture of benefit/cost ratios for the programs that were intuitively understandable but radically different from expectations.


Abstract 4 Title: Challenges of Evaluating Costs and Effectiveness of Parent-Child Interaction Therapy From a Quasi-Experimental Design

Presentation Abstract 4:

Often when evaluating children’s mental health programs, researchers tend to focus on effectiveness primarily, with cost of resources gaining less attention and interest. As a result, examining the cost-effectiveness of these programs generally involve collecting cost data post hoc. This panelist will discuss the challenges of collecting cost data post hoc, especially from various stakeholder perspectives. Specifically, the panelist will review problems identified from a quasi-experimental study comparing cost-effectiveness of clinic- versus home-based Parent Child Interaction Therapy (PCIT) from several perspectives (i.e., client, provider, administrative, and overall). The objective of this discussion is to review the challenges with collecting outcome and cost data in a quasi-experimental study, including how to handle missing and nonnormal data and how to collect cost data post hoc. In addition, the panelist will review best practices for collecting cost data in real time.


Audience Level: All Audiences

Session Abstract: 

Resource → Activity → Process → Outcome Analysis (RAPOA) model offers a flexible yet comprehensive design for programs, evaluation of programs, and communication of findings of program evaluation. Four evaluations performed by graduate students explore the degree to which the RAPOA integration of program, evaluation, and information design actually work in the real worlds of health and human services. These cost-inclusive evaluations were conducted in suicide prevention hotlines, multisite community-based programs for substance abuse treatment, a clinical trial prevention of recurrence of seasonal affective disorder, and a quasiexperimental study of training parenting skills in minority and majority families. Together these papers explore the advantages and challenges of using one model to design programs, cost-inclusive evaluations, and to describe and analyze findings for diverse programs in multi- as well as single-site settings.