Evaluation 2017: From Learning to Action

View Printable Version

New Approaches to Evaluating Complex Social Service Systems

Session Number: 2091
Track: Use and Influence of Evaluation
Session Type: Panel
Tags: advancing evidence, Community-Based Settings, evaluation strategies, Human Services Evaluation
Session Chair: T'Pring R. Westbrook [Senior Associate - The Annie E. Casey Foundation]
Discussant: Allison Holmes
Presenter 1: Kathleen Dwyer
Presenter 2: Margaret Sullivan [Researcher - Mathematica Policy Research]
Presenter 3: Susan J Popkin [Director, Neighborhoods and Youth Development - Urban Institute]
Presentation 1 Additional Author: Carli Wulff
Presentation 3 Additional Author: Marla McDaniel [Senior Research Associate - Urban Institute]
Time: Nov 10, 2017 (01:45 PM - 03:15 PM)
Room: Maryland C

Abstract 1 Title: Toward A Research Agenda for Integrated Child and Adult Services
Presentation Abstract 1:

Policymakers, providers, and funders are increasingly interested in approaches that meet the needs of low-income families through the intentional combination of child-focused and adult-focused services. While evaluations of programs implemented in the 1990s to serve both children and parents generally failed to show evidence of effectiveness, new programs may address their weaknesses. Evaluators and others hypothesized that the early programs may have provided services that were of insufficient quality or intensity to result in impacts for both children and parents, or that the services were not fully integrated. Drawing on work sponsored by the Administration for Children and Families and the Annie E. Casey Foundation, we will discuss considerations for future evaluations, including:

  • the need for descriptive, implementation research;
  • measurement needs (e.g., service alignment, coordination across organizations); and
  • data systems (e.g., capturing quality and intensity of services, integrated child and parent data, long-term outcomes).

Abstract 2 Title: Workforce Development through Bundled Services: Evaluation Challenges
Presentation Abstract 2:

The Working Families Success Network and affiliated Working Students Success Network exemplify the bundled approach to seamlessly delivering workforce and other training programs and supports to help disadvantaged people achieve self-sufficiency. Evaluating the effectiveness of these initiatives—which are implemented at community-based organizations and colleges around the country—requires an understanding that they are, by design, customized to fit organizational capacities and participant needs. This presentation will highlight how Mathematica is addressing three related challenges in our evaluation of these programs. First, we are conducting factor, network, and qualitative analyses of site-level data to clearly define “the intervention” by seeking to identify patterns in bundled service delivery. Second, we are providing technical assistance to sites to support their collection of service-receipt and outcomes data. Finally, we are working closely with the client to refine the theory of change and clearly define outcomes to lay the groundwork for future impact analyses.


Abstract 3 Title: Building Evidence for Place-Based Initiatives: HOST and FCCC
Presentation Abstract 3:

The Urban Institute’s HOST (Housing Opportunities and Services Together) Demonstration and the Casey Foundation’s Family Centered Community Change (FCCC) Initiative are place-based two-generation approaches with the shared goal of improving the life chances and economic circumstances of low-income families. HOST uses public and assisted housing as a platform for targeting families with complex needs; FCCC builds two-generation approaches into existing community change initiatives. Because of the complexity of these efforts, we have used a formative, multi-method evaluation approach that includes staff interviews and program observations; participant surveys, focus groups, and interviews; and partnering with local teams to develop program indicators to use for monitoring key outcomes. The findings from this research offers insight into the challenges of developing effective partnerships, integrating services and systems, and providing services of sufficient intensity for both adults and children that will truly improve their long-term outcomes.


Theme: Learning to Enhance Evaluation
Audience Level: All Audiences

Session Abstract (150 words): 

Increasingly, programs aimed at serving individuals and families with low-incomes are embedded in complex systems.  Gaining economic self-sufficiency requires support across areas such as child-care, housing, education, and health. To provide comprehensive services, implementers are developing partnership networks, integrated systems, and bundled services allowing program participants to experience seamless delivery of multiple services under the umbrella of a single program. While this “no wrong door” approach simplifies a program participant’s experience, it complicates evaluation efforts.  Traditional methodologies that rely on isolating effects, determining core components, and standardization of procedures are challenged by complex systems providing individualized protocols to meet shifting participant needs. This panel will address new approaches to evaluating these complex systems in the areas of intentional integration of services for parents and children, place-based initiatives, and workforce development. Examination of these diverse areas will reveal common issues and common solutions to conducting program evaluation in complex systems.



For questions or concerns about your event registration, please contact registration@eval.org or 202-367-1173.

For questions about your account, membership status, or help logging in, please contact info@eval.org.



Cancellation Policy: Refunds less a $50 fee will be granted for requests received in writing prior to 11:59 PM EDT October 16, 2017. Email cancellation requests to registration@eval.org. Fax request to (202) 367-2173. All refunds are processed after the meeting. After October 16, 2017 all sales are final. For Evaluation 2017, international attendees and presenters who encounter complications due to the international travel environment will have up to 30 days after the event to request a refund and submit appropriate documentation. No administrative fee will apply for the international requests. The $50 fee will be waived for registrants who planned to travel into the US and experienced international travel issues.