Evaluation 2017: From Learning to Action

View Printable Version

Building a National Evaluation in Diverse Community Settings with a Common Protocol

Session Number: 1892
Track: Government Evaluation
Session Type: Panel
Tags: "hard to reach" populations, , multi-year strategy, adaptive design, capacity building, CNCS, Data-based decision making, Evaluation capacity building, evaluation goals, evaluation strategies, evidence based decision-making, Impact evaluation, longitudinal evaluation, monitoring and evaluation (M&E), Nonprofit Evaluations, nonprofits, pre-post surveys, quantitative methods, senior volunteers, volunteerism
Session Chair: Gail Kohn [Age-Friendly City DC Coordinator - Age-Friendly City DC]
Discussant: Gail Kohn [Age-Friendly City DC Coordinator - Age-Friendly City DC]
Presenter 1: Mary Hyde [Corporation for National and Community Service]
Presenter 2: Annie Georges [JBS International]
Presenter 3: Robert Cox [Corporation for National and Community Service ]
Presentation 1 Additional Author: Anthony Nerino
Time: Nov 08, 2017 (06:15 PM - 07:15 PM)
Room: Roosevelt 4

Abstract 1 Title: Performance Measurement to a Rigorous Evaluation Strategy
Presentation Abstract 1:

The first presentation will discuss the steps taken to assess evidence about the validity, reliability and utility of performance measures data on the effectiveness of community-based programs that support adult volunteers and beneficiaries. The first step created project work plans to identify community needs and service activities; identify inputs, outputs and outcomes of services; and provide data that allows grantees to develop, track and report services.

The second step built on the project work plans and grantees’ experience to refine performance measurements for inclusion in an evaluation design. The third step implemented the evaluation using national standard surveys to measure inputs and outcomes, and assessing the effectiveness of programs on participants and beneficiaries over time. Presenters will discuss the benefits of engaging with content experts and project directors with experience in implementing the programs. They will discuss how the agency relied on building grantees’ evidence-based infrastructure to inform policy discussions.

Abstract 2 Title: Maintaining Fidelity in a National Evaluation in Diverse Community Settings
Presentation Abstract 2:

The second presentation will discuss how the evaluation was implemented using a strategy that engaged and sustained grantees’ involvement. The presenter will discuss evaluation methods used to (1) assess readiness of grantees to participate in a multi-year evaluation using a common protocol, (2) secure buy-in from grantees and stakeholders from design to execution, (3) engage with content experts and programs to learn what is working and adapt implementation for a more successful evaluation, (4) develop training materials and provide support to help grantees successfully participate, (5) implement a practical adaptive survey design including different contact modes to non-respondents for higher retention rate through all evaluation phases. The presenter will share lessons learned on the benefits and costs of implementing a longitudinal evaluation with diverse groups of community-based grantees versus a shorter time-based design. The presenter will discuss the staff and monetary resources to support this type of longitudinal progressive evaluation.

Abstract 3 Title: Using the Evaluation Results to Inform Policy Discussions
Presentation Abstract 3:

The third presentation will discuss how the evaluation results are informing ongoing policy discussions to strengthen program delivery, understand participants’ motivation, inform recruitment strategies, and promote organizational learning that focuses on performance evaluation. The presenter will discuss effective processes used to continuously build the grantees’ evidence-based infrastructure, and how to use evaluation results to build on the strengths and experience of performance measurement and management to promote clarity for grantees. Part of the evidence-based strategy includes the CNCS Evidence Exchange, a digital repository of research, evaluation reports, and data focusing on national service, social innovation, civic engagement, and volunteering. The Evidence Exchangeprovides grantees access to information about best practices for each component of their performance measurement. The presenter will discuss how the results are informing processes to develop, track and report services.

Theme: Learning What Works and Why
Audience Level: All Audiences

Session Abstract (150 words): 

Foundations and non-profits that fund community-based programs are increasingly requiring their grantees to conduct or participate in evaluation to demonstrate program effectiveness. The challenge is to develop an evaluation design that accommodates diverse groups of grantees; thus in order to be effective and successful the evaluation cannot be a “one-size” fits all. The session will demonstrate how a longitudinal evaluation can be successfully implemented and build capacity with decentralized programs providing services to vulnerable populations. The session will describe (1) the process of moving from work plans, to performance measurement then ultimately to a more rigorous evaluation, (2) challenges faced in maintaining fidelity when working with diverse organizations, and (3) using evaluation results to inform policy discussions. Panelists will share successes and challenges of designing a national evaluation study that was then implemented with more than 500 grantees of varying capacity, staff resources and expertise.

For questions or concerns about your event registration, please contact registration@eval.org or 202-367-1173.

For questions about your account, membership status, or help logging in, please contact info@eval.org.

Cancellation Policy: Refunds less a $50 fee will be granted for requests received in writing prior to 11:59 PM EDT October 16, 2017. Email cancellation requests to registration@eval.org. Fax request to (202) 367-2173. All refunds are processed after the meeting. After October 16, 2017 all sales are final. For Evaluation 2017, international attendees and presenters who encounter complications due to the international travel environment will have up to 30 days after the event to request a refund and submit appropriate documentation. No administrative fee will apply for the international requests. The $50 fee will be waived for registrants who planned to travel into the US and experienced international travel issues.