Evaluation 2017: From Learning to Action

View Printable Version

How to Use a “Bundled” Evaluation Methodology to Translate Learning into Action for Grant-Makers and Smaller Programs and Non-Profits

Session Number: 1997
Track: Organizational Learning & Evaluation Capacity Building
Session Type: Panel
Tags: "hard to reach" populations, adolescents, AmeriCorps, capacity building, Capacity Building Evaluation, cluster, multi-site and multi-level eval, Community Engagement, Corp. for National & Community Service, Data Visualization, Data-based decision making, Disconnected youth, Education, Evaluation capacity building, evaluation coaching, evaluation tools, evidence based decision making, federal evaluation, federal government investments, federal grants, Government, government evaluation, Impact Assessment, Impact evaluation, learning community, learning from evaluation, methodology, monitoring and evaluation (M&E), multi-level modeling, multi-site evaluation, Multiple Imputation, Nonprofit Evaluations, Opportunity Youth, propensity score matching, quantitative methods, quasi-experimental design, volunteerism, youth-focused evaluation
Session Chair: Rebecca S. Frazier [Research Associate II - JBS International]
Discussant: Gina Cardazone [Research Associate - JBS International]
Presenter 1: Adrienne DiTommaso [Research Assistant - Corporation for National and Community Service]
Presenter 2: Rebecca S. Frazier [Research Associate II - JBS International]
Presenter 3: Jennifer A Best [Extension Educator - Iowa State University Extension and Outreach, Scott County]
Presentation 2 Additional Author: Gina Cardazone [Research Associate - JBS International]
Presentation 3 Additional Author: Donald J Pratt [Research Associate - JBS International]
Time: Nov 10, 2017 (01:45 PM - 03:15 PM)
Room: Thurgood Marshall North

Abstract 1 Title: How Grant-Making Organizations Can Use a Bundled Approach to Inform Effective Grantmaking and Support Grantees
Presentation Abstract 1:

In our first presentation, we will explore how grant-making organizations (including government agencies, foundations, and non-profits) can successfully can use a “bundled” evaluation approach to invest in evidence-based programs, leverage limited resources, and build grantee programs’ evaluation capacity through experiential learning. Specifically, we will share lessons learned by CNCS’ Office of Research and Evaluation as they conducted a bundled evaluation of AmeriCorps opportunity youth programs. We will walk grant-makers through the process of selecting, designing, and implementing a bundled evaluation design and explore how this design emerged from CNCS’ dual objectives of building both evidence of effectiveness (“finding what works”) and program evaluation capacity (by involving grantees in a rigorous impact evaluation). Finally, we will discuss how grant-makers can develop a theory of change for a bundled evaluation, the role of the grant-making organization throughout the evaluation, and how grant-makers can translate the results of the evaluation into action.

Abstract 2 Title: How Evaluators Can Use Bundled Evaluation to Increase Evaluation Use and Sustainability
Presentation Abstract 2:

In our second presentation, we will explore how evaluators can successfully use a “bundled” evaluation approach to enhance the interest and involvement of key stakeholders and promote evaluation sustainability for participating organizations. Specifically, we will discuss (1) how to assess if bundled evaluation is a good fit for the evaluator and the organizations involved, (2) what characteristics individual programs need to be successful in a bundled evaluation, (3) how to involve programs in evaluation design and execution, (4) how to create learning communities for evaluation participants, and (5) specific methods for providing individualized feedback and evaluation capacity building services to participants. We will share templates for delivering interim individualized results for programs, a checklist for assessing programs’ evaluation priorities and sustainability goals, and a list of evaluation resources for programs interested in continuing evaluation activities after the end of the bundled evaluation.

Abstract 3 Title: How Programs Can Use Bundled Evaluation to Promote Organizational Learning and Evaluation Sustainability
Presentation Abstract 3:

In our third presentation, we will discuss how individual programs can use participation in a bundled evaluation to promote organizational learning and evaluation sustainability. We will hear from an AmeriCorps grantee about what it is like for a smaller non-profit to participate in a bundled evaluation and to use this methodology with diverse, disconnected youth. She will discuss her organization’s goals for participating in the bundled evaluation, challenges they experienced along the way (including difficulties with comparison group recruitment, survey administration, and tracking youth), and how they addressed these challenges and built their evaluation capacity. Additionally, we will share key comments submitted from the other 18 programs who participated in the bundled evaluation. These comments will highlight which types of technical assistance are most helpful to programs and how the resources from our second presentation can be used to institutionalize evaluation practices and incorporate them into decision-making and program management.

Theme: Learning About Evaluation Use and Users
Audience Level: Beginner, Intermediate, Advanced, All Audiences

Session Abstract (150 words): 

Smaller organizations and non-profits often lack the resources, sample sizes, and expertise to conduct rigorous evaluations. However, by “bundling” smaller programs together into a single evaluation programs can attain higher levels of evaluative evidence, more effectively utilize evaluation resources, promote organizational learning, and build evaluation capacity. Panelists will share strengths and challenges of using an innovative “bundled” evaluation methodology and will explain how this approach can facilitate learning from the diverse perspectives of (1) a grant-maker, (2) an evaluator, and (3) a grantee/non-profit. The presentation will draw on lessons learned from the Corporation for National and Community Service’s (CNCS) propensity-score matched evaluation of 19 AmeriCorps programs serving opportunity youth (16-24 year-olds who are disconnected from school or work). Several hands-on resources will be shared including sample individualized results reports, a checklist for identifying programs’ evaluation priorities, and a list of evaluation sustainability resources for programs.

For questions or concerns about your event registration, please contact registration@eval.org or 202-367-1173.

For questions about your account, membership status, or help logging in, please contact info@eval.org.

Cancellation Policy: Refunds less a $50 fee will be granted for requests received in writing prior to 11:59 PM EDT October 16, 2017. Email cancellation requests to registration@eval.org. Fax request to (202) 367-2173. All refunds are processed after the meeting. After October 16, 2017 all sales are final. For Evaluation 2017, international attendees and presenters who encounter complications due to the international travel environment will have up to 30 days after the event to request a refund and submit appropriate documentation. No administrative fee will apply for the international requests. The $50 fee will be waived for registrants who planned to travel into the US and experienced international travel issues.