Evaluation 2016: Evaluation + Design

View Printable Version

Evaluating Community Sustainability: Issues and Approaches

Session Number: 1654
Track: Independent Consulting
Session Type: Multipaper
Tags: after-action report, climate change, community change, Developmental Evaluation, government evaluation, Place based Initiatives, sustainability, systems change, Theories of change, theories of practice, transforming policy, urban planning, utilization-focused evaluation
Session Chair: Stephen C Maack [Owner and Lead Consultant - REAP Change Consultants]
Discussant: Bob Williams [Independent Evaluation and OD Consultant - All over the place]
Presenter 1: Stephen C Maack [Owner and Lead Consultant - REAP Change Consultants]
Presenter 2: Arlene Hopkins [Principal - Skye Labs]
Presentation 1 Additional Author: Arlene Hopkins [Principal - Skye Labs]
Presentation 2 Additional Author: Stephen C Maack [Owner and Lead Consultant - REAP Change Consultants]
Time: Oct 28, 2016 (03:30 PM - 04:15 PM)
Room: A601

Abstract 1 Title: Evaluating Grants for City and County Plans for Sustainable Communities
Presentation Abstract 1:

This is the story of what happened when two evaluators responded to a California RFP to do a statewide process and outcomes evaluation of five-year grants to cities and counties to develop related to sustainable communities. The authorizing legislation focused on greenhouse gas reduction to mitigate climate change. However, some grant assumptions were unreasonable. The “theory of change” about how plans would work did not match well with actual human behavior. Commonly accepted “evidence-based” approaches favored by urban and regional planners were questionable. Local community views of climate change varied and often differed from the client’s perspectives. The client asked for evidence related to the Governor’s speech at his 2015 inauguration whose concerns and targets were not even apparent in 2010 when the grants were awarded. The evaluators turned from Michael Quinn Patton’s “Utilization Focused” to his “Developmental Evaluation” approaches to adjust the evaluation analyses and reports.


Abstract 2 Title: Take Two: Evaluating City and County Plans for Sustainable Communities
Presentation Abstract 2:

Drawing on a report by international evaluator Rachel Kleinfeld, this presentation reflects on how it might have been more appropriate to have designed the evaluation from a systems perspective. It will show how systems evaluation rather than program evaluation approaches might be especially appropriate for evaluating short-term process and outcomes of programs attempting to influence processes such as climate change that have long-term impacts. It will discuss problems faced in designing evaluations when politically driven laws confuse program focus by trying to meet too many policy objectives in one set of grants. It will discuss approaches to client relationships when the client is focused on current policy concerns in a rapidly changing policy climate when grants were awarded years earlier when those policies and standards for “success” were different. Finally, it will discuss the relevance of time and timing to evaluation design and what currently constitutes “evidence-driven” theories of change.


Audience Level: Intermediate

Session Abstract: 

In 2015 Arlene Hopkins and Associates landed a statewide contract to evaluate 18 of 50 five-year Strategic Growth Council grants awarded to California cities and counties to plan sustainable communities with a focus on greenhouse gas reductions and other important public policy goals.  This session presents the evaluation approach used and summarizes findings about the urban and county plans that resulted, the process of creating them, and their short-term outcomes.  It then takes a critical “after action review” and reexamines the evaluation project, the client and evaluator assumptions and considers whether a different evaluation approach might have been more appropriate.  In particular, it will seek to show how taking a systems evaluation view of sustainability from the outset might have yielded additional insights and understanding of the plans and their outcomes with due attention to matters of time, timing of planning and evaluation work, local urban/county culture, history, and context.



For questions or concerns about your event registration, please contact registration@eval.org or 202-367-1173.

For questions about your account, membership status, or help logging in, please contact info@eval.org.



Cancellation Policy: Refunds less a $50 fee will be granted for requests received in writing prior to 11:59 PM EDT October 3, 2016. Email cancellation requests to registration@eval.org. Fax request to (202) 367-2173. All refunds are processed after the meeting. After October 3, 2016 all sales are final.