Evaluation 2015: Exemplary Evaluations in a Multicultural World

View Printable Version

Quality in Evaluation- Who Defines, Who Determines and Who Cares?

Session Number: 1199
Track: Theories of Evaluation
Session Type: Panel
Tags: equitable evaluation, evaluation quality, evaluation use, metaevaluation
Session Chair: Leslie Cooksy [Sierra Health Foundation]
Discussant: Thomas Kelly, Jr. [Hawai`i Community Foundation]
Presenter 1: Thomas Schwandt [University of Illinois at Urbana Champaign]
Presenter 2: Melvin M Mark [Penn State]
Presenter 3: Valerie Jean Caracelli [U.S. Government Accountability Offi]
Presenter 4: Jara Dean-Coffey, MPH [jdcPartnerships]
Presenter 5: Leslie Cooksy [Evaluation Director - Sierra Health Foundation]
Time: Nov 11, 2015 (04:30 PM - 06:00 PM)
Room: Comiskey

Presentation Abstract 1: Tom Schwandt, 2002 recipient of the Paul F. Lazarsfeld Evaluation Theory award, has written extensively about quality in evaluation, going back as far as 1988 with Linking Auditing and Metaevaluation: Enhancing the Quality of Applied Research (co-authored with Edward Halpern) to the more recent “On Discerning Quality in Evaluation,” written with Robert Stake and published in the Sage Handbook of Evaluation (Shaw, Mark, & Greene, 2011). In addition to his expertise, Schwandt brings experience with evaluation in international venues to the panel. His role is to open the panel with some observations about how we think about quality in evaluation.
Presentation Abstract 2: Mel Mark’s comments will focus on evaluation quality from the perspective of scientific criteria. Validity -- what it means and how we apply the construct in our work – has been a focus of much of his work. For example, he recently co-edited (with Chen and Donaldson) a New Directions for Evaluation issue on validity typologies for outcome evaluation. Mark was recognized in 2012 with AEA’s Paul F. Lazarsfeld Evaluation Theory award in part for his contributions in this realm. His presentation will also emphasize how considerations other than traditional technical issues are involved in judgments of scientific quality.
Presentation Abstract 3: Valerie Caracelli has been a leader in research and thinking about evaluation use. She co-edited a New Directions in Evaluation issue on evaluation use (with Hallie Preskill) and led the Evaluation Use TIG. In addition, her experience in the US Government Accountability Office gives her insight into how context can influence what it means to make a claim that an evaluation meets quality standards. Her observations will address the idea of evaluation use as a criterion of quality, including considerations of whose use counts and how use is assessed.
Presentation Abstract 4: Jara Dean-Coffey recently published an article about integrating cultural competence and equity for equitable evaluation (Foundation Review, 2014). As an independent consultant, she has dedicated her work to organizational development and building the evaluative capacity and culture of client partners. As a member of the panel, Jara will provide the perspective of a new way of thinking about quality and its linkage to AEA’s public commitment to culturally competent evaluation.
Presentation Abstract 5: Leslie Cooksy has written about evaluation quality and metaevaluation from the perspective of a meta-evaluator and researcher of metaevaluation practice. As AEA president in 2010, she chose evaluation quality as the conference theme. Her perspective is based on experience in multiple settings—government, academic, and philanthropic. She also has experience in metaevaluation of evaluations of international programs. Her comments will focus on practical implications of different ways of thinking about quality in evaluation, including the “who cares” question.
Audience Level: Intermediate

Session Abstract: 

Responding to the conference theme of Exemplary Evaluation, this panel addresses the nature and dimensions of quality in evaluation. Ideas about what constitutes and who defines “quality” have evolved over time, as have views about how quality is achieved and assessed. The evolution and its implications for current and future considerations of quality warrant discussion. Following brief presentations from panelists, the session will consist of a moderated discussion among the presenters and with the audience. Tom Kelly will serve as moderator. Thomas Schwandt will open by providing context on the variety of ways of thinking about quality. The next three presenters will discuss quality in evaluation from three perspectives: scientific quality (Melvin Mark), utilization (Valerie Caracelli) and equity as an indicator of quality (Jara Dean-Coffey). Leslie Cooksy will offer comments on the implications of these perspectives on quality for practice.

For questions or concerns about your event registration, please contact registration@eval.org or 202-367-1173.

For questions about your account, membership status, or help logging in, please contact info@eval.org.

Cancellation Policy: Refunds less a $50 fee will be granted for requests received in writing prior to 11:59 PM EDT October 6, 2015. Email cancellation requests to registration@eval.org. Fax request to (202) 367-2173. All refunds are processed after the meeting. After October 6, 2015 all sales are final.