Evaluation 2019: Paths to the Future of Evaluation: Contribution, Leadership, and Renewal

View Printable Version

Strengthening the Theory and Practice of Valuing in Evaluation

Session Number: 1369
Track: Theories of Evaluation
Session Type: Multipaper
Tags: values, valuing
Session Chair: Thomas A. Schwandt [Professor Emeritus - University of Illinois at Urbana Champaign]
Presenter 1: Rebecca Teasdale [Assistant Professor - University of Illinois at Chicago]
Presenter 2: Mathea Roorda [Evaluator - Allen and Clarke]
Presenter 3: Emily Frances Gates [Assistant Professor - Boston College]
Presenter 4: Charlotte Aldrich [Boston College]
Presentation 3 Additional Author: Eric Michael Williamson [Research Assistant - Boston College]
Time: Nov 14, 2019 (08:00 AM - 09:00 AM)
Room: CC M100 C

Abstract 1 Title: An Integrated Model of Criteria: Domains and Sources
Presentation Abstract 1:

Evaluative criteria define a “good” or “successful” evaluand and provide the basis for judgement of merit and worth (Julnes, 2012; Schwandt, 2015; Scriven, 2007); yet, criteria are often assumed and implicit in valuing (Greene et al., 2011). This paper presents an empirically-supported model that can be used to illuminate, innovate, and adapt criteria selection. The framework describes and integrates two aspects of criteria: domain and source. Domain identifies the focus or substance of a criterion, while source describes the individual(s) or document(s) from which it is drawn. Developed from a review of evaluation literature and analysis of evaluation reports, the model defines 11 criteria domains and nine sources and reveals the relationships among them. It can be used as a framework in research to illuminate which definitions of “success” inform evaluative conclusions and who advances those definitions and as a thinking tool to guide criteria selection in evaluation practice.

 


Abstract 2 Title: Bringing Values into Evaluation: A Tool for Practitioners
Presentation Abstract 2:

Values anchor an evaluation; they provide the basis against which evaluative judgments are made. Yet there are few empirically-tested tools to help practitioners identify and develop criteria of merit.  In this session, participants will be introduced to a framework intended to support evaluators to identify all relevant dimensions of value for a public sector program.  The framework draws on two approaches to valuing: prescriptive (normative ethical perspectives on ‘good’ and ‘bad’) and descriptive (understanding how a program is valued by different interest groups). A handbook for helping evaluators to use the framework was submitted to a validation process informed by the Western Michigan University’s Evaluation Checklists Project. The steps for using the framework will be outlined, and then findings from the initial field-testing will be discussed.


Abstract 3 Title: What Can Valuing Methods Tell Us About Valuing Theory?: A Review of Eight Methods Used in Educational Evaluation

Presentation Abstract 3:

Valuing methods are those uniquely suited to the evaluative task of bringing values and evidence together to inform judgments about the overall quality, performance, or successfulness of what’s being evaluated. Despite the necessity for many evaluators and evaluations to engage in valuing, there is little research to inform the theory and practice of valuing. To begin to address this gap, this paper critically reviews eight valuing methods used in educational evaluation to determine the ways in which these methods (mis)align with valuing theory based on Scriven’s four-step logic. The methods reviewed include the following: success case method, most significant change, geographical information systems, cost benefit analysis, social return on investment, and rubrics. This review provides practical guidance on how methods can be used to carry out valuing as well as theoretical implications for ways in which valuing theory falls short of capturing the dynamic and sociopolitical complexity of valuing.


Presentation 3 Other Authors: Sebastian Moncaleano
Boston College
Research Assistant
Abstract 4 Title: Value Arguments: Using Validity Theory to Inform How Evaluators Make Value Claims in Social Interventions
Presentation Abstract 4:

While claims of the value of social programs are crucial in many evaluations, little satisfactory guidance exists on how to make and defend value claims. This exploratory paper draws on Kane’s work (1992; 2009) on interpretation-dependent theories for validity argumentation to introduce Validity-Informed Valuing (VIV) to construct robust evaluative claims. Taking further inspiration from Kirkhart’s multicultural validity (2010), VIV is a holistic framework that creates space for contextual, historical, theoretical, and observed evidence towards support of defensible value statements. VIV is designed to elucidate and prioritize power differentials and contextual features in the valuing process. This paper presents the VIV framework and explores how it might be used through a hypothetical case evaluating a dual-language program in a public elementary school in the US. This paper presentation will invite attendees to reflect on  what constitutes robustness in making value claims and the potential utility of the VIV framework.


Audience Level: All Audiences

Session Abstract (150 words): 

Valuing refers to the processes by which value is determined. Reaching warranted conclusions about value is both a responsibility and an expertise claimed by professional evaluators. Yet, there is little practical guidance and limited research on how evaluators carry out (and should carry out) valuing. The logic of valuing provides a theoretical foundation for valuing processes. This logic involves identifying values as the basis of criteria; determining standards; collecting evidence; and synthesizing evidence to reach warranted conclusions about value. This multi-paper session aims to strengthen the theory and practice of valuing by examining two novel frameworks for identifying criteria; considering what constitutes a defensible value argument; and critically reviewing methods used for valuing. Together, the papers address how we might adapt valuing methods and practices to better address a plurality of values and be both methodologically and ethically defensible.

 



For questions or concerns about your event registration, please contact registration@eval.org or 202-367-1173.

For questions about your account, membership status, or help logging in, please contact info@eval.org.



Cancellation Policy: Refunds less a $50 fee will be granted for requests received in writing prior to 11:59 PM EDT October 11, 2019. Email cancellation requests to registration@eval.org. All refunds are processed after the meeting. After October 11, 2019 all sales are final. For Evaluation 2019, international attendees and presenters who encounter complications due to the international travel environment will have up to 30 days after the event to request a refund and submit appropriate documentation. No administrative fee will apply for approved international requests.