Evaluation 2015: Exemplary Evaluations in a Multicultural World

View Printable Version

Who Uses Evaluation Findings, When, and for What Purpose? Evaluation Use from Multiple Perspectives

Session Number: EU4
Track: Evaluation Use
Session Type: Multipaper
Session Chair: Erin Burr [ORAU]
Presenter 1: Derek Alan Wilson [Faculty of Medicine, University of British Columbia]
Presenter 2: Karol Olejniczak [EUROREG, University of Warsaw]
Presenter 3: Helen Hsu, MSc CE [Faculty of Medicine, University of British Columbia]
Presenter 4: Suppiramaniam Nanthikesan, Sr Evaluation Advisor [United Nations]
Presentation 1 Additional Author: Chris Y. Lovato [Professor - University of British Columbia]
Presentation 1 Additional Author: Tamiza Abji [Evaluation Studies Unit]
Presentation 2 Additional Author: Tomasz Kupiec [EGO - Evaluation for Government Organizations S.C.]
Presentation 3 Additional Author: Derek Alan Wilson [Director, Evaluation Studies Unit - Faculty of Medicine, University of British Columbia]
Presentation 3 Additional Author: Chris Y. Lovato [Professor - University of British Columbia]
Presentation 4 Additional Author: Juha Uitto [Director - Global Environment Facility]
Time: Nov 12, 2015 (03:00 PM - 04:30 PM)
Room: Jackson Boardroom

Abstract 1 Title: Strengthening the CQI Loop: Strategies for Promoting Evaluation Use
Presentation Abstract 1: In organizations building a quality improvement culture, the evaluation of programs is a continuous process which does not end when the final report is delivered. Rather, evaluation results are actively used by leadership to guide planning and decision-making, a key phase of the program improvement cycle. At the University of British Columbia's Faculty of Medicine, a quality improvement process has been established whereby program evaluation results and recommendations are routinely brought forward to leadership for action. The monitoring and oversight elements of this process have recently been strengthened, through a series of structural and procedural enhancements, to ensure recommendations are actioned and the "loop is closed". This presentation will outline the strategies developed and how they are facilitating use of evaluation results, including positive gains, challenges, and our insights on lessons learned.
Abstract 2 Title: Evaluation as a Source of Knowledge for Public Programs - the Users' Perspective
Presentation Abstract 2: Public decision-makers need different types of knowledge to run programs successfully. These are: knowledge about the context in which intervention is implemented, operational knowledge (technical know-how), knowledge about the effects, and explanatory knowledge about causal mechanisms. Knowledge comes from different sources. Evaluation studies are just one of them.
This article takes the perspective of knowledge users. It explores to what extent evaluation is a useful source of knowledge for public decision-makers. Findings are based on an extensive study of Polish Cohesion Policy system that included a survey with 945 program managers (n=945), followed by interviews with key policy actors (n=78) and three case studies in regional administration. The article concludes that: (a) evaluation reports are mainly a source of knowledge on effects and mechanisms, however (b) utility of evaluation studies, in comparison to other sources of knowledge, is limited, and (c) "effects" are shallowly interpreted as smooth money spending, not socio-economic change.
Abstract 3 Title: From Reactive to Proactive: Using a Developmental Approach to Guide Program Planning
Presentation Abstract 3: Evaluation is often not engaged at the 'front end' to inform and guide program planning, rather it is included at the conclusion of a program cycle to reflect on program delivery, and/or to demonstrate outcome or impact. In such instances, stakeholders 'react' to the evaluation results and recommendations by making improvements for the next iteration of program delivery. This presentation will describe how the developmental evaluation approach was used in the University of British Columbia's medical school to support 'proactive' planning and decision-making in a multi-year curriculum renewal initiative. The presenter will outline how the evaluation evolved from end-of-phase feedback to ongoing forward feeding, and how this approach enhanced the level of stakeholder engagement and understanding of evaluation. Key learnings will be shared on how stakeholder priorities were used to develop a guiding framework to ensure that different parts of the evaluation inquiry contributed to the overall initiative.
Abstract 4 Title: Walking with two legs: Upward and Downward Accountability in Evaluations
Presentation Abstract 4: Many evaluations are driven by the interests of the agencies and donors who wish to assess the effectiveness and efficiency of their interventions. In those cases, accountability is directed upward to the agency and its funders, frequently the taxpayers in the donor country. But what about the people the interventions are intended to benefit? Downward accountability to claim-holders (beneficiaries) is equally if not more important. In the evaluation field, the focus on downward accountability has been weak. In an ideal world, the methodologies of evaluations commissioned by the duty-bearers would be no different from those commissioned by or on behalf of claim-holders. In that world, both types of evaluations would assess the full range of consequences (intended and unintended) that interventions had for claim-holders. These would be goal-free evaluations (Scriven and Patton 1972). This paper discusses how evaluations can promote downward accountability and help identify unintended consequences of interventions.
Audience Level: None

Session Abstract: 

Who uses evaluation findings, when, and for what purpose? Evaluation use from multiple perspectives.

For questions or concerns about your event registration, please contact registration@eval.org or 202-367-1173.

For questions about your account, membership status, or help logging in, please contact info@eval.org.

Cancellation Policy: Refunds less a $50 fee will be granted for requests received in writing prior to 11:59 PM EDT October 6, 2015. Email cancellation requests to registration@eval.org. Fax request to (202) 367-2173. All refunds are processed after the meeting. After October 6, 2015 all sales are final.