Evaluation 2016: Evaluation + Design

View Printable Version

Evaluating Complex Processes and the Implications for Evaluation Theory: Examples of Evaluating Sustainable Development, Programs, and Evaluation Efforts

Session Number: 2291
Track: Theories of Evaluation
Session Type: Panel
Tags: theory
Session Chair: George Julnes [Professor, School of Public and International Affairs - University of Baltimore]
Presenter 1: Andy Rowe [ARCeconomics]
Presenter 2: Stephen Porter [Univerity of the Witwatersrand]
Presenter 3: Debra J Rog [Associate Director - Westat]
Presenter 4: George Julnes [Professor, School of Public and International Affairs - University of Baltimore]
Presentation 2 Additional Author: Penelope Jane Hawkins [Head of Evaluation - DFID]
Time: Oct 28, 2016 (08:00 AM - 09:30 AM)
Room: L501

Abstract 1 Title: Connectivity is the key concept for bringing sustainability into evaluation
Presentation Abstract 1:

Evaluation generally operates within a siloed setting. Rooted in social science and performance management, evaluation has been slow to recognize and address the growing urgency to bring sustainability into core evaluation thinking and practice. This is doubly limiting; evaluation has fallen seriously behind in its ability to address sustainability and as a result we have not grasped the opportunity to bring into evaluation important and useful methodological concepts from ecology and other natural and physical sciences. Connectivity is conceptually central to ecology but has had limited play in evaluation. Sustainability-incorporating evaluation requires vigorous application of connectivity and will shift the focus of evaluation towards public interest goals and away from accepting partitioned discrete performance units as our evaluands. This paper will show how game-changing connectivity is and outline the directions that evaluation and evaluators need to take to gain competency in evaluating sustainability and sustainable development.


Abstract 2 Title: After the Champion: Focusing on Public Value to Build Sustainable Evaluation Systems
Presentation Abstract 2:

Actions to develop evaluation systems often focus upon local champions. A champion provides a short-term boost, but alone is not enough to build a sustainable system. This paper argues that to build a sustainable institutional evaluation system there needs to be a focus on the delivery of public value. The conceptual framework of public value (Moore, 1995) helps focus the attention of those developing evaluations systems on the institutional sources of demand for evaluation by requiring an articulation of the relationship between the evaluation system and the outcomes that are valued within the political environment with a recognition that the evaluation system needs to build a mandate amongst a broad range of stakeholders. This paper demonstrates the helpful application of public value concepts through two case studies from the UK’s Department for International Development and South Africa’s Department for Performance Monitoring and Evaluation.


Abstract 3 Title: Adapting Evaluations to Help Systems and Initiatives Adapt and Sustain
Presentation Abstract 3:

Increasingly, evaluators are asked to evaluate complex initiatives rather than single programs. These initiatives, such as systems change efforts, rarely lend themselves to traditional outcome designs. The presenter will share lessons learned in evaluation design and implementation from evaluations of three complex initiatives: multipronged initiative to reform homeless systems in three counties, multi-sector collaboratives designed to create movements to address adverse childhood effects in 14 communities, and a broad foundation portfolio of over 200 grants aimed at fostering healthy children. Although each of these initiatives has a theory of change, the work often “goes off script” to respond to challenges and new developments. Outcomes for each of these efforts are important to measure, yet equally important is assessing the system’s or initiative’s capacity to continue to change. The presenter will highlight examples of how the ability to adapt, especially guided by data and feedback, may be a key to sustainability.


Abstract 4 Title: Evaluation Models in Support of Managing Complex Processes
Presentation Abstract 4:

There has been greater attention in evaluation theory on the role of underlying metaphors in guiding what we assess and our resultant interpretations. For example, Dahler-Larsen (2012) highlighted the implications of the rational, learning, and institutional systems models for evaluation, arguing that the simpler rational and learning models function as blinders, discouraging evaluators from recognizing more complex dynamics in programs being evaluated. Similarly, increasing attention to sustainability and systems dynamics highlights the need for more complex process models when judging value in evaluation. These examples illustrate the importance of program personnel managing processes and so call for greater dialogue about the implications of more complex process models for evaluation theory and methods. Summarizing the points made in the other presentations in this session, this presentation will offer a framework for a process orientation to evaluation that serves to promote the needed dialogue around evaluation in support of managing complex processes.


Audience Level: All Audiences

Session Abstract: 

The design of programs is partly a matter of resources and other structural elements but is more a matter of organizational processes designed to influence the processes of others—educational programs designed to support the cognitive and intellectual development of students, social service programs designed to help homeless individuals attain stability and autonomy, and international development projects designed to promote self-sufficiency in targeted populations. The underlying claim of this session is that a process orientation requires a change in our theories of evaluation, a change that is emerging in the areas of system dynamics, complexity, sustainability, and evaluating overarching strategies rather than discrete projects and programs.  This session addresses, and brings together, different aspects of this process orientation, including the importance of ecological “connectivity” for evaluating sustainable development, the centrality of “public value” in building sustainable evaluation efforts, and the role of strategic adaptation in managing sustainable systems. 



For questions or concerns about your event registration, please contact registration@eval.org or 202-367-1173.

For questions about your account, membership status, or help logging in, please contact info@eval.org.



Cancellation Policy: Refunds less a $50 fee will be granted for requests received in writing prior to 11:59 PM EDT October 3, 2016. Email cancellation requests to registration@eval.org. Fax request to (202) 367-2173. All refunds are processed after the meeting. After October 3, 2016 all sales are final.