Evaluation 2015: Exemplary Evaluations in a Multicultural World

View Printable Version

Evaluation in Complex Ecologies of our Multicultural World

Session Number: 2930
Track: Research on Evaluation
Session Type: Panel
Tags: complexity, equity
Session Chair: Rodney K Hopson [George Mason University]
Discussant: Michael Quinn Patton [Utilization-Focused Evaluation]
Presenter 1: Donna M Mertens [Independent Consultant]
Presenter 2: Robin Lin Miller [Michigan State University]
Presenter 3: Andy Rowe [ARCeconomics Ltd]
Presenter 4: Jill Chouinard [University of North Carolina-Greensboro]
Presenter 5: Oran Hesterman [Fair Food Network]
Presentation 1 Additional Author: Arlinda Boland [Gallaudet University]
Presentation 5 Additional Author: Ricardo A Millett [Community Science]
Time: Nov 12, 2015 (04:45 PM - 06:15 PM)
Room: Burnham

Abstract 1 Title: Complex Ecology in International Development Evaluation Focusing on Women and People with Disabilities
Presentation Abstract 1: The international development community uses seven Millennium Development Goals (MDGs) as a basis for measuring progress of initiatives intended to reduce poverty. Women are specifically mentioned in two of the MDGs, one related to gender equality in education and the other related to improvement of maternal health. People with disabilities (PWD) are not specifically mentioned in any of the MDGs. This raises concerns related to the extent that the MDGs and rights laid out in the UN conventions and declarations related to women and people with disabilities will be realized if countries attempt to meet the MDGs without conscious inclusion of these constituencies or their advocates in program development and evaluation. UNICEF and UN Women (2013) issued a report that recognized the need to address structural inequalities for these two groups because they represent the poorest and most marginalized in many countries. The UN organizations argue for the need for transformative measures to reduce inequalities and to reach economic and social goals that will allow these people to realize their rights. In this paper, we explore case studies that illustrate how evaluators can address challenges that arise in conducting relevant and responsible evaluations with women and PWD in complex ecological systems, including possible corruption at high levels of government and lack of action on the part of policy makers or power brokers. Evaluators who work with these populations need to be aware of the challenges due to local customs, values and beliefs on the role of women and PWD in the society.
Abstract 2 Title: Evaluating HIV Evidence-Based Practices in Complex, Community Contexts
Presentation Abstract 2: In the first part of the paper, I will briefly describe current trends in the field of HIV, including biomedical advances toward treatment as prevention, community movements toward human rights and structural change approaches, and the broader national and international push toward evidence-based practice. The evidence-based landscape will be contrasted against the context of AIDS-impacted communities in terms of the symbolic meanings of the disease, the material context in which AIDS is most severe, and the relational context in which AIDS-affected groups exist within society. I will use these two portraits, one of the evidence-driven movement and the other of AIDS-affected communities in context, to underscore the need for evaluators to attend to population-setting-program fit in their work because the contexts for developing evidence-based approaches and prescriptions may differ substantially from the contexts in which these approaches are implemented. Supported by emergent realist principles, I will argue that the relevance of evaluations in the context of community-based AIDS efforts depends on how well the contextual dynamics of fit are addressed in study design, implementation, and dissemination. In the second part of the paper, I will address the unique relationship issues that arise for evaluators working in AIDS-related community organizations because of the history and mission of these groups and the nature of the populations they serve. I will describe principles of interpersonal reconnaissance and relationship development drawn from theories in evaluation and also from theories in community psychology that may help evaluators understand how to cultivate genuine, productive relationships on which to build an evaluation effort. In the final part of the paper, I will summarize by addressing the responsibility of evaluators to document what these kinds of organizations value in programming, what can be learned from community-based efforts to innovate their own solutions to HIV, and to tell the story of what happens when these organizations implement evidence-based practices that have been imported to their communities.
Abstract 3 Title: Silo Reinforcing or Silo Busting – Will the Real Evaluation Step Forward?
Presentation Abstract 3: The author will draw from over thirty years working as an economist and evaluator and in this paper will use sustainable development examples. The paper will first provide an overview of how ecological thinking with an emphasis on connectivity can prove challenging for evaluation. It will then describe principles for evaluating sustainable development interventions, addressing coupled human and natural systems with the challenges this brings such as having to incorporate the different temporal and spatial scales of the two systems. Also coupled systems have a greater number and usually more complex moving parts than singles systems and this can present serious challenges to usual evaluation methods such as experimental designs. Contemporary evaluation practices for sustainable development will then be described using the example of a community based effort building resilient and sustainable coastal communities in a number of Pacific island countries. Two evaluation scenarios will be developed, one using ecological thinking, the other applying business as usual evaluation approaches. Comparing the scenarios in terms of key elements such as their reach, what behaviours and outcomes they consider, potential applicability of evaluation methods and the likely salience of the evaluation to stakeholders will highlight what ecological thinking can bring to evaluation. The discussion of the two evaluation scenarios will also compare the potential differences in advice provided and how incorporating ecological thinking shifts evaluation from a partition-reinforcing to a partition-busting contributor to governance.
Abstract 4 Title: Mapping the Boundaries and Borders of a Restorative Justice Program for Sex Offenders Through Evaluation
Presentation Abstract 4: In an attempt to name and restore voice to the demonized ‘other’, as well as to deconstruct and critique the notion of mythologized sexual predator, the approach adopted for this evaluation is based on the principles of restorative justice, where the focus shifts to capturing the needs, perspectives, and experiences of both the community and the sex offenders themselves. This collaborative approach is consistent with the view of evaluation as advocacy (Greene, 1997), highlighting the centrality of values (Hall, Ahn & Greene, 2012), as the focus moves from quantitative measures of program success to capturing the voices and perspectives of those who continue to be marginalized and excluded from the conversation. As Stake (1975) has argued, “we need a reporting procedure for facilitating vicarious experience. We need to portray complexity. We need to convey holistic impression, the mood, even the mystery of the experience” (p. 23). The focus of this paper will be on using evaluation for creating new knowledge claims and new ways of looking at the world by bringing together primary program stakeholders as a way to restore voice and generate understanding between sex offenders and community program stakeholders. To convey the unique experiences of diverse program stakeholders, the use of narrative as an approach to evaluation will enable and privilege an understanding of both local program dynamics and the personalized experiences of participants and community members, what is really a restorative justice approach to evaluation.
Abstract 5 Title: Creating and Evaluating a Sustainable and Equitable Food System
Presentation Abstract 5: Creating a food system that is healthy, green, fair, and affordable requires modification in individual behavior and in institutional and public policy. The food system is by nature a complex ecology, with interacting biological, social, economic, and political currents. Solving food systems problems (e.g. obesity, pollution, soil loss, hunger) one at a time is unlikely to create lasting change in the system due to unintended and unanticipated consequences. To create sustainable and equitable change requires a system redesign – which in this, as in many systems, means changing public policy.
The big questions then are: In what direction should policy change? And how do we gather and communicate compelling evidence to policy decision makers at every level? Evaluation has a key role to play in answering both of these questions. In this paper, we will suggest how, in principle, philanthropy, the nonprofit sector, and evaluation have key roles to play in food system redesign[FM1:We note that the foundation community has and hopefully will continue to support ‘incentive matches’ and related program innovations that motivate SNAP beneficiaries to purchase and consume more fresh fruits and vegetable. We realize, however that we should not and cannot rely on their funding support for ever. The problem of food insecurity and access is too large and the population that needs to be addressed is much larger than those current being reached. At the time we launched this study we hoped to get the federal government attention and support. This is where program evaluation methods, sufficiently capable of addressing variations in our programs and to discerning common outcomes came into play in our in to play in promoting system redesign pathways.] This paper presents an example of how foundation support of incentive programs, and appropriately designed evaluation can provide clarity in addressing complex public policy issues in the real world and in real time. The example we use is Fair Food Network’s Double Up Food Bucks program.
Audience Level: Intermediate
Other Information: Additional Discussant:

Fiona Cram, Katao, Ltd

Session Abstract: 

This presentation captures the efforts of leading thinkers and problem-solvers in the evaluation field. Each contributor brings insights from their practical and personal experiences of grappling with how to expand the understanding and application of evaluation contribute to the development of solutions to the increasingly complex social and economic issues facing communities. This panel canvasses diverse problem-solving contexts that represent the frontiers of evaluation. The papers are bound by a unifying theme of how a focus on relationships, relevance and responsibility ensures that contributors have a better understanding of the complex ecologies in which they work.  This understanding then flows into their evaluation questions, practice, theory, and design.  Together their experiences and expertise provide policy makers, practitioners, and scholars with an array of theories, strategies, and approaches that enable evaluation to contribute the framing of effective solutions to the problems small and large complex ecologies with which our communities are grapping. 

For questions or concerns about your event registration, please contact registration@eval.org or 202-367-1173.

For questions about your account, membership status, or help logging in, please contact info@eval.org.

Cancellation Policy: Refunds less a $50 fee will be granted for requests received in writing prior to 11:59 PM EDT October 6, 2015. Email cancellation requests to registration@eval.org. Fax request to (202) 367-2173. All refunds are processed after the meeting. After October 6, 2015 all sales are final.