Evaluation 2017: From Learning to Action

View Printable Version

Innovative Participatory and Empowering Evaluation Methods to Enhance Learning and Use

Session Number: 1999
Track: Collaborative, Participatory & Empowerment Evaluation
Session Type: Panel
Tags: appreciative inquiry, collaborative evaluation, empowerment evaluation, participatory evaluation, Rubrics Methodology
Session Chair: Thomaz Kauark Chianca [Managing Partner - COMEA Relevant Evaluations]
Discussant: David M Fetterman [President - Fetterman & Associates]
Presenter 1: Tessie Tzavaras Catsambas [President and CFO - EnCompass LLC]
Presenter 2: Mônica Rodrigues Dias Pinto [Institutional Development Manager - FRM - Roberto Marinho Foundation]
Presenter 3: Rita O'Sullivan [Univ of NC EvAP]
Presenter 4: Thomaz Kauark Chianca [Managing Partner - COMEA Relevant Evaluations]
Presentation 2 Additional Author: Carolina Proietti Imura [Consultant - Herkenhoff & Prates]
Presentation 2 Additional Author: Rosalina Maria Soares [Coordinator of Evaluation - Roberto Marinho Foundation]
Presentation 3 Additional Author: Fabiola Salas Villalobos [University of North Carolina Chapel Hill ]
Time: Nov 11, 2017 (08:00 AM - 09:00 AM)
Room: Thurgood Marshall North

Abstract 1 Title: Facilitating participation for learning for a UN evaluation
Presentation Abstract 1:

In his “Reflections on Learning,” Deputy Director Daniel Seymour called this evaluation “the best corporate evaluation UN Women has conducted.” (TRANSFORM Magazine https://issuu.com/ieounwomen/docs/unw-transform-issue9, p.26) When the evaluation was presented to the Executive Board, 23 countries spoke and appreciated the usefulness of this evaluation. This presentation will trace how participation was structured for this evaluation and the key roles that enabled meaningful participation and resulted in learning from the evaluation. Another important contributing factor was the choice of three method/prisms for the evaluation: Appreciative Inquiry, Outcome Mapping, and Quality Improvement (which has many links to Empowerment Evaluation). We will focus on the intentionality and strategy for participation, and the navigation of unexpected developments and challenges. We will also cover the role participation and learning played in light of the evaluation methods used.

Abstract 2 Title: Use of playful methods to assess children and of systemic resources to analyze and communicate results
Presentation Abstract 2:

Evaluating the performance of an environmental education project in a school context, especially the acquisition of socio-environmental values by children from 7 to 9 years of age, is not an easy task. Among other challenges, the evaluators need to find methodological strategies to better understand and value the children’s universe as well as gain credibility from the organizations proposing/supporting the project and the ones implementing it. This paper highlights the use of playful methods that, in addition to capturing the necessary information for the evaluation, enabled project implementers to reflect on their practices, stimulating the development of new pedagogical strategies. We will also discuss the evaluators’ capacity to analyze and organize the outcomes, including the children's perspectives as part of the evaluative evidence, and using an analytical and systemic approach that includes textual and visual resources.

Presentation 2 Other Authors: Maíra Mascarenhas (maira.mascarenhas@frm.org.br)
Fundação Roberto Marinho
Research and Evaluation Analyst
Abstract 3 Title: Collaborative evaluation strategies that enhance the usefulness of evaluations for stakeholders: Evaluation of three dual language program in a school district
Presentation Abstract 3:

Public school district program coordinators often lack expertise, opportunity, perspective, or just plain time to use evaluation. This case study shares how using collaborative techniques in evaluating dual language programs in three elementary schools proved useful to program stakeholders. Initially, evaluators met with the Program Coordinator to review the program and identify key evaluation questions. Evaluators conducted focus groups at each of the three schools in the primary language of the teachers (English, Spanish and Mandarin). Member checking was used to verify information gathered and to build trust with teachers. Multi-year achievement scores were used to identify trends between regular and dual language students. Meetings were held at the schools to share the information gathered and, only after that, were teachers surveyed about their opinions regarding the curriculum and its improvement. In this manner, the usefulness of the evaluation to stakeholders was enhanced.

Presentation 3 Other Authors: Wenyang Sun (ritao@unc.edu)
Evaluation, Assessment, & Policy Connections
School of Education
University of North Carolina – Chapel Hill
Abstract 4 Title: Using rubrics methodology approach to ensure validity and usefulness of evaluations: The participatory evaluation of an early childhood program in Brazil
Presentation Abstract 4:

It is common to evaluations to provide findings with abundant descriptive data on several indicators, leaving stakeholders without a clear sense of whether their program was a great success or a complete failure. In contrast, rubrics methodology has, as an important feature, the power to combine complex quantitative and qualitative data, using a consistent analytical framework, that leads to clear evaluative conclusions about the quality and importance of whatever is being evaluated. The intelligent involvement of different key stakeholders in specific moments of the evaluation is essential to rubrics methodology. This paper presents a case of a highly participatory evaluation of an early childhood program in several municipalities in southern and northern states of Brazil using rubrics methodology. The presentation will focus mainly on the strategies to involve the different stakeholders in the design, implementation, reaching evaluative conclusions, establishment of causal relationships, drawing lessons and planning for action.

Theme: Learning to Enhance Evaluation
Audience Level: All Audiences

Session Abstract (150 words): 

Stakeholder involvement approaches to evaluations have been widely accepted as good practice within the evaluation profession to ensure evaluation validity, credibility, learning and use. However, putting it in practice in challenging situations or contexts is not straightforward and demands great creativity, flexibility and commitment by evaluators and evaluation commissioners. In this session, internationally recognized professionals with extensive experience in practicing participatory, collaborative and empowerment evaluations will share their reflections about some of the innovative strategies they have adopted when facing especially challenging contexts. Four cases will serve as the basis to inspire discussions with attendees, including the evaluations of: 1. The global strategy to maximize a UN agency’s capacity to deliver on its mandate; 2. An environmental education project in Brazilian elementary schools; 3. Three dual language program in an American school district; and 4. An early childhood development program in several municipalities in Brazil.

For questions or concerns about your event registration, please contact registration@eval.org or 202-367-1173.

For questions about your account, membership status, or help logging in, please contact info@eval.org.

Cancellation Policy: Refunds less a $50 fee will be granted for requests received in writing prior to 11:59 PM EDT October 16, 2017. Email cancellation requests to registration@eval.org. Fax request to (202) 367-2173. All refunds are processed after the meeting. After October 16, 2017 all sales are final. For Evaluation 2017, international attendees and presenters who encounter complications due to the international travel environment will have up to 30 days after the event to request a refund and submit appropriate documentation. No administrative fee will apply for the international requests. The $50 fee will be waived for registrants who planned to travel into the US and experienced international travel issues.