Evaluation 2015: Exemplary Evaluations in a Multicultural World

View Printable Version

Project Evaluation from the Evaluand’s Perspective—Utilization-Driven Evaluation Preparation

Session Number: 1298
Track: Evaluation Use
Session Type: Multipaper
Tags: developing countries, ethics, Evaluation practice, evaluation preparation, organizational learning, USAID
Session Chair: Michael Cohen [Management Sciences for Health]
Presenter 1: Michael Cohen [Management Sciences for Health]
Presenter 2: Tobey Busch [Management Sciences for Health]
Presenter 3: Alice Willard [Independent Consultant]
Presenter 4: Ruth Musila [Management Sciences for Health]
Presentation 1 Additional Author: Tobey Busch [M&E Advisor - Management Sciences for Health]
Presentation 2 Additional Author: Michael Cohen [M&E Director - Management Sciences for Health]
Time: Nov 13, 2015 (01:45 PM - 03:15 PM)
Room: Picasso

Abstract 1 Title: Setting the Stage for Successful Evaluation - From the Evaluand’s Perspective
Presentation Abstract 1: Defining a “successful evaluation” is often reminiscent of US Supreme Court Justice Stewart’s legendary threshold test: “I know it when I see it”. Similarly, an evaluation’s success can be defined differently by various stakeholders. From the perspective of the organization being assessed by an external team, success is often gauged by simply having gotten through the process unscathed after “passing” what feels like arbitrary trials. Reaping the full benefit of a program evaluation requires comprehensive and proactive planning by the evaluands.
To illustrate, we will present the recent example of a global health systems strengthening project funded by USAID. Preparations, which began in earnest more than a year prior to the evaluation, included all levels of the organization – senior management through junior technical staff, headquarters, and 18 country offices. This paper will elaborate general approaches, advocacy, and management issues.

Abstract 2 Title: Tools and Templates for Utilization-Driven Evaluation Preparation
Presentation Abstract 2: Utilization-driven evaluation preparation is an active process that is intended to encourage collaborative evaluation resulting in more useful outcomes for both the evaluand and other key stakeholders. To help achieve this, we have built on previous guidance by developing a set of tools and activities that can be adapted to a wide range of programs. Our motivation was to enable a global health systems strengthening program evaluation—working across multiple locations and contexts—to contribute to and benefit from a program evaluation. We needed a standard process by which internal stakeholders could prepare to ensure that evaluators could process complex and diverse program information. The program developed and used these tools to both inform the evaluators and assess our own results and evaluation readiness. By using the tools we will present here, staff were better prepared, and the evaluators had easier access to relevant evidence.
Abstract 3 Title: Reading from a Script
Presentation Abstract 3: Two main challenges with preparing staff for an evaluation are teaching them to 1) present key messages without sounding rehearsed and 2) not feel threatened or defensive during the interview. One additional factor that affected staff’s ability to respond was the tremendous diversity of both headquarters and field staff: age, gender, and cultural norms changed the landscape for both the interviewer and the interviewees. The MSH M&E team (and a consultant) worked with staff directly through short lectures, role plays, webinars, and FAQS of interview do’s and don’ts to help allay nerves over the interview process, as well as give staff a better understanding of their rights as evaluands. This gave the staff a degree of empowerment to participate in the interview, rather than being controlled or controlling. This presentation introduces an evolving list of evaluand rights and strategies.
Abstract 4 Title: Evaluation Preparation from the Field Offices' Perspective—Completing a Narrative
Presentation Abstract 4: A single evaluand may, in fact, be a diverse set of groups defined by objective, geography, or management. Yet the evaluation preparation must be consistent so as to shape a coherent and cogent narrative. The collation of documents and their rational organization is only a first, albeit crucial, step toward preparation. Given the complexities of implementing a global pharmaceutical systems strengthening program funded by diverse US government bureaus with different health priorities, it was imperative that country teams reflected on key achievements, vis-à-vis program objectives, and took stock of implementation challenges and lessons learned. We developed a standard approach to facilitate this, within unique country contexts, and simultaneously relate outcomes to the global program’s objectives and funders’ goals. This process not only prepared country programs to articulate evidence, but also provided an opportunity to derive comprehensive information to aid strategic planning, as we head toward the program’s end.
Audience Level: Advanced

Session Abstract: 

A well-prepared evaluand benefits all stakeholders. Yet most program staff are provided scant guidance on how to achieve this. We present a recent case study, a complicated global pharmaceutical systems strengthening program, to illustrate novel tools and techniques to prepare both people and products for an evaluation. The session incorporates a preliminary examination of the evaluand’s rights in an evaluation. Processes included the usual steps of organizing materials, logistics, and project bibliographies. Additional innovations included preparing staff for evaluators and interviews and helping staff consider the uses and utility of evaluation outcomes. The panel presents a series of four papers from both the organization’s staff and a consultant who worked with field and headquarters staff. One challenge was balancing requirements: preparing versus “over-rehearsing” staff, presenting confidently versus the appearance of concealing negative findings, and minimizing anxiety without sacrificing quality or honesty.

For questions or concerns about your event registration, please contact registration@eval.org or 202-367-1173.

For questions about your account, membership status, or help logging in, please contact info@eval.org.

Cancellation Policy: Refunds less a $50 fee will be granted for requests received in writing prior to 11:59 PM EDT October 6, 2015. Email cancellation requests to registration@eval.org. Fax request to (202) 367-2173. All refunds are processed after the meeting. After October 6, 2015 all sales are final.