Evaluation 2018: Speaking Truth to Power

View Printable Version

Innovative Approaches to Qualitative Data Collection

Session Number: QM2
Track: Qualitative Methods
Session Type: TIG Multipaper
Session Chair: Hannah Betesh [Senior Associate - Social Policy Research Associates]
Presenter 1: Michelle Searle, Searl [Queen's University]
Presenter 2: Lisa Dyson [University of Auckland]
Presenter 3: Karla Stroud [Senior Result Consultant - Taylor Newberry Consulting]
Presentation 3 Additional Author: Laine Bourassa [Research Consultant - Taylor Newberry Consulting]
Time: Nov 01, 2018 (03:45 PM - 04:45 PM)
Room: Hilton - Center St. Meeting Room C

Abstract 1 Title: Using poetic forms of data collection and representation to speak truth to power in an educational evaluation
Presentation Abstract 1:

Poetry can have dimensions that are aesthetic, spiritual, and/or intellectual endeavours. Poetry may use language which is arranged to evoke emotions, present images drawn together to represent an experience, or offer the distilling of emotions laid bare. Ultimately, poetry can convey experiences and, as such, has limitless potential in our work as evaluators. This paper examines poetic voice as a way of accessing different kinds of programmatic experiences. Using empirical evidence gathered across multiple evaluation projects the involved collaborative poetic voice, I explore the purposes and processes of using this approach. Some of the successes and challenges are reported. Samples of raw data are offered as well as insight into the different approaches to analysis. Findings from the poetic methods used are framed for their connections to power, truth and voice.


Abstract 2 Title: Realist Interviewing for Assessing How Programs Work
Presentation Abstract 2:

Commissioners of evaluation are increasingly interested not only in whether programs or policies work, but how, why, for whom, and under what conditions they work. These questions are the domain of realist evaluation, an approach focused on developing and testing theories about how processes and mechanisms work in particular contexts to generate outcomes. Realist evaluators recommend a specific form of interview for identifying and analyzing these mechanisms. Using a real-world example, this session will describe how to develop initial theories about program processes, how to construct realist interview questions to test these theories, and how to use participants’ stories to help refine these theories. Drawing on research using realist interviews with New Zealand educators, the presenter will offer reflections on the researcher’s and participants’ responses to using this method. Attendees will learn about a valuable qualitative method for assessing whether and how programs and policies work.


Abstract 3 Title: Going Places: The Benefits of Observational Methods in Evaluation
Presentation Abstract 3:

Site visits are a valuable tool for evaluators, yet they are often not proactively included in evaluation designs. Observing where and how programs happen provides a more comprehensive understanding of program context and the extent to which program outcomes are achievable. Other methods that assess implementation (e.g., interviews, surveys, focus groups) carry limitations as they often fail to capture the nuances and variations present in program environments.  Triangulating standard process information with observational, site-based information can improve our assessment of validity assumptions and our theories of how programs work (or do not work).  Site visits give greater depth to the characteristics of the program, while also giving evaluators the opportunity to interact with program staff and participants in a more informal manner, contributing to improved buy-in and communication.  This session provides practical examples of how site visits can contribute to stronger evaluations and more contextually grounded theories of change.


Audience Level: All Audiences

Session Abstract (150 words): 

Innovative Approaches to Qualitative Data Collection



For questions or concerns about your event registration, please contact registration@eval.org or 202-367-1173.

For questions about your account, membership status, or help logging in, please contact info@eval.org.



Cancellation Policy: Refunds less a $50 fee will be granted for requests received in writing prior to 11:59 PM EDT October 8, 2018. Email cancellation requests to registration@eval.org. All refunds are processed after the meeting. After October 8, 2018 all sales are final. For Evaluation 2018, international attendees and presenters who encounter complications due to the international travel environment will have up to 30 days after the event to request a refund and submit appropriate documentation. No administrative fee will apply for the international requests.