Evaluator Roles in Methodological Decision-Making: Reflections from New Evaluators

Session Number: 2655
Track: Graduate Student and New Evaluators
Session Type: Poster
Tags: Decision making, Evaluation Design, evaluator's role, methodology, novice evaluator
First Author or Discussion Group Leader: Julianne Zemaitis [Graduate Student/Assistant - University of North Carolina at Greensboro]
Second Author or Discussion Group Leader: Meltem Yumsek [Graduate Student - University of North Carolina at Greensboro]
Time: Oct 26, 2016 (07:00 PM - 09:00 PM)
Room: Poster 89

Audience Level: Beginner

Session Abstract: 

Previous research suggests that quantitative methodologies (e.g., randomized-control trials) are often more highly valued in research and evaluation due to their objectivist nature and the assumption that an nomothetic approach is best to able to capture definitive outcomes and impacts of an intervention. However, in accordance with AEA’s Guiding Principles, methodological preferences should not be enough to drive methodological decisions in an evaluation design. Evaluators are required to address the appropriateness of methodologies within the context of the program and in relation to the evaluation questions while communicating any and all limitations of the methodologies and potential findings to stakeholders. Adherence to professional standards while attempting to balance stakeholder wants and needs is necessary, albeit difficult, for novice evaluators to navigate tactfully in practice. Besides, power issues might arise in real contexts with stakeholders that might further burden novice evaluators in this situation, as they push novice evaluators for pre-determined designs to fulfill funding requirements. Through literature review, weekly journal entries and retroactive reflections, this paper will investigate how two novice evaluators navigate decisions of evaluation design and the selection of appropriate methodologies and data analysis within the confines of different programs. We will share a thought experiment to exemplify how methodological decisions concerned primarily with statistical rigour but not contextual warrants can lead to inaccurate findings. Finally, we will explore the roles new evaluators take in methodological decision making within programs.