Evaluation 2017: From Learning to Action

View Printable Version

Whose Voices Determine Success?

Session Number: PREK1211
Track: PreK-12 Educational Evaluation
Session Type: TIG Multipaper
Session Chair: Pauline Dickinson [Evaluation Team Leader - SHORE & Whariki Research Centre - Massey Universit]
Discussant: Sheila Arens [Executive Director of Research and Evaluation - McREL International]
Presenter 1: Deborah K. Reed [Director and Associate Professor - University of Iowa]
Presenter 2: Keith Trahan [Assistant Director, Collaborative for Evaluation and Assessment Capacity - University of Pittsburgh]
Presenter 3: Lisa Dillman [Senior Advisor - Applied Research and Evaluation - Education Northwest]
Presenter 4: Dana Linnell Wanzer [Research and Evaluation Associate - Claremont Graduate University]
Presentation 1 Additional Author: Ariel Aloe [Faculty - The University of Iowa ]
Presentation 2 Additional Author: Cynthia A Tananis [University of Pittsburgh]
Presentation 2 Additional Author: Everett Herman [Lead Evaluator - University of Pittsburgh]
Presentation 3 Additional Author: Erin Bock [Director of Research and Evaluation - The Sherwood Foundation]
Presentation 4 Additional Author: Tiffany D Berry [Research Associate Professor - Claremont Graduate University]
Time: Nov 09, 2017 (03:15 PM - 04:15 PM)
Room: Washington 1

Abstract 1 Title: Interpreting the Effectiveness of a Summer Reading Program: The Eye of the Beholder
Presentation Abstract 1:

In over one-third of the U.S., summer reading programs are required or recommended for elementary students who are not reading proficiently. Given the cost of providing such programs, there is an interest in evaluating their effectiveness. Using data obtained on 1,026 students in grades 2-5, this study explored different approaches to interpreting student outcomes based on the nature of the question being asked. The district officials were interested in knowing whether students improved their raw reading performance from pre- to posttest. Funders and policy makers were interested in knowing whether students improved their categorical designations from “basic” or “below basic” to “proficient” readers. Finally, researchers were interested in knowing whether summer school participants performed significantly differently on the immediate and delayed posttests from non-participating students who also had reading difficulties. This session will present the results of each approach and address the need to clearly define “effectiveness” in evaluation studies.


Abstract 2 Title: Making Success: Evaluating the integration of Making and related innovation into a school district’s middle and high school
Presentation Abstract 2:

Elizabeth Forward School District in Southwestern Pennsylvania has been an early integrator of MakerSpaces in formal education. The context of this district has made this reform possible: Pittsburgh has become a hub of technological and maker innovation. Therefore, the milieu of the district is crucial to the success of this reform. This presentation will describe how an evaluation framework can incorporate a clients’ context as a part of the measurement of the factors that contribute to successful implementation. In this way, an evaluation can provide information about crucial contextual factors to others interested in implementing such a reform.


Presentation 2 Other Authors: Stephanie Romero; University of Pittsburgh; Lead Evaluator
Abstract 3 Title: Promoting Accountability and Use in a Sea of Complexity and Controversy: A Case Study of a Sex Ed Curriculum Implementation Evaluation
Presentation Abstract 3:

What happens when a use- and user-focused evaluation effort finds itself embroiled in a sea of conflicting priorities, a highly politicized context, and reluctant advocacy? How can evaluators balance the needs of a wide variety of primary users while maintaining rigor and high ethical standards? How can the evaluation function simultaneously at a variety of levels to promote accountability and use of findings? In response to high STD and teen pregnancy rates, funders and other leaders in a Midwest City began planning in 2015 to support health providers, develop an awareness campaign, and revise the public school curriculum in order to enhance the reproductive and sexual health of youth in the community. The evaluator and funder will present an in-process evaluation of the curriculum implementation in a school district embedded in a larger context of a community-wide public health initiative that serves as a case study to explore these questions.


Abstract 4 Title: Using Vignettes to Improve Staff Knowledge about Program Quality
Presentation Abstract 4:

Training program staff about what quality means and what high program quality looks like is an important first step towards improving program quality. This presentation explores how vignettes—short stories about hypothetical characters in specific circumstances—can be useful for teaching program staff how to think about program quality and, for organizations with high evaluation capacity, learn how to conduct observations prior to going out “into the field.” Through our work, we used this activity across three different groups of people (two afterschool programs and one group of budding evaluators); their usefulness for teaching them what program quality looks like and preparing them for conducting observations will be discussed. Implications of using vignettes to promote evaluative thinking in organizations and as a strategy to promote continuous quality improvement will be discussed.


Theme: My presentation doesn't specifically relate to the theme
Audience Level: All Audiences

Session Abstract (150 words): 

Whose Voices Determine Success?



For questions or concerns about your event registration, please contact registration@eval.org or 202-367-1173.

For questions about your account, membership status, or help logging in, please contact info@eval.org.



Cancellation Policy: Refunds less a $50 fee will be granted for requests received in writing prior to 11:59 PM EDT October 16, 2017. Email cancellation requests to registration@eval.org. Fax request to (202) 367-2173. All refunds are processed after the meeting. After October 16, 2017 all sales are final. For Evaluation 2017, international attendees and presenters who encounter complications due to the international travel environment will have up to 30 days after the event to request a refund and submit appropriate documentation. No administrative fee will apply for the international requests. The $50 fee will be waived for registrants who planned to travel into the US and experienced international travel issues.