Arts Education Collective Impact Initiatives: Meeting the Challenge of Shared Measurement across Sites

Session Number: 2286
Track: Arts, Culture, and Audiences
Session Type: Multipaper
Tags: arts and culture, arts education, Collective Impact
Session Chair: Patricia Moore Shaffer [Deputy Director, Research & Analysis - National Endowment for the Arts]
Presenter 1: Patricia Moore Shaffer [Deputy Director, Research & Analysis - National Endowment for the Arts]
Presenter 2: Dennie Palmer Wolf, Ed.D [Principal Researcher - WolfBrown]
Presenter 3: Don Glass [Research Manager - The Kennedy Center]
Presentation 2 Additional Author: Steven J Holochwost [Senior Research Scientist & Associate Principal - WolfBrown]
Time: Nov 01, 2018 (03:45 PM - 04:45 PM)
Room: CC - 12

Abstract 1 Title: Developing Program Theory for a National Collective Impact Grant Program

Presentation Abstract 1:

The national agency made its first Collective Impact grant awards in 2014. These awards are intended to increase student access to arts education through collective, systemic approaches. The agency is developing an evidence-based logic model and measurement model in close consultation with its grantees. This conceptual framework will provide the agency with valuable information to 1) inform guidelines revision (including refining, if needed, the program model), 2) document grantee successes, challenges, and successful practices for sharing with the grantee community, and 3) establish the basis for future evaluation activities. This presentation will describe the process for developing this framework, including a literature scan, consultation with project directors and evaluators, and grantee testing of the logic model using their knowledge of their own Collective Impact program.


Abstract 2 Title: Developing Nested Models of Collective Impact

Presentation Abstract 2:

A national performing arts center’s advocacy project for access to arts education, now operates in 27 sites that vary on geography, population, density of cultural resources, as well as many other dimensions. In its second major evaluation of the initiative, the arts center is developing a suite of evaluation strategies designed to acknowledge both the common goals that unite the sites and the diversity of strategies and resources individual sites bring to the work. In partnering with the arts center, researchers are collaborating with community teams in eight focus sites to develop a set of common tools that capture impact in several major areas such as: 1) student access to arts education and 2) the size and range of social networks that support arts education. In addition, the work with focus communities is identifying other potential areas of impact relevant to subsets of communities such as: 1) funding sources for arts education and 2) impact of public policy. A major goal -- and challenge -- of this work is to develop differentiated, or nested, models of collective impact that feature a combination of common and more local results of common efforts in contrasting ecologies.


Presentation 2 Other Authors: Additional Author: Christine Witkowski

Abstract 3 Title: Developing Practical Measures for Equitable Access and Quality

Presentation Abstract 3:

Outcomes measures help us take a longer view of what we aim to improve. These “end-state” goals are often more global in scope and data is collected less frequently. In arts education collective impact initiatives, these can measure changes in populations served, opportunities for arts learning, attendance, and student proficiency. To help us reliably know what progress we are making toward our outcomes, we need more frequent and formative process measures that monitor the program drivers. These practical measures should not be a burden to administer, analyze, and use for decision-making. This paper examines the development process of designing and adapting a set of measures that have constructs and items with a solid basis in the literature, good psychometrics (Cronbach’s Alpha, Factor Analysis, and Rasch Analysis), and are focused and adaptable enough for practical and frequent use by practitioners to improve their work.*

*Bryk, Yaeger, et.al. Practical Measurement. (2013). https://www.carnegiefoundation.org/wp-content/uploads/2013/12/Practical_Measurement.pdf


Presentation 3 Other Authors: Additional Author: Charles Beekman

Audience Level: Intermediate, Advanced

Session Abstract (150 words): 

Collective Impact initiatives represent a fundamental shift in how we organize our collective work, optimize often limited resources, and make reliable improvements across complex and varied contexts. Of the five common conditions for collective impact, a shared measurement system is the most germane to evaluation. For multi-site Collective Impact initiatives, the challenge is to create a measurement model that assesses both common as well as local implementation and outcomes across multiple locations. In this panel, presenters discuss approaches to the development of shared measurement systems for Collective Impact arts education initiatives that take place in multiple sites, including the development of program theory to inform the design of a shared measurement system; the development of process measures that monitor the program drivers; and the development of differentiated, or nested, models of collective impact evaluation designed to assess both common and local results of common efforts.