Evaluation 2017: From Learning to Action

View Printable Version

New strategies for evaluating community networks: Learning on the ground in a multisite initiative

Session Number: 2051
Track: Community Development
Session Type: Panel
Tags: collaboration, multi-site evaluation, Process tracing, social network analysis
Session Chair: Debra J Rog [Vice President - Westat]
Discussant: Hallie Preskill [Managing Director - FSG]
Presenter 1: Nanmathi Manian [Sr. Study Director and Evaluator - Westat]
Presenter 2: Tamara Cohen Daley [Senior Study Director - Westat]
Presenter 3: Debra J Rog [Vice President - Westat]
Time: Nov 11, 2017 (08:00 AM - 09:00 AM)
Room: PARK TWR STE 8222

Abstract 1 Title: Assessing the “What” in Networks: Learning How to Achieve Rigor Through Parsimony
Presentation Abstract 1:

Evaluating complex interventions implemented by community networks presents many challenges. Given the many interacting components of a complex intervention, the critical issue is knowing what to evaluate, and then developing measures that are rigorous yet feasible to address the varying nature of the network activities and the different outcomes that they lead up to. Using more focused ways of tracking the network activities that matter most to the networks and the outcomes of those that matter, we developed measures to assess the activities of the 14 MARC community networks. This presentation describes how we used a case-study approach to understand the focus of activities within each network as well as a cross-site approach to analyze the effectiveness of the activities. We highlight the usefulness of this approach for the networks, issues in triangulation of findings, and the contribution of the findings to our learning about coalition and movement building.

Abstract 2 Title: The Formative and Summative Benefits of Social Network Analysis in Evaluation of Community Collaboratives
Presentation Abstract 2:

Social Network Analysis (SNA) has increasingly been used as a tool to examine community collaboratives and cross-sector partnerships. In the context of the MARC initiative, a network survey within each community administered at two time points provided a unique opportunity to use SNA not only as a mechanism for the evaluation team to examine network structure and network change across the 14 communities, but also as a tool to help communities understand, develop and strengthen their individual networks. This latter and more formative use of SNA was strongly facilitated by positioning the MARC Network survey as a collaborative effort between the 14 MARC communities and the evaluation team. This presentation will describe the shared development, administration and analysis of the network survey; how the communities have used the tool themselves; and what we have learned about cross-sector community networks through this process.

Abstract 3 Title: Shining the Light Where Change Occurs: Assessing Organic and Unplanned Outcomes
Presentation Abstract 3:

Community networks often have multiple, simultaneous efforts to create synergy for change.  Traditional program evaluation approaches are not always sensitive to this synergy and the more organic ways networks can contribute to change.  Moreover, in a multisite network initiative, change can occur from the combined work of the sites in ways that may not be fully documented.  The presenter will describe the lessons learned in the first year of a multisite network evaluation that caused the evaluators to redesign the outcome component.  The presenter will highlight two strategies added to understand the role of networks in creating change related to ACEs, trauma, and resilience.  Contribution analysis is being used to assess the contribution of each network and the initiative as a whole to particular targeted outcomes.  Process tracing is also being used to trace backwards the initiative’s influence on key observed changes that may not have been initially targeted. 

Theme: Learning to Enhance Evaluation
Audience Level: Intermediate

Session Abstract (150 words): 

New developments in initiatives and programming often create new demands for evaluation.  Coalitions and networks have long been strategies of funders to carry out community efforts that are too large for any single organization.  There is a rich history of evaluation work on community collaboratives but much less on evaluating multisite efforts involving community networks and the extent to which networks can spark or be part of broader movements for change.  Through our evaluation of a foundation funded multisite community initiative focused on addressing adverse childhood experiences (ACEs), this panel will describe how we are learning to more sensitively assess the contributions of the networks singularly and together through applying both traditional and new methods.  Social Network Analysis, contribution analysis and process tracing are being used together with more traditional case study methods.  We will describe the challenges faced and opportunities realized through this effort.

For questions or concerns about your event registration, please contact registration@eval.org or 202-367-1173.

For questions about your account, membership status, or help logging in, please contact info@eval.org.

Cancellation Policy: Refunds less a $50 fee will be granted for requests received in writing prior to 11:59 PM EDT October 16, 2017. Email cancellation requests to registration@eval.org. Fax request to (202) 367-2173. All refunds are processed after the meeting. After October 16, 2017 all sales are final. For Evaluation 2017, international attendees and presenters who encounter complications due to the international travel environment will have up to 30 days after the event to request a refund and submit appropriate documentation. No administrative fee will apply for the international requests. The $50 fee will be waived for registrants who planned to travel into the US and experienced international travel issues.