Evaluation 2017: From Learning to Action

View Printable Version

How do we know what difference advocacy makes? Using contribution analysis in policy work

Session Number: 2187
Track: Advocacy and Policy Change
Session Type: Panel
Tags: Advocacy and policy change, advocacy evaluation, Contribution analysis
Session Chair: Sarah Stachowiak [CEO - ORS Impact]
Presenter 1: Robin Kane [Principal - RK Evaluation & Strategies, LLC]
Presenter 2: Claire Reinelt [Independent Consultant]
Presenter 3: Carlisle Levine [President and CEO - BLE Solutions, LLC]
Presenter 4: Carlyn Orians [Director of Evaluation - ORS Impact]
Time: Nov 09, 2017 (02:15 PM - 03:00 PM)
Room: Jefferson

Abstract 1 Title: When is contribution analysis appropriate for policy work? What risks does it carry with it, and how can these be addressed?
Presentation Abstract 1:

We will describe contribution analysis and discuss under what conditions contribution analysis is appropriate to use with policy work. Some conditions relate to mechanics, such as the time and effort allocated to the activity, or the access an evaluator has to advocates, allies, opponents, and policymakers. Other conditions relate to the sensitivity of the advocacy that was undertaken. For example, was the advocacy public or quiet and behind the scenes? In the latter case, what would be the consequences of bringing the advocacy out into the open? In addition to discussing conditions, we will also discuss risks associated with using contribution analysis, and how they can be mitigated. In advocacy evaluation, there are risks associated with many approaches, and contribution analysis shares these. That contribution analysis attempts to identify linkages between advocacy efforts and policy outcomes creates others. In this presentation, we will discuss both.

Abstract 2 Title: How to conduct a contribution analysis of a policy initiative
Presentation Abstract 2:

Contribution analysis involves working with advocates to develop a theory of change describing how a policy change came about, and then testing that theory of change and possible alternate explanations through interviews with allies, opponents, and policy makers in order to make a credible assessment of how the change came about. We will discuss unique challenges of developing and testing theories of change for policy advocacy work, and ways to address them. We will describe tools we used to assess advocacy efforts, and how we identified critical junctures in the policy process where interventions were important factors in shaping an outcome. We will discuss what sources and methods of data collection are most useful for testing and corroborating contribution claims, and why understanding context and motivations of various actors is important for interpreting those claims. Finally we will present suggestions for how to use contribution stories to advance advocacy efforts.

Abstract 3 Title: Assessing a network’s contribution to national-level policy change: The case of the Modernizing Foreign Assistance Network
Presentation Abstract 3:

When looking at national-level policy change that results from many factors and the work of many actors, how do we know what influence a particular actor had over the ultimate outcome? And how can we assess how the outcome might have been different, absent that actor’s participation? We will present how we used contribution analysis to address these challenges, when evaluating the Modernizing Foreign Assistance Network’s efforts to influence national-level legislation and policies. Whether we had access to advocates, allies, opponents, and decision makers greatly affected our ability to elicit theories of change and test alternative explanations. The amount of time passed since a change took place also affected our ability to gather credible evidence, since most advocacy efforts were not documented and memories fade. Finally, we were aware of political sensitivities, and sought to ensure, at a minimum, that relationships remained intact for future advocacy efforts.

Abstract 4 Title: Contribution analysis in a current, distributed advocacy context: The case of the Alliance for Early Success
Presentation Abstract 4:

Contribution analysis aims to systematically develop a contribution story by refining a results chain, gathering evidence of linkages, and explicitly testing the story against plausible alternative explanations. What happens when advocacy is unfolding across multiple jurisdictions concurrent to the evaluation? This presentation will share the strengths and challenges associated with evaluating the Alliance for Early Success’ contribution to advancing state policies for early education as state legislative seasons were concluding. Strengths of the method in this context included: 1) the ability to talk directly with advocates to clarify the results chain, and 2) access to decision makers to verify linkages and surface alternative explanations. Challenges arose in accurately articulating the state expression of the national theory of change, defining result in cases where the outcome was not yet completely known, lack of distance to accurately reflect on the contribution story, and filtering the stories through each interviewee’s frame and motives.

Theme: Learning to Enhance Evaluation
Audience Level: Intermediate

Session Abstract (150 words): 

Providing evidence of how advocacy activities influence policy outcomes is one of advocacy evaluation’s greatest challenges. In this session, evaluators will discuss how they have used contribution analysis in advocacy: What is contribution analysis? Why did they select it? Under what conditions does it work? How did they apply it in a rigorous way? What risks and challenges did it present, and how did they overcome them? Two cases will illustrate how they have used it at state and national levels. ORS Impact has been working with the Alliance for Early Success to assess advocates’ influence related to state policies for children. BLE Solutions worked with the Modernizing Foreign Assistance Network (MFAN) and the William and Flora Hewlett Foundation to assess MFAN’s success advancing U.S. foreign assistance reform. These cases, although different, offer similar lessons regarding using contribution analysis in advocacy evaluation.

For questions or concerns about your event registration, please contact registration@eval.org or 202-367-1173.

For questions about your account, membership status, or help logging in, please contact info@eval.org.

Cancellation Policy: Refunds less a $50 fee will be granted for requests received in writing prior to 11:59 PM EDT October 16, 2017. Email cancellation requests to registration@eval.org. Fax request to (202) 367-2173. All refunds are processed after the meeting. After October 16, 2017 all sales are final. For Evaluation 2017, international attendees and presenters who encounter complications due to the international travel environment will have up to 30 days after the event to request a refund and submit appropriate documentation. No administrative fee will apply for the international requests. The $50 fee will be waived for registrants who planned to travel into the US and experienced international travel issues.