Evaluation 2018: Speaking Truth to Power

View Printable Version

Breaking News: We Can’t Control Everything! - Using Systems Thinking to Understand Context in Development Projects

Session Number: 2325
Track: Systems in Evaluation
Session Type: Multipaper
Tags: International development, system evaluation, systems, systems approaches, systems mapping, Systems Thinking, systems-oriented evaluation
Session Chair: Sibel McGee, Ph.D. [Distinguished Analyst - ANSER]
Presenter 1: Sibel McGee, Ph.D. [Distinguished Analyst - ANSER]
Presenter 2: Patrick Sommerville [Managing Director - LINC]
Presenter 3: Frances Christine Fisher Veasey [Analyst - ANSER]
Presenter 4: Patrick Sommerville [Managing Director - LINC]
Presentation 1 Additional Author: Frances Christine Fisher Veasey [Analyst - ANSER]
Presentation 2 Additional Author: Rich Fromer [Managing Director - LINC LLC]
Presentation 3 Additional Author: Frances Christine Fisher Veasey [Analyst - ANSER]
First Author or Discussion Group Leader: Sibel McGee, Ph.D. [Distinguished Analyst - ANSER]
Second Author or Discussion Group Leader: Patrick Sommerville [Managing Director - LINC]
Third Author or Discussion Group Leader: Frances Christine Fisher Veasey [Analyst - ANSER]
Time: Nov 02, 2018 (10:30 AM - 11:15 AM)
Room: Hilton - Veterans Meeting Room C

Abstract 1 Title: Critical Success Factors for Development Initiatives: Which Local System Attributes Help Shape Development Outcomes?

Presentation Abstract 1:

The Local Systems Practice consortium, sponsored by USAID’s localworks initiative, conducted initial research into country-level context factors and how they shape development outcomes. This revealed a gap in development researchers’ and practitioners’ understanding of local system-related attributes, and how much of the variation in outcomes they explain. This paper presents findings of exploratory research identifying and characterizing important context factors. As the first phase of a long-term project, the study team compiled and reviewed research and evaluations from various fields and used qualitative data analysis techniques to identify seven local system-related factors with implications for development outcomes. The paper presents a framework involving the seven attributes, related definitions, and conceptual extreme points, and sets the stage for further refinement and testing of these attributes through concrete country engagements. Its final results can inform evaluators’ and project designers’ understanding of which local system attributes may qualify as critical success factors.

Abstract 2 Title: USAID Strategic Program for Analyzing Complexity and Evaluating Systems (SPACES MERL): Systems and Complexity White Paper

Presentation Abstract 2:

The “Systems & Complexity White Paper” is the product of a USAID-funded collaboration between Johns Hopkins University, LINC, the Global Knowledge Initiative (GKI) and Makerere University’s Resilient Africa Network (RAN). It is a how-to manual on the application of systemic design, monitoring & evaluation practices into international development programming. This White Paper is an excellent resource for local and international development practitioners considering new methods for design and evaluation of their projects, ways in which local context and complexity can be more effectively captured and designed into program strategy. Profiling several systems thinking methods, this White Paper is available for public download on the USAID website and includes content on:

  • Defining Complex Systems
  • Framework of Systems-based Approaches to Design, Monitoring and Evaluation
  • Visualization Methods: Mapping and Modelling
  • Narrative-based Approaches
  • Indicator-based Approaches

Abstract 3 Title: What does Stability Programming in Afghanistan Tell Us?: A Systems Perspective on the Significance of Local Context

Presentation Abstract 3:

In this paper, ANSER used a systemic approach to investigate the causes for poor performance of USAID’s stability programming in meeting many of its intended objectives in Afghanistan. The study team leveraged two systems thinking tools (causal loop diagrams and leverage point analysis) to characterize theory of change, program implementation context, and their interaction. Based on the “hearts and minds” doctrine, the team first developed a generic theory of change for promoting stability in counterinsurgency contexts and then assessed how operationalization of this ideal theory unfolded in reality in Afghanistan. In doing so, they uncovered key dynamics and processes that shaped and conditioned program performance outcomes, and identified reasons for unintended consequences and paradoxical effects. The study revealed that, among other factors, inadequate consideration of the local context and related confounding factors led to the design and implementation of programs with conflicting goals and potential to exacerbate the original problems.


Presentation 3 Other Authors: Nathan Mariano (ANSER), Andres Uribe (ANSER)
Abstract 4 Title: Systems Thinking Meets Adaptive Management: A Network Analysis of USAID Grantees for the Rice and Diversified Crops (RDC) Project in Bangladesh

Presentation Abstract 4:

This Bangladesh Network Analysis Report is the product of a pilot initiative conducted in 2017, applying network analysis to a USAID-funded agricultural initiative, the Rice and Diversified Crops (RDC) project. Our challenge was to develop a network analysis tool that could be easily understood, transferred and utilized by project managers, while still generating meaningful insights to inform both strategy and adaptive management of the program. We achieved this through utilization of a hybrid “Egonet” approach conducted with grantees of the RDC project in multiple iterations. The egonet tool was particularly useful in identifying structural dynamics and social norms and biases in the system. Time and resources required were modest. Our multi-iteration approach shows how the tool can be used for monitoring purposes. The egonet approach was nonetheless limited in the breath of data collected, thus requiring add-on qualitative systems analysis, including direct follow-up with respondents (egos) and their relations (alters). 

Presentation 4 Other Authors: Eric Derks (Market Systems Expert, The Canopy Lab)
Audience Level: All Audiences

Session Abstract (150 words): 

Most evaluations and literature that assess success factors in development focus on project management related factors – things we can control, like how interventions are designed and implemented. Although we realize intuitively that projects and programs take place in a complex, dynamic environment, we struggle to incorporate context into design, monitoring, and evaluation. In this session, we will examine what a context-based approach to evaluation might look like using systems thinking tools. First, we examine seven context attributes that should be considered for how they help or hinder positive change. Then, we review several systems tools that can be used to understand complex environments. We then present two case studies using systems thinking tools for monitoring and evaluation practice: a systems-thinking examination of the factors and context leading to the failure of stability programming in Afghanistan, and a network analysis assisting the monitoring of an agriculture project in Bangladesh. 

For questions or concerns about your event registration, please contact registration@eval.org or 202-367-1173.

For questions about your account, membership status, or help logging in, please contact info@eval.org.

Cancellation Policy: Refunds less a $50 fee will be granted for requests received in writing prior to 11:59 PM EDT October 8, 2018. Email cancellation requests to registration@eval.org. All refunds are processed after the meeting. After October 8, 2018 all sales are final. For Evaluation 2018, international attendees and presenters who encounter complications due to the international travel environment will have up to 30 days after the event to request a refund and submit appropriate documentation. No administrative fee will apply for the international requests.