Evaluation 2019: Paths to the Future of Evaluation: Contribution, Leadership, and Renewal

View Printable Version

Systems and complexity informed methods and tools in action

Session Number: SE1
Track: Systems in Evaluation
Session Type: TIG Multipaper
Session Chair: Margaret B. Hargreaves [Senior Fellow - NORC at the University of Chicago]
Presenter 1: Sibel McGee, Ph.D. [Distinguished Analyst - ANSER]
Presenter 2: Sibel McGee, Ph.D. [Distinguished Analyst - ANSER]
Presenter 3: Hernando Grueso Hurtado [Ph.D. Student - Trachtenberg School of Public Policy & Public Administration, George Washington University]
Presenter 4: Amelia DeFosset [Research Analyst - Los Angeles County Department of Public Health]
Presentation 1 Additional Author: Frances Christine Fisher Veasey [Analyst - ANSER]
Presentation 2 Additional Author: Frances Christine Fisher Veasey [Analyst - ANSER]
Presentation 4 Additional Author: Gabrielle Green [Research Analyst - Los Angeles County Department of Public Health]
Presentation 4 Additional Author: Megala Sivashanmugam [Research Analyst - Los Angeles County Department of Public Health]
Presentation 4 Additional Author: Noel Barragan [Manager, Special Projects and Strategic Initiatives - Los Angeles County Department of Public Health]
Time: Nov 15, 2019 (04:30 PM - 05:30 PM)
Room: CC M100 B

Abstract 1 Title: Should We Use Causal Loop Diagramming for Program Design or Evaluation? Why Not Both?
Presentation Abstract 1:

The Local Systems Practice consortium, sponsored by USAID’s localworks initiative, presents a concept for a systems thinking-based approach to program design and evaluation as interdependent phases of a holistic approach to development problems. Specifically, we show how causal loop diagramming technique can help map key cause-effect relationships and uncover underlying structures in a local context. These qualitative models can guide more realistic theories of change as effects of program activities can be traced through critical causal pathways to understand desired and unintended consequences. When implemented as part of a broader systemic approach, the same model can be used and dynamically updated to identify key variables and relationships whose changes are critical to monitor during program implementation and to evaluate outcomes following program completion. Such an approach will not only seamlessly marry program design and evaluation, but also go beyond establishing whether a program worked or not by answering “why?” questions.

Abstract 2 Title: Poverty from the Perspective of Complex Systems: An Application of Bayesian Networks in Program Evaluation
Presentation Abstract 2:

The Measuring Impacts of Stabilization Initiatives (MISTI) evaluation, conducted over a 3-year period at a cost of $19.3 million, concluded that the targeted stability programs had little to no positive effect on the situation in Afghanistan, but it said little about WHY this was the case. ANSER identified this as a situation where systems thinking tools may provide insight into the complex dynamics, emergent behaviors, and root causes at play. In-depth understanding of problems like this also offers more powerful mitigation approaches for similar programs in the future.  The study team used causal loop diagramming (CLD) combined with leverage point analysis to capture the complex dynamics that conditioned program performance outcomes, and identified reasons for unintended consequences and paradoxical effects. This led to specific, concrete, and actionable recommendations that should be followed for future programs in active conflict or insurgency environments.

Abstract 3 Title: Poverty from the Perspective of Complex Systems: An Application of Bayesian Networks in Program Evaluation
Presentation Abstract 3:

There is a growing consensus in international development about poverty as a multidimensional phenomenon, however, there is no consensus on a single definition of multidimensional poverty. Poverty is context specific and affected by a complex interaction of multiple factors. In addition, the use of multidimensional results implies a practical difficulty in program evaluation: the changes in the results could be driven by one dimension, multiple dimensions or the interaction between them. This paper proposes a methodological framework to do impact evaluations in international development, by analyzing the effect of a program on a system of interrelated variables that define poverty as a multidimensional concept. An example of this analytical framework is provided based on a Living Standards Measurement Survey from Colombia (2017), and using network analysis to plot different structures of a multidimensional poverty system by different population subgroups. 

Abstract 4 Title: Using Data to Evaluate and Drive Improvements in Complex Systems: The Application of Systems Science to Build Effective Community-Clinic Linkages in Los Angeles
Presentation Abstract 4:

Integrating community-based resources with clinical care can help address the intricate and interrelated issues that underlie poor health. As this task requires substantial changes within complex systems, data-driven, systems science approaches are well suited for these efforts. However, there are few examples of how this approach has been adapted as a pragmatic evaluation strategy. This presentation will describe the systems science approaches that were used in Los Angeles to refine and institutionalize an innovative clinic-community linkage model. Key aspects of the project will be discussed, including: 1) how clinical and community stakeholders were engaged as core partners in learning and action, and 2) approaches for collecting, synthesizing, and depicting diverse data within a rapid-cycle format. Select results of iterative data collections will be presented, alongside a discussion of how findings were used to guide and test incremental program improvements. Results provide practical tools for incorporating systems science into evaluative practice.

Presentation 4 Other Authors: Tony Kuo (tkuo@ph.lacounty.gov)<br />Los Angeles County Department of Public Health<br />Director, Division of Chronic Disease and Injury Prevention
Audience Level: All Audiences

Session Abstract (150 words): 

Systems and complexity informed methods and tools in action

For questions or concerns about your event registration, please contact registration@eval.org or 202-367-1173.

For questions about your account, membership status, or help logging in, please contact info@eval.org.

Cancellation Policy: Refunds less a $50 fee will be granted for requests received in writing prior to 11:59 PM EDT October 11, 2019. Email cancellation requests to registration@eval.org. All refunds are processed after the meeting. After October 11, 2019 all sales are final. For Evaluation 2019, international attendees and presenters who encounter complications due to the international travel environment will have up to 30 days after the event to request a refund and submit appropriate documentation. No administrative fee will apply for approved international requests.