Measuring Agency-Level Results: Can it Work, and Does It Matter? Findings From an InterAction White Paper

Session Number: 1732
Track: International and Cross Cultural Evaluation
Session Type: Panel
Tags: Non-Profit, organizational learning
Session Chair: Jennifer Heettner [American Jewish Joint Dist. Committee Director, Global Program Knowledge Management]
Discussant: Jennifer Heettner [American Jewish Joint Dist. Committee Director, Global Program Knowledge Management]
Presenter 1: Carlisle Levine [President and CEO - BLE Solutions, LLC]
Presenter 2: Tosca Bruno-van Vijfeijken, Ms. [Director, Transnational NGO Initiative - Maxwell School of Citizenship and Public Affairs, Syracuse University]
Presenter 4: Christie Getman [Director, Program Performance and Quality - Mercy Corps]
Time: Oct 28, 2016 (08:00 AM - 09:30 AM)
Room: M302

Abstract 1 Title: The Current Landscape of INGO Agency-Level Measurement Systems
Presentation Abstract 1:

International non-governmental organizations (INGOs) that work in many countries and sectors are challenged to measure their results at an agency level. One approach that has been taken by many INGOs is to establish agency-level measurement systems, often based on quantitative indicators, which are often complemented by qualitative data. However, many INGOs have had difficulty designing their systems so that they are feasible to implement and produce meaningful results. To address this, InterAction commissioned a study to better understand and learn from why INGOs set up agency-level measurement systems, how they are designed, what they contain, and how they are used. In this presentation, which covers the first part of the study, drivers, design and use will be discussed. Particularly, the presentation will focus on the range of existing designs and their strengths and weaknesses, as well as what it takes to promote use of the data they produce.

Abstract 2 Title: Lessons learned from INGO experiences in developing agency level measurement systems
Presentation Abstract 2:

For those INGOs who have decided to adopt agency level measurement systems, the complexities of having such a system go beyond methodology and technical details, and focus as well on who should own such a system, which set of stakeholders to prioritize, how to incentivize both data generation as well as usage, how to manage the change process including potential culture change that ALM implies, and how to ensure that the information management system supports and does not lead the approach chosen – among others. This presentation will reflect key lessons as well recommendations based on the experiences of US INGOs who are developing agency level measurement systems, and beyond.

Abstract 3 Title: Developing an INGO agency-level measurement system: what does it take?
Presentation Abstract 3:

Improving the International Rescue Committee's project-level M&E and its tracking of progress toward strategic objectives has been an organizational priority for over three years. Based on an assessment of current IRC status and approaches taken by other INGOs, the IRC system integrates four critical components: 1) “regional measurement action coordinators” to support measurement initiative roll-out and field M&E; 2) DHIS-based online data platform for management and analysis of data within and across projects and countries; 3) core indicators for use as high quality project indicators, also tracked globally; and 4) Monitoring for Action, an initiative designed to improve quality and use of project monitoring data by providing concrete support to country teams, including tools, guidance, and establishing clear and measurable standards for accountability. A strategic commitment to systematic, organization wide measurement in its 2015-2020 five-year strategy was critical to ensure senior leadership and senior management engagement and proper resourcing of the initiative. Broad field participation has also been critical for success to date.

Abstract 4 Title: Alternatives to Agency-Level Common Indicator Systems: What and Why?
Presentation Abstract 4:

The InterAction White Paper on Measuring Agency Level Results points out many of the conditions and resources that should be in place, in order to successfully and effectively set up an agency-level measurement system, in particular a set of common indicators. However, not all agencies have the resources or infrastructure to make a system like that worth it. Lutheran World Relief examined the options in 2012, and opted for alternative ways to measure agency level results. Focusing on project evaluations, meta-evaluation reviews, basic portfolio data and performance dashboards are examples of alternative tools that can still prove highly useful to senior leaders. This session will describe why some agencies might choose against setting up agency level indicator systems and give examples of other options. It will conclude on the “now what?” point of how those alternatives have been valued within the agency and what lies in store for the future.

Audience Level: Intermediate

Session Abstract: 

What do international non-governmental organizations (INGOs) achieve at a global level? How many people are INGOs reaching? To what degree are they achieving their missions and making a difference in people's lives? INGOs try to address these questions in a variety of ways. Many develop agency-level measurement systems. However, while INGO leaders want compelling dashboards of key data summarizing an INGO’s health and effectiveness, evaluators leading their design and implementation are challenged to develop meaningful measures and ensure data quality that make them useful. In response, InterAction, the largest alliance of international NGOs and partners in the U.S, and 11 members commissioned a white paper to learn from experiences in building these systems: what works well in what ways and under what conditions? What challenges have INGOs faced in building and using these systems, and how have they addressed these challenges? InterAction, INGOs and evaluators will answer these questions based on their study’s findings.