Evaluation 2015: Exemplary Evaluations in a Multicultural World

View Printable Version

Alternative Measures of Organizational Evaluation Capacity

Session Number: 2234
Track: Government Evaluation
Session Type: Multipaper
Tags: Evaluation capacity building
Session Chair: Stephanie Shipman [US Government Accountability Office]
Discussant: Cynthia Clapp-Wincek [Independent consultant]
Presenter 1: Stephanie Shipman [US Government Accountability Office]
Presenter 2: Isabelle Bourgeois [University of Quebec]
Presenter 3: Jim Rugh [EvalPartners International]
Presenter 4: Colin Meredith [Office of the Auditor General of Canada]
Presentation 1 Additional Author: Valerie Jean Caracelli [Senior Social Science Analyst - U.S. Government Accountability Offi]
Time: Nov 12, 2015 (04:45 PM - 06:15 PM)
Room: Skyway 283

Abstract 1 Title: Key Elements of Federal Organizational Capacity to Conduct and Use Evaluation
Presentation Abstract 1: Drawing on an array of domestic and international literature, GAO identified three general categories of elements that provide a useful and practical framework for examining national organizational evaluation capacity. The first category, an enabling environment, includes features of the governing context such as legislation that supports the use of evidence in management and policy making generally. A second category, organizational resources, pertains to how evaluation capacity has traditionally been defined, including clearly defined roles, a stable source of funding, and staff expertise. The third category, evaluation results and use, addresses the quality and quantity of completed evaluations and their use in decision making. This expanded conceptualization was used to probe the extent of federal agency evaluation capacity in a governmentwide survey of senior agency officials. The survey results as well as alternatives for conducting individual agency assessments will be discussed.
Abstract 2 Title: Measuring Organizational Evaluation Capacity through a Self-Assessment Instrument: Process and Implications
Presentation Abstract 2: This presentation will describe an organizational evaluation capacity self-assessment instrument, developed by Bourgeois, Toews, Whynot and Lamarche (2013) and implemented in a number of public sector organizations in Canada and the United States. The instrument, based on a framework of evaluation capacity developed by Bourgeois and Cousins (2013), measures both the organization’s capacity to do and use evaluation through a consensus-building exercise. The rationale for the instrument and organizational self-assessment process will be described in detail, drawing from real-life examples. The information produced by the instrument and implications for the development of organization-wide evaluation capacity building strategies will also be presented to illustrate the outcomes and utility of the self-assessment process.
Abstract 3 Title: How Evaluation Associations in Other Countries Are Influencing and Assessing Evaluation Capacities of Their Governments
Presentation Abstract 3: This spring, the International Organisation for Cooperation in Evaluation (IOCE) is finalizing a new survey questionnaire going out to our contacts with VOPEs (Voluntary Organizations for Professional Evaluation) in over 100 countries to obtain the perspectives on how our colleagues in many countries assess the capacities of their own governments. Jim Rugh, on behalf of IOCE, will share some of the initiatives of VOPEs to address their enabling environments (e.g. evaluation policies and systems of their national governments), and lessons learned from those experiences.
Abstract 4 Title: Auditing the Evaluation Function at Canadian Federal Agencies
Presentation Abstract 4: The Canadian federal government has had an evaluation policy outlining expectations for the scope and quality of federal program evaluations for many years. Colin has led two audits of the federal evaluation function, in 2009 and 2013, and will discuss the scope and criteria for these assessments.
Audience Level: Advanced

Session Abstract: 

Federal, state, and local governments, foundations, international aid agencies, community-based organizations–all increasingly expect agencies to use evidence and evaluation in decision making: whether to improve program effectiveness and efficiency, support policies backed by evidence, or increase accountability for results. However, it is also widely acknowledged that many agencies lack experience with collecting and analyzing such evidence. Evaluation capacity, i.e., the capacity to both conduct and use evaluation in decision making, involves more than technical competence, yet there are serious measurement challenges to assessing the extent to which evidence informs decisions. This panel aims to discuss different approaches to assessing agency capacity – their features, purposes, and implications. Three presenters will discuss different approaches to assessing agency or organization evaluation capacity and how the purposes and context shaped these approaches. A fourth presenter will discuss one agency’s perspectives on these approaches.

For questions or concerns about your event registration, please contact registration@eval.org or 202-367-1173.

For questions about your account, membership status, or help logging in, please contact info@eval.org.

Cancellation Policy: Refunds less a $50 fee will be granted for requests received in writing prior to 11:59 PM EDT October 6, 2015. Email cancellation requests to registration@eval.org. Fax request to (202) 367-2173. All refunds are processed after the meeting. After October 6, 2015 all sales are final.