Evaluating into Lean and Leaning into Evaluation: Lessons Learned from Environmental Evaluation

Session Number: 1921
Track: Environmental Program Evaluation
Session Type: Multipaper
Tags: Environment program evaluation
Session Chair: Terell P Lasane [Lead Program Evaluator - US Small Business Administration]
Discussant: Katherine Dawes [US Environmental Protection Agency]
Presenter 1: Yvonne M Watson [Program Analyst - US EPA]
Presenter 2: Daniel Kaufman [Industrial Economics, Incorporated]
Presenter 3: Terell P Lasane [Lead Program Evaluator - US Small Business Administration]
Presentation 2 Additional Author: Catherine M Foley [Associate - Industrial Economics, Inc.]
Presentation 2 Additional Author: Terell P Lasane [Lead Program Evaluator - US Small Business Administration]
Presentation 3 Additional Author: Yvonne M Watson [Program Analyst - US EPA]
Presentation 3 Additional Author: Daniel Kaufman [Industrial Economics, Incorporated]
Presentation 3 Additional Author: Catherine M Foley [Associate - Industrial Economics, Inc.]
Time: Oct 27, 2016 (04:45 PM - 06:15 PM)
Room: L502

Abstract 1 Title: Evaluation and Lean: From Frenemies to BFFs (Why Evaluators Should Be Familiar with Lean)
Presentation Abstract 1:

A decision to expand EPA’s capacity for Lean while decreasing program evaluation capacity unwittingly created “a sibling rivalry” between Lean and program evaluation within the U.S. EPA’s Evaluation Support Division (ESD). Several developments helped transform this relationship: 1) the AEA 2013 conference Evaluation Practice in the 21st Century challenged ESD to expand its tent to embrace and learn from other tools; 2) an article, Performance Management and Lean: Frenemies? (Jacobson and Chrisinger 2013) reinforced the complementary nature of the tools; and 3) ESD’s first-hand experience with Lean. Since 2013, ESD’s appreciation of Lean and evaluation has blossomed into a body of work that maximizes the strengths and embraces the benefits of each tool to answer different performance management questions. In keeping with this year’s conference theme of Design, this presentation highlights the value of each tool to enhance program design and efficiency and why evaluators should be familiar with Lean.

Abstract 2 Title: Designed for Performance: Leveraging the Best of Lean, Program Evaluation, and Performance Measurement Abstract
Presentation Abstract 2:

This paper discusses synergies among program evaluation, performance measurement, and Lean within the context of an EPA program. Following an evaluation with process and outcome components, program managers sponsored a Lean event to streamline the annual reporting process, which the evaluation identified as ripe for streamlining. The Lean event replaced the traditional annual reporting process with an electronic reporting system, and established a workgroup to identify a core set of Agency-wide performance measures. Workgroup members identified more than 100 potential measures, but had their own diverse perspectives on which measures to prioritize. A breakthrough occurred during a facilitated workshop that combined the participatory elements of Lean with the systematic tools of evaluation, resulting in the workgroup’s unanimous agreement on a core set of measures. This presentation identifies key design elements of the evaluation, Lean event, and measures workshop that empowered EPA staff to redesign the program’s reporting and measurement system.

Abstract 3 Title: Evaluation-Lean Discussion Tool
Presentation Abstract 3:

The US Environmental Protection Agency has recently served client program offices that have conducted both program evaluations and Lean events. Two staff members trained in Evaluation and Administrative Lean have served as the technical advisor/evaluation manager of these dual approach information collections. One presenter will present a tool that may be useful in determining the conditions under which a program evaluation or a Lean process event will be optimally deployed for a program’s use. The second presenter will discuss a tool derived from the application of Lean Waste principles and the synthesis of over 70 program evaluations to determine whether a program is considering evidence-based characteristics of high performing organizations. The pair of tools that highlight the similarities, differences, and synergies of program evaluation and Lean will be presented and discussed within the context of program, evaluation, and information design.

Presentation 3 Other Authors: Dawes, Katherine, US Environmental Protection Agency
Audience Level: All Audiences

Session Abstract: 

In this panel, evaluators working at or with US EPA will illustrate some of the key observations gleaned through our application of program evaluation and Lean. Moreover, we will illustrate that the inextricable elements of program design, evaluation design, and information design are best embodied by a full integrated approach of these tools. One presentation will address how Lean and Evaluation complement and benefit each other. A second presentation will discuss a number of cases studies of programs that have conducted both program evaluations and Lean events at the Agency and will discuss the desirable characteristics from each approach and the refined findings, conclusions, and recommendations that resulted from the dual application of these tools.  Finally, presenters trained in both program evaluation and Lean methods will discuss and present tools that inform the use of Evaluation, Lean, or both for program improvement.