Four Key Design Elements for Successful Evaluation Use in the Federal Government

Session Number: 1143
Track: Use and Influence of Evaluation
Session Type: Panel
Tags: Evaluation impact, evaluation use, government evaluation
Session Chair: Cassandra Cantu [Engineering Psychologist - USDOT/ VOLPE CENTER]
Discussant: Kathryn Newcomer [Professor - George Washington University]
Presenter 1: Stephen Popkin [USDOT National Transportation Systems Center]
Presenter 2: David J Bernstein, Ph.D. [Senior Study Director - Westat]
Presenter 3: Jonathan Morell [Director of Evaluation - Syntek ]
Presentation 2 Additional Author: Jennifer Hamilton [Senior Study Director - WESTAT]
Presentation 3 Additional Author: Joyce Ranney [Dept of Transportation]
Presentation 3 Additional Author: Melinda Davey [Senior Analyst & Project Manager]
Presentation 3 Additional Author: Cassandra Cantu [Engineering Psychologist - USDOT/ VOLPE CENTER]
Time: Oct 27, 2016 (04:45 PM - 06:15 PM)
Room: International South 2

Abstract 1 Title: History and Context: Four Key Design Elements for Successful Evaluation Use in the Federal Government
Presentation Abstract 1:

The Government Performance and Results Act (GPRA) and the GPRA Modernization Act of 2010 require Federal agencies to assess the manner and extent to which their programs achieve intended objectives. In responding to these requirements, and in an effort to improve program performance, many federal agencies have developed and implemented evaluation units and evaluative functions within their organizations. This presentation will discuss the history and evolution of building evaluation capacity in a Research and Development unit within the Department of Transportation, specifically the implementation of the CIPP evaluation framework, evaluation standards, and four key design elements for building successful evaluations: 1) balancing internal and external evaluations, 2) effectively using contractors for external evaluations, 3) using stakeholder review panels (SRPs) as a key engagement strategy in successful evaluation design, and 4) evaluation reporting strategies for increasing evaluation use.


Abstract 2 Title: External Consultants Roles in US Federal Government Evaluation Design and Capacity Building
Presentation Abstract 2:

The majority of US federal government evaluations are conducted by external evaluators. However, conducting evaluations is just one of the roles that external evaluators can play to help expand and build evaluation capacity in federal agencies. This presentation will focus on some of the challenges of involving contractors within the federal evaluation framework, and offer potential solutions to those challenges. In particular, the presentation will focus on the role that external evaluators can play in building evaluation capacity. The presentation will address four specific areas: facilitating education of federal staff (internal evaluators and program staff) through education and training; addressing resource issues by expanding federal capacity to design and conduct evaluations; help ensure continuous use of evaluation methods and results; and developing increased evaluation capacity by expanding skill sets and methodologies available to support federal evaluation and accountability.


Abstract 3 Title: Evaluation Reporting in Support of Evaluation Utility: Content, Mode, Relationships, and Impact
Presentation Abstract 3:

This presentation describes the role of evaluation in moving a close call accident prevention reporting system from an experimental pilot program, to a routine part of operations at the Federal Railroad Administration. Topics covered include: 1) formal and informal reporting and communication mechanisms that existed between the evaluators and all the relevant stakeholder groups, 2) relationship between the process of stakeholders’ commenting on evaluation findings, and stakeholders providing evaluation data, 3) how evaluation reporting was embedded in a rich network of organizational and personal relationships, and 4) the role of the utility standards as drawn from the Program Evaluation Standards.


Audience Level: All Audiences

Session Abstract: 

Promoting evaluation use in government requires organizational process, human capital, and culture.  “Process”, because attention to evaluation needs to be routinized. “Human capital”,  because people need evaluation expertise to appreciate evaluation. “Culture” because evaluation must be a natural aspect of planning. An agency within the Department of Transportation has been developing and testing a process to institutionalize systematic evaluation practice in a manner that promotes collaboration among the stakeholders. To date most of this effort has been done within a single division, but it is now beginning to spread through the rest of the organization. Two tactics are being employed – engagement with leadership, and a manual designed to act as an “instruction book”, as a repository of evaluation tools, and as an expression of evaluation methodology. This panel will report on progress to date, with a focus on the four key elements needed to further the cause of evaluation use.