Session Number: 1203
Track: Government Evaluation
Session Type: Panel
Tags: Centralized Government Review, Data Collection, evaluation plan, Evaluation Policy
Session Chair: Nick Hart [George Washington University]
Presenter 1: Nick Hart [George Washington University]
Presenter 2: David J Bernstein, Ph.D. [Senior Study Director - Westat]
Presenter 3: Patricia Moore Shaffer [Deputy Director, Research & Analysis - National Endowment for the Arts]
Presenter 4: Clarisa Rodrigues Coelho [U.S. Agency for International Development]
Time: Oct 27, 2016 (01:00 PM - 01:45 PM)
Room: International South 7
Abstract 1 Title: Evaluation Amidst Paperwork Reduction Act Reviews: Overcoming Design Challenges from a Data Collection Process Evaluators Love to Hate
Presentation Abstract 1:
Evaluators who conduct studies of Federal government programs routinely encounter challenges navigating various data collection protocols and processes, such as Information Collection Requests (ICR) required by the Paperwork Reduction Act (PRA). This presentation will provide contextual information on why processes like the PRA and ICRs exist, how information collection burdens have changed over time, and highlight challenges routinely encountered in navigating such processes. The presentation will then discuss how evaluators can work with the ICR process to achieve target production timeframes, work within existing agency processes, and address and integrate public comments, while improving the overall quality of an evaluation. Data collected from original interviews related to activities of the Environmental Protection Agency (EPA) will be presented in highlighting evaluation challenges and solutions used to address limitations.
Abstract 2 Title: Aye Aye Captain: Navigating Evaluation Data Collection as an External Evaluation Consultant
Presentation Abstract 2:
This presentation will explore data collection issues faced by external evaluators (contractors) to meet evaluation design and data collection needs, including:
• the challenge of meeting client needs that are not always fully conveyed in requests for proposal (RFP) or Question and Answer interactions;
• ensuring validity when cost/time limitations may allow limited instrument pre-testing;
• addressing Institutional Review Board (IRB) issues; and
• working with federal government evaluators to fulfill US Paperwork Reduction Act (PRA) reviews; topics will include the structure of the review process and types of information needed and the complications introduced by time and resource constraints.
The presentation will describe how issues related to federal evaluations can be broadly applied across different types of organizations and include substantive information that will be useful to those who have not gone through an IRB or OMB review process to help build the evaluation capacity of both novice and experienced attendees.
Abstract 3 Title: Challenges in Data Collection for Local Government and Non-Governmental Organizations
Presentation Abstract 3:
Outside of the Federal government setting, evaluators working with other government agencies and nonprofit organizations also encounter challenges navigating various data collection protocols and processes, ranging from Institutional Review Board (IRB) reviews and research clearance processes to the need to educate clients about the rights of human subjects. This presentation will provide an independent evaluator’s perspective on the challenges routinely encountered in collecting data in local government and non-governmental organizations and discuss strategies for successful implementation of evaluation studies in these settings.
Abstract 4 Title: Challenges of Monitoring a Complex High Level Initiative - The Politics of Telling Our Story…
Presentation Abstract 4:
As a federal agency that’s able to collect data without the restrictions of PRA requirements, the U.S. Agency for International Development has a unique set of experiences. This presentation will highlight the challenges of collecting data for a variety of purposes (such as performance monitoring, evaluation, stakeholder communication, project design, and strategic planning) in an highly politicized international setting. The discussion will review examples from one of the Agency’s premier development initiatives, then explore lessons learned from USAID’s progress toward a more data-driven culture. The presentation will conclude with suggestions on how evaluators can help shape data collection practices that are diplomatic and “provider-focused.”
Audience Level: Beginner, Intermediate, Advanced, All Audiences
This panel will highlight challenges evaluators face navigating data collection processes in government and offer suggestions for dealing with complexity from multiple perspectives. The first presentation offers an overview of the relevant processes and insights on challenges in Federal government data collection, including suggestions for working constructively with formal centralized government reviews. The second presentation provides the viewpoint of an external evaluator, including unique challenges faced by contractors working with government agencies. A third presentation from an external evaluator highlights issues that arise when organizations conduct evaluation outside formal review processes. A final presentation provides insights from an internal Federal evaluator and former OMB official in constructively engaging data collection challenges in government. Each presentation will highlight relevant challenges and offer suggestions and opportunities that apply broadly to design issues across the evaluation community.