READING

Panel Comments: Bernice Anderson

Thank you. Evaluation is about change and I will share with you how we are changing at the National Science Foundation (NSF) with regard to how we think about evaluation. As the Foundation keeps pushing the envelope about STEM education evaluation, we always need the help of AEA as NSF aims to have a leadership role in evaluation practices in government. Regarding GPRA and PART, NSF has been effective to date. However, we still need input from AEA to help us maintain our high standards with regard to evaluation.

The first question that we had to address is: how is evaluation policy established in their agency? I am from the Directorate for Education and Human Resources (EHR) and that’s the directorate that has the responsibility for STEM education evaluation. Since 1992, EHR has been trying to deal with the mandate that all of its education programs must be evaluated. This Congressional mandate that started with EHR is now impacting the other research directorates at the Foundation because of OMB’s PART and ACC activities that we will discuss later.

In addition to Congress asking us to evaluate our programs, the National Science Board (NSB) has taken a close examination of our programs NSB has recommended that NSF evaluate its education investment and develop new strategies to increase their impact on the quality of STEM education. It is important to point out that only do they [Board members] want us to evaluate the investment; but they are hoping that, because of these evaluations, we’ll be able to develop some new strategies. This is in line with your 2007 theme of evaluation and learning. We need to learn from these evaluations some insights for moving the frontier of science.

In addition, the Committee on Equal Opportunities in Science and Engineering (CEOSE) has examined our programs focused on equity and diversity and recommended that over the next two years the Foundation should assess the outcomes of our programs and determine their impact on broadening participation and transforming institutions. We are also asked to provide some capacity-building activities; CEOSE recognizes that, even as you require evaluation to move to a higher standard, you need to build capacity in the field. There is a need to provide additional information to our grantees regarding the reporting requirements so that the annual and final reports that they submit are evidence-based.

The second question was: what types of evaluation related input would our agency welcome from AEA? As I said, we are about change, so we need you to help us think through what we are doing correctly and what we else we need to do to improve our performance assessment framework.

While we think what we are doing is correct, there are a lot of new things that are happening in evaluation and we need your thoughts about new directions. For example, help us think through our practices of linking evaluation activities for various stakeholders (program staff, OMB, Congress, etc.).

You’ve already heard about GPRA, you’ve heard about PART and you’ve heard about the R&D criteria used by our Committee of Visitors (COVs). The Foundation has used expert panels for over 25 years as an effective means for assessing the quality of our merit review process, its integrity as well as the results of the award portfolio. Greater emphasis is being placed on examining the outputs and the outcomes of our projects. In the area of program evaluation, we need a lot of AEA help, in terms of being available to give advice about program evaluation designs and be willing even to conduct project -level evaluations that are appropriate and rigorous. Again, we need your input about a comprehensive evaluation activity that looks at implementation, advances the knowledge base and meets the accountability reporting requirements.

We fund independent evaluations through contracts and grants. When an evaluation study is funded as a grant, the primary purpose is to advance the field. However, just because it advances the field does not mean that NSF cannot use that same information to justify appropriate use of its resources.

Overall, EHR supports the multiple-method evaluation approach that uses mixed-methods. We fund studies using both qualitative and quantitative data. We also believe that one study won’t provide you with all the answers; therefore we have a continuum of evaluation activities that is informed by the program development cycle. Most of our program evaluations are conducted for multiple purposes.

We would like for you to help us reinforce in the field that while it is important to conduct evaluations for compliance purposes, evaluations should help our project directors to understand and explain what they’re doing. Also, we want to be better Federal program managers in terms of monitoring; and most of all, we want to learn from project and program evaluations. And then based on multiple evaluations, we should be able to use the evaluation findings to inform our future plans for the efficient use of the program resources. Again, AEA plays a very critical part in evaluation by emphasizing the use of evaluation to advance the knowledge base.

Let’s talk about hierarchy of study designs used by the Academic Competitiveness Council (ACC). The top two tiers (experimental and quasi-experimental designs), according to ACC, define rigorous evaluation. We don’t argue with that, as it relates to an impact study of an intervention. Our issue is that not all of our programs are interventions. Not all of our awards are implementation projects/studies. Because we have different types of projects supported by the Foundation, we need your help to identify a framework that goes beyond the ACC model.

For example, when a program is funding research and development (R&D) projects, should the evaluation framework differ from the ACC model? When do you use or what is the role of the randomized control trials (RCT) in the early stages of development of R&D projects? Our AEA President was most recently at a conference at the Foundation where the discussion focused on projects going through various phases of development and that it was important to not inappropriately use RCTs. We are not saying we’re against using experimental/quasi-experimental designs. They have an appropriate place, but we need to know when the RCT is appropriate in terms of the maturity of the project, as well as the type of the project. We need your support in helping us to define the various types of STEM education projects that we have and what are the appropriate ways to document and assess that a particular project or program has been successful and effective.

The last question was about the way that you can provide input. Whenever we contract a program evaluation, we ask for the contractor to have an advisory group. When you get calls to serve on the advisory panel for a particular program evaluation, I would encourage you to say yes because it would be an opportunity for you share your expertise and to advance the field of evaluation. You may get a call serve as a member of a COV, the Committee of Visitors, as EHR takes a more careful examination of the evaluation effort and the results of an award portfolio. Some of the COVs will have a need for more evaluators to serve on the review team. Whenever we have evaluation planning workshops, I will be encouraging the program officers as well as the evaluation staff to look at AEA as a source for identifying evaluation experts to attend these workshops and/or serve as major presenters. When we begin our next series of evaluation seminars, we’ll be reaching out to you again, hoping that you will be willing to share what you’re learning - how you are advancing the field of evaluation.

This sums it up in terms of the assessment cycle for EHR. Compared to 1992, EHR is in a different place in evaluation with different programs and different needs, but we do think that AEA provides the foundation, the richness, the scholarly information that undergirds the evaluation demands for today and the future. NSF looks forward to your help as we think about innovative evaluation designs, ways to improve our monitoring systems, and increased use of evaluative research and longitudinal studies.

Finally, let’s work with the community of practitioners/managers in terms of building their capabilities to help shape designs and use evaluation results. We often think of evaluation in terms of program change and success. It is important to note that at the Foundation, regardless of the results, a part of a successful evaluation is making certain that program managers know as much about evaluation as they should and know how to use the data/results for multiple purposes. This is a new area and NSF wants your help so that we can be at the forefront of the evaluation/accountability discussion.

Panel comments: Debra Rog