Questions and Answers
Q: I heard a lot of openness, but I still have two questions. I want to know what would be the best way for AEA to start to develop these relationships. How do we identify these teachable moments? I like that topic but how do we know that? How do we strike when the iron is hot? That’s one question. The second one is what are the cautions? What do we need to be careful about?
George Grob: I’ll venture an answer to the second question. I think that, in terms of the cautions, one of the cautions would be to avoid over-stepping our representative-ness of AEA on particular and detailed matters about which there may be some legitimate differences of opinion. I think we have to be very systematic in how we sort out the advice we might give on very particular questions about methods or approaches at a very concrete level, so that we don’t misrepresent the organization as a whole. There’s not an easy answer to this question as to how we find the common ground that everyone is comfortable with, but I think that’s something we have to do and that’s a caution. I think a related caution, too, is that if we begin to broker the availability of resources to help others, we have to find a way to make sure that we’re confident, that we’re not ending up organizing and offering bad advice, if you will, in a systematic way, so that’s an answer to your second question.
I just may approach the first one, but let others answer as well.
Michael Morris: Does the panel want to comment on Debra’s first question?
Tom Chapel: On the first one, in our case, there are two or three big actual moments, Bernice referred to several tangible moments, ways in which you could serve in roles that we very well may need you for. There I think it’s just a matter of having a mole inside the organization that keeps you attuned to the opportunity. People need some expert advice on evaluation or this panel is being formed of experts and if I can get you in there, would you be willing to comment and speak or serve, or whatever. We’re fortunate at CDC, we have one of the few regional partnerships with AEA every summer, and if we do our job and start making sure that the crowd and the audience at the institute includes people who are users of evaluation, people making performance measurement decisions, that’s a great opportunity for the right people to be up there preaching and stuff. Beyond that, the idea of using our framework review as an opportunity to reflect on what’s good and bad about the utilization focus we’ve had and it creates natural opportunities for experts from the AEA to come in.
George Grob:: Let me add a little more to that. If you pop up a couple of levels and you see who’s setting the policy, which might be staff members and members of Congress, or staff members at OMB, or senior staff in the department, some of whom may not be careerist, but we may see some incoming political appointees, or people of that nature, who have an interest in this topic and I’ve seen that does happen, is that if we can reach those people and we can explain to them that evaluation has come a long way and that we have a lot to offer, they may believe it and they may want to, themselves, advocate for some systematic improvements within the departments where they work or within the laws that they can fashion, if we can show them that there’s value in what we’re offering, they may adopt it.
Bernice Anderson: I think another opportunity will be your dissemination efforts regarding this meeting. Get the word out about what has happened at this '07 meeting in terms of new learning in evaluation and lessons that are still being confirmed. Prepare the document in such a manner that it is very user- friendly for program managers, as well as policy makers. The second suggestion is give more attention to longitudinal studies and follow up studies. I know that NASA is interested in longitudinal studies and I hope that NASA and NSF would partner with or reach out to AEA to address this evaluation need.
Patrick Clark: Popping into my mind is in addition to what I already offered – the Institute for Justice used to, in the Clinton Administration, starting in 1991, we used to have quarterly professional conference series at the JW Marriott in Washington DC and we concentrated in that regard, inviting congressional staff as well as executive branch staff to these one -day, half-day series and perspective lectures in the context of criminal justice. They had an impact in the molding of the '94 Crime Act. There are many Congress people who want their staff to be knowledgeable of things like methods and staying abreast of developments in that regard, and if you look at the schedule of Congress and you schedule on the calendar that works with their calendar, it’s possible to fill a room for a couple of hours with staff who have a break and can come and learn something new and different.
Michael Morris: Questions from the audience?
Q: I guess the question that I have is that as AEA embarks on really trying to begin influencing federal policy, you know we always take a mixed methods kind of stance with respect to evaluation methods and I suspect that with respect to influencing policy we would want to take a mixed method type of stance and I’ve heard that from a number of people, and we have representatives on this panel from three Federal agencies, but I’ve also heard you saying, not only do you have meetings at your own level, but that there are mandates coming at you, you all mentioned OMB, PART and so on. I’m worried about the mix here and what advice, if any, do you have for us as we embark on this about how we should be thinking about…to what extent should we be trying to influence the mandate, versus to what extent we should be providing direct kind of support to people in your position and your agencies and recognizing we’re going to have to do all of these things to some degree. What advice or thoughts do you have for us?
Bernice Anderson: I think you do both simultaneously. You juggle.
Patrick Clark: I'll second that. No, at both levels, seriously, I wasn't kidding. As a lobbyist, I had no money, so if you can’t buy a ticket to the party, what are you going to do? Honestly, to influence those mandators, it was a matter of putting myself in the room and offering testimony and sooner or later, staff would say, “Let’s get that guy here because he might have something to say...”
Tom Chapel: At the CDC, certainly there’s this brush fire approach, the lower level can influence the boss, but the boss pays a lot of attention to what her boss and her boss’s boss are saying. If you get into the head of the OMB examiners, and I know they change on a regular basis, it gives you a broader sense of what evaluation is and you’re a credible spokesman on that. I think that does a lot of good, if you can get in the head of the PART folks and teach them how to apply PART in a case specific way so that it’s less of a blunt instrument, then that would be a great benefit and pay off leverage that then seeps down to the practitioner level.
Michael Morris: We have time for one more question.
Q: The Consortium of Social Science Associations (COSSA) actively lobbies to achieve better legislation in all fields for the social sciences. One of the strategies they use, but not the only strategy, is to scan incoming legislation for provisions related to performance management, accountability, evaluation, and research. This is while the legislation is in committee, and affects budget as well as authorization committees. Is AEA considering such a strategy? (2)
George Grob: That’s an excellent strategy and that is something we definitely need to do and we obviously need to be systematic about it, both for the mechanics of scanning, so you don’t miss things and also the human factor of people…some people are more in touch with that stuff and so we need to organize ourselves so that the people who are in touch can let us know when things are rising, but I think you’re absolutely right about what you’re saying.
Patrick Clark: COSSA, as a lobbying organization, may have the resources that AEA, but I’m not quite sure how you’re putting resources to this effort, may have more resources in that regard, but monitoring legislation is easier than it used to be, you don’t have to sit on a Federal register or on a print news to see what’s coming up in the hopper, so some key term searches, as it were, and you’re making me think that even with part time volunteer work it may be feasible to stay on top of blue backs as they come down the line. I also am prompted to talk about a little bit of, as you brought up COSSA. The Campbell collaboration, some years ago, actually taking off of another idea, like our perspective lectures has been having every spring a little summit on the hill where they invite legislators to come and staff to come in as they talk about a variety of public policy issues that are very important to the legislature as they’re working on these same policies. The difference is that you’re not in a particular subject area, so other than methods, getting an array of human services CDC type, public health, maybe even criminal justice issues to present and hold forums on.
Michael Morris: We have come to the end of our time; I’d like to take an opportunity to say a couple of things. One is to thank your panelists. They did two things that frequently do not happen in sessions. One, they actually responded to the questions that were presented to them. For someone who is as anal retentive about time management as I am, they actually kept to their allotted times for their presentations, I’d like to thank them for that and I would be remiss if I did not offer a special thanks to George Grob. When I say he appeared here on short notice, only my bladder control can tell you how short that notice was that was involved here and I will forever in hid debt for stepping in at this time, so thank you to our panel.
(1) One comment and response were deleted due to limitations in the audio recording.
(2) Since this question was unable to be heard on the recording, the questioner later paraphrased via email.