Evaluator Competencies meet Evaluation Use: A global scan

Session Number: 3015
Track: AEA Sponsored Sessions
Session Type: Panel
Tags: Competencies, Evaluation capacity building, evaluation use, international collaboration, use
Session Chair: Susan Tucker [Principal - Evaluation and Development Associates LLC]
Discussant: Susan Tucker [Principal - Evaluation and Development Associates LLC]
Presenter 1: Harry Cummings
Presenter 2: Donna Podems [University of Johannesburg]
Presenter 3: Ziad Moussa [Development Management International]
Presenter 4: Kate McKegg [Evaluator - The Kinnect Group]
Presenter 5: Alexey I Kuzmin [Director - Process Consulting Company]
First Author or Discussion Group Leader: Susan Tucker [Principal - Evaluation and Development Associates LLC]
Time: Nov 08, 2017 (06:15 PM - 07:15 PM)
Room: Wilson C

Other Authors: Donna Podems, donna@otherwise.co.za
Harry Cummings, president@evaluationcanada.ca
Alexey Kuzmin, alexey@processconsulting.ru
Kate McKegg, kmckegg@me.com
Ziad Moussa, ziad@ioce.net
Abstract 1 Title: Canada's Voluntary evaluator credentialing: moving from competency list to demonstrating competence
Presentation Abstract 1:

While many evaluation organizations have developed competency lists, Canada is one of the few to have implemented voluntary credentialing. CES conducted a comprehensive crosswalk of documents such as Canada’s Treasury Board Secre­tariat Competencies for Evaluators, Joint Committee Program Evaluation Standards, AEA’s Guiding Principles for Evaluators, and United Nations Competencies for Evaluators in the United Nations System. The crosswalk identified gaps and overlaps among ex­isting evaluator competencies. After member consultation and expert validation for two years, the CES membership approved the Competencies for Canadian Evalua­tion Practice in 2009. In 2010, CES launched Credentialed Evaluator Program where members prepare portfolios demonstrating their competence, followed by expert panel review CE status, including suggestions for additional preparation. In his role as current President of CES, the panelist will share learnings regarding what competencies are necessary to promote use, relationships with soft skills, value of mixed methods, and communication skills enhancing utilization.


Abstract 2 Title: Evaluator competences in South Africa: learnings about evaluation use and an illustrative case study
Presentation Abstract 2:

South Africa’s government has 71 competencies for their evaluators, however are they the right ones for others? The panelist's leadership experiences (i.e., a leader in AFREA as well as South Africa's national evaluation association,  recent AEA board member service, and a current member of AEA's Competency Task Force) give her a unique perspective.   In 2010, the Department of Performance Monitoring and Evaluation (DPME) was established within the Office of the Presidency. To encourage evaluation use, a utilization-focused approach guides evaluators, with a recognition of systems and an emphasis on learning. The panelist will discuss the shift from a compliance culture to one that has a greater emphasis on improvement, learning, and efficiency. A case study from South Africa’s health and government sector will illustrate at how competencies were used to promote use and influence with evaluation, how competencies played out and concludes by discussing how that influences her current work, and those that she mentors and teaches.

 

 


Abstract 3 Title: Is it time to put together a universal set of competencies?
Presentation Abstract 3:

The Global Evaluation Agenda (Agenda'2020) stresses that evaluation is not a "value neutral" discipline. Furthermore, without use, evaluation will remain an "empty shell". Are there a universal set of competencies related to use?  Besides the five core questions, the panelist will address the following questions from his perspective as President of International Organization for Cooperation in Evaluation (IOCE) and Co-Chair of EvalPartners:

  • Why (and why not) should the AEA competencies be different from those of CES or a VOPE from the Global South?
  • What about the new UNEG competencies?
  • How to put use in the spotlight? Is it time to reach-out to new categories of stakeholders to promote use?
  • How can we build momentum around use by harnessing the global attention around the SDGs?

Abstract 4 Title: Why place valuing, culture, and cultural values at the centre of an evaluation competencies framework? Learnings from the Aotearoa New Zealand Evaluation Association (ANZEA) experience
Presentation Abstract 4:

Evaluators needed to be clearer about who they are professionally. A new set of evaluator competencies from the Aotearoa New Zealand Evaluation Association (ANZEA) emerged in 2011 after two years of development work. These competencies are distinct from other frameworks developed around the world. As a founding member of ANZEA, this panelist will outline the process for developing the competencies as well as the reasoning and powerful exchange of ideas underpinning their development and subsequently infused in the competencies.

Additional questions will address:

  • Why do we believe this approach is key to promoting high quality, culturally sound, responsive, and ethical evaluation practice in Aotearoa New Zealand?
  • Who benefits and who loses in the adoption of evaluator competencies?
  • Are there some competencies that are more essential than others for evaluation use?

Abstract 5 Title: Emergence of independent evaluators in Russia: Consulting skills as an essential part of evaluator competencies
Presentation Abstract 5:

With a growing demand for evaluation services, independent evaluation consulting is emerging in Russia. Transitioning from self-education and learning by doing, Russian evaluators submitted 8 core competencies to the Russian Ministry of Labour in 2014. While similar to those proposed by AEA, one unique area likens evaluation to “consulting” where evaluators are in a position to have some influence over an individual, a group, or an organization, but who have no direct power to make changes or implement programs (Block, 2011). Seeing evaluation through the prism of consulting models allows us to identify (a) practical ways of increasing evaluation use, and (b) competencies evaluators should develop.

From his perspective as a member of IPEN, a regional VOPE in the Newly Independent States since 2000 as well as President of Association of Specialists in Program and Policy Evaluation (ASPPE) this panelist will address the four core questions as well as:

  • How do evaluators to get their knowledge, skills, experience and personal energy used to help others when you have no direct power to make changes?

Theme: Learning About Evaluation Use and Users
Audience Level: All Audiences

Session Abstract (150 words): 

Evaluation associations outside the US have taken the lead on developing the professional status of program evaluators. Countries such as New Zealand and South Africa and Canada as well as regional organizations and global VOPE’s illustrate different pathways this can take. In response to the growing international discussion of evaluator professionalism, the American Evaluation Association’s (AEA) Board approved a Competencies Task Force (CTF) led by Jean King to develop evaluator competencies in 2015. Since Fall 2015, a CTF sub-committee has designed and implemented engagement strategies to support stakeholder dialog across AEA’s membership toward a validation of these competencies. CTF-led focus groups conducted in spring 2016 revealed the criticality of use-related competencies. Panelists representing five alternative pathways to evaluator competency development will share their learnings regarding five questions regarding evaluation use, impact, and utilization. Strategies to engage the physical and virtual audiences in this dialog will use a World Café protocol.