Evaluation 2017: From Learning to Action

View Printable Version

Games we can play: experiences and prospects of applying gaming simulations in evaluation practice

Session Number: 2105
Track: Presidential Strand
Session Type: Panel
Tags: Evaluation methods, evaluation use, games, simulations
Session Chair: Karol Olejniczak [Assistant Professor - EUROREG, University of Warsaw]
Discussant: Kathryn Newcomer [Professor - George Washington University]
Presenter 1: Karol Olejniczak [Assistant Professor - EUROREG, University of Warsaw]
Presenter 3: Michał Wolański [associate professor - Warsaw School of Economics]
Presenter 4: Sebastiaan Meijer [Professor - KTH Royal Institute of Technology]
Presenter 5: Sebastiaan Meijer [Professor - KTH Royal Institute of Technology]
Presentation 1 Additional Author: Sebastiaan Meijer [Professor - KTH Royal Institute of Technology]
Presentation 1 Additional Author: Kathryn Newcomer [Professor - George Washington University]
Time: Nov 09, 2017 (03:15 PM - 04:15 PM)
Room: Maryland C

Abstract 1 Title: Definitions and types of games for evaluation
Presentation Abstract 1:

This presentation provides a conceptual framework for a discussion on the perspectives of using gaming simulation for advancing the practice of evaluation.

We start by explaining three main challenges in evaluation practice that could be potentially addressed with the help of games.

Then we define "gaming simulation" and briefly describe this field of practice in relation to domains of public policy and learning. We close our presentation by providing the audience with a typology of games. The four identified types of games form a framework that is used in follow-up presentations in this panel to provide exemplars of games' practical application in evaluation.


Abstract 2 Title: Exemplar A - Measuring effects: assessing competences
Presentation Abstract 2:

Skills and knowledge are notoriously difficult to evaluate, especially when it comes to more complex and dynamic situational competences. This exemplar shows the achievements in using a game for assessing competences in a learning setting. The results from the game were used to formally grade students, leading to an interesting set of methodological and validity questions.


Presentation 2 Other Authors: Filip Tomaszewski - Pracownia Gier Szkoleniowych (PGS)
Abstract 3 Title: Exemplar B - Testing mechanisms of public intervention
Presentation Abstract 3:

Evaluation studies are often expected to provide decision makers with the insight on how certain target groups will react to the public intervention.

This exemplar shows how we can the use games to test complex regulatory solutions. Central government of Poland has been preparing new rules on regulating public transportation in rural communities. They would substantially change complex and fragile dynamic between central government, local municipalities, owners of transportation companies, and passengers. The team of Polish researchers developed and run gaming simulation to test the theory of change anticipated by the authors of two alternative versions of regulation. Set of game sessions with real policy actors also helped to identify unintended effects of planned policy.


Presentation 3 Other Authors: Łukasz Kozak, Wiktor Mrozowski, Michał Pieróg, Igor Widawski
Abstract 4 Title: Exemplar C - Teaching our stakeholders
Presentation Abstract 4:

In evaluations we often try to educate stakeholders of the policy intervention about certain rules, concepts, and regularities of socio-economics reality that determine functioning of public policy.

Instead of explaining this on paper or presentation, we can help stakeholders learn in the most effective way - by experiencing those phenomena at work. The example of game "Fishbanks" shows how stakeholders can learn complicated mechanism behind interventions directed at renewable resource management. We will play the game with all in the audience.


Abstract 5 Title: Exemplar D - Exploring system dynamics
Presentation Abstract 5:

The design of future systems requires reasons into the future about the dynamics of changes applied to the current system, and therefore often use simulations of the past with structural changes to predict the future. Gaming adds the possibility to add dynamic knowledge of expert players to the simulation, extending its validity and usefulness.

Two examples will be discussed: ProRail – the Dutch railway administration - uses gaming to design future railway operations and pre-assess the design. The city of Rome used the Protoworld gaming environment to design its infomobility strategy during the Jubilee year.

Both examples balance between complex and complicated systems as the simulation perspective takes away some of the complexity and converts it into intrinsically comprehensible systems.

Theme: Learning from Others
Audience Level: All Audiences

Session Abstract (150 words): 

This panel explores innovative and dynamically developing practice of gaming simulations. We argue that it can advance our evaluation practice by helping us to address current challenges of our field: (1) measuring effects of policy solutions, (2) explaining change mechanism that underlies successful interventions, and (3) facilitating learning of policy stakeholders.

We start by providing two-dimensional typology of games for evaluation, distinguishing between level of complexity of policy issue, and intended primary learners.

Then we present four exemplars of real-life application: Type A - Games for measuring the effects, Type B - Games for testing mechanisms, Type C - Games for teaching stakeholders, and Type D - Games for exploring system dynamics.

In the last part, we ask our audience to assess, using real-time survey, potential utility of games for advancing evaluation practice. We summarize the discussion by reflecting on perspectives and implications of using gaming simulations in evaluation.


For questions or concerns about your event registration, please contact registration@eval.org or 202-367-1173.

For questions about your account, membership status, or help logging in, please contact info@eval.org.

Cancellation Policy: Refunds less a $50 fee will be granted for requests received in writing prior to 11:59 PM EDT October 16, 2017. Email cancellation requests to registration@eval.org. Fax request to (202) 367-2173. All refunds are processed after the meeting. After October 16, 2017 all sales are final. For Evaluation 2017, international attendees and presenters who encounter complications due to the international travel environment will have up to 30 days after the event to request a refund and submit appropriate documentation. No administrative fee will apply for the international requests. The $50 fee will be waived for registrants who planned to travel into the US and experienced international travel issues.