This is a blog post by creative producer Hannah Nicklin on methodologies for playtesting – why they’re useful, and how you can use them to guide your own playtesting processes. 

In November 2016 Twisted Tree playtested Forest of Sleep for the first time with a series of people over 2 days in London. It was in some ways similar to the ‘silent playtesting’ that Ed advocates – forcing yourself (as dev) to silently watch people as they play, not interrupting or helping in any way. We did that – but then we followed it up with a new approach which I particularly brought to the team from my background of ‘work-in-progress’ showings in performance.

I come from a theatre and performance background, and as well as working as a creative producer on Forest of Sleep, I’m also a narrative designer/game designer myself, and an academic who teaches at universities like UAL, UEL and Goldsmiths on creative and critical processes in games, performance, and the arts more generally.

Additionally, I believe playtesting is an essential aspect of the game development process. It enables us, the developers, to gather feedback from players to refine elements such as graphics, gameplay mechanics, and overall experience, even for a slot game.

When I work in performance, I work in a part of it particularly called ‘devised’ performance – which is much more like the way a band puts music together, than a composer (who is like a playwright to theatre). In devised perofrmance people set rules and games and make and test material iteratively as part of the process. And a really common part of the process are ‘work in progress’ performances, where the team will test the performance in front of an audience, and see how they react to it. It’s basically a playtest. But because you can’t always see how the audience are receiving it, because you’re often in it – it’s important to talk to the audience afterwards.

This is why post-show discussions are so prevalent. But they’re hard. They can sometimes be great, but more often than not they end up being run off track, or moderated by an artist who doesn’t know what their questions are, or who is asked questions by an audience that make them defensive.

One of the main skills of feedback is knowing how to listen to and give it – but it’s not something we’re all naturally versed in. Which is why I particularly love a system devised by Liz Lerman called Critical Response. It basically is a mechanic for supportive, good feedback. It’s incredibly valuable, I teach it to students, use it in my own practice, and recently ran our 2-day playtest for Forest of Sleep based on it. Ed and the rest of the team found it really useful and supportive for that first-time silent playtest, and so I thought it might be valuable to share it with others in the game design community.

Before I set it out, though, the most important thing is to understand that you should be prepared to be flexible with the Lerman system; work out how it can work for you, and not use it as a series of survey questions, but as a means of opening up a conversation. It’s about facilitating dialogue between designers/artists, peers and audiences/players, not about making you stare at a piece of paper and ask the same exact questions of everyone.

If you want to read it all first hand, do purchase her e-book, here. But otherwise, read on, for how we took the system, and applied it to a videogame playtest.

Lerman first talks about the key roles involved in a good critical response dialogue,  how they should be approached, and then about 4 stages to facilitate the feedback:

The Roles

1. The artist

The artist (for us, specifically, the game designer/game design team) needs to be ready and open to offering their work-in-progress game. That means that they understand the context of the feedback (where they’re at, what they hope it will discuss), that they have questions they want answered, and that they also know what they don’t want feedback on. They need to be prepared to question their work in conversation with other people. This role is not to be underestimated in terms of preparation! It’s not just producing a build: it’s asking yourself things like

• What is it I’m showing and how can I present it usefully?
• Who do I invite to look at it?
• What questions do I have that will help me right now?
• What isn’t it helpful to discuss right now?
• How can I prepare myself to listen usefully – to understand what is useful criticism, and what might be more taste-related or tackling something I haven’t addressed yet?

2. Responders (playtesters)

These people will offer feedback, and should be willing to follow the frameworks for the feedback you’re setting up. They will have been invited depending on what the artist wants to test – are they peers (more articulate at early stage feedback, but not the majority of your audience), are they your target audience (and how can you support them to feel articulate)? Are they new audiences, or regular gamers? They should know why they’ve been invited, feel comfortable about what they need to do, and how the day will work.

3. The facilitator

This can sometimes be the artist/dev, though it’s often better if it’s a neutral person. It’s the facilitator’s job to keep the process in step, and use the process to frame useful questions and responses from the responders. They will lead the responders through 4 basic steps (which we’ll twist slightly for our playtesting context).

The following ‘stages’ of feedback, are what the facilitator should lead the playtester through after they have sat with the game for a period of your choosing (we left people for 30 mins). They are particularly arranged like this because they stop responders offering opinions straight away that could make the artist react defensively, which would stop the discussion, and because they direct the responder’s thoughts in useful directions, and then open it up for feedback and questions that might not have occurred to the artist.

The 4 stages of feedback

1. Statements of Meaning: I usually summarise this as ‘how did the work make you feel?’ you might want to use a different phrase for different genre of game, something like ‘how did it feel to play?’ – but generally this question is about the overall effect of it. It should give you a useful impression of how the game works – the word ‘feel’ is important because it’s subjective, and unconnected to judgements. ‘What did you think’ might tend to focus on problems and perceived solutions, which eventually can be useful, but first up it’s useful to understand the effect it’s having – not the effect people think it should have.  It’s also an easy way to begin (not technically challenging – everyone has a subjective experience) and gives a neat framework to understand the effect/vibe/experience of the game, how it’s working as a ‘big picture’.
2. Artist as Questioner: The artist/dev asks questions about the work. I’ve given examples of what our key questions were below.  After each question the responder can answer, they can express opinions in direct response to the question, but it’s best to not ask for change suggestions here, yet.
3. Neutral Questions: Responders are invited to ask neutral questions of the artist about the work. The artist/dev responds – but try not to apologise, make excuses, or talk at length – it’s the questions which are the useful things here. Questions are neutral when they don’t disguise an opinion. For example, if you are discussing the colour scheme, “Why do you use that weird red?” isn’t a neutral question. “What guided your colour choices?” is.
4. Opinion Time: Responders state opinions, subject to permission from the artist/dev – the permission is basically so you talk about things that are useful, rather than things you can’t or haven’t yet work on. You begin: “I have an opinion about ______, would you like to hear it?” Theartist/dev can say ‘yes’ or ‘not right now’ etc., and doesn’t have to give a reason. It might be someone asks to give an opinion on the sound you’ve used, but it’s all placeholder at the moment, in that case you might say ‘not right now’. 

How did this work for us?

This is an example of how I then turned this into a playtesting framework for Forest of Sleep. This is what I wrote with/for the team.

• Attitude

We need to make sure that we go into the playtest ready to have our views on the game challenged, to listen carefully, and to make sure we ask non-guiding, open questions.

• Who we’ve invited

We have invite a mix of people for this 2 day playtest – we have a lot of time to absorb different kinds of feedback so we have room to speak to different people. This is our first playtest outside our game design peers/immediate friends and family. We’ve invited a mix of game design and interactive media arts professionals, and of ‘non games’ folk – people who we would eventually hope would be a market for our game, but who currently aren’t part of the game design community. They are arts and media or games literate people, including an art critic and mother bringing her 2 young children, to a theatre general manager, and an early 30s tech worker (and games player). There is an age range from 6-late 40s, and 4 women and 5 men, we also made sure the people we invited weren’t all white/English.

• Questions

Guided by the above format, the following plan is what I’m suggesting we use as a guide for conversation. The player will play alone for 20-30 minutes (or until they want to stop) and then this conversation should take about 30-40 minutes following. I encourage us to write it up asap while it’s still in our memories, or record and transcribe. We might all want to listen to the conversations but we should also be careful not to make the playtesters feel overwhelmed by attention. 

Our Plan:

Here are the things they will be told before playing:

• You play only with the mouse, by move, hovering and clicking with the cursor
• We’d like you to play through, trying to work out how to interact with the game
• We’d like you to see what happens, and talk to us afterwards about what you understood, thought and felt, but don’t worry, we’ll guide that conversation.
• The game is in very early stages of story structure, you can take actions, but the journey of the first act won’t conclude. Feel free to start again if you want to.

Questions. It’s important to note this is not a rigid framework but points from which conversations can be had. Explain that there will be 4 phases, make sure they know they can ask questions/offer opinions later on. You might want to return to the game to help ask/answer these.

• How did the game make you feel?/How did you feel about the game?

• (The artist as questioner) each question should be followed up through conversation and supplementary questions.

1. What stories did you tell/discover?
2. Did you understand what to do?
3. What did you understand about how to affect the game?
4. Did you understand the graphical language?
5. What was enjoyable?
6. Were you frustrated or lost at any point?
7. How do you imagine the game would develop?
8. How did you feel about the tooltips?
9. You might want to ask any other specific questions you have here. Spend some time thinking about what they might be.

• What questions do you have about the game? (Encourage neutral Qs) Prompts:

1. Interaction design/mechanics
2. Minute to minute gameplay
3. Art and animation
4. Graphical storytelling
5. Sound design

Permissioned Opinion: This is where we’d love your ideas and opinions about the game, what you might change/work on next, on any aspect of the design/setting/story/art and animation/sound etc. In our case, as it’s a slice of the game, there’s not much we don’t want opinions on right now, so we can probably drop the permissioned bit.

The whole team felt like this worked really well, kept the conversations productive, to time, and we usually found that by the time we got to the end the playtester felt that they had said everything they felt they wanted to, and we had a really useful selection of feedback. I found, when running the process, that I changed up some of the ‘artist as questioner’ questions to suit the experience/previous conversation with the playtester in question, and I also added questions if they came up. It’s good to be responsive there, because a question that works for one person, might not for another.

Stay responsive and conversational as a rule of thumb.

That’s it! I hope you’ve found that useful! And if you’d ever like consultancy in this area, or workshops teaching at your university/college,  Or drop me a line on Twitter.