THIAGI GAMELETTER: November 2007

Evaluation

The Final Performance Test

The second thing I do at the beginning of a training design project is constructing the final performance test. (For an explanation of the first thing I do, check the article about designing Level 4 evaluation in the September TGL.) I may never have an opportunity to use this test, but the time I invest in constructing it makes the subsequent training design faster, cheaper, and better.

I do not use paper-and-pencil tests as final tests—unless my training deals with some paper-and-pencil activity such as business writing. In most situations, my final tests present authentic scenarios and require actual performance.

What We Planned as a Final Test

Here's an example:

I recently designed a training package for Customer Service Representatives (CSRs) on how to empathize with upset customers during telephone conversations.

This was the plan for our final test:

It will involve scenario-based roleplays over the telephone. This test will present unpredictable telephone calls from upset customers. The participant's performance will be evaluated through an objective checklist.

To create the scenarios, we will work with experienced CSRs, subject-matter experts, and real-world customers. The “final” version of the roleplay performance test will involve 10 different situations that represented different types of internal and external customers complaining about different products and services. We will create five different personality profiles for the customers (ranging from the meek and compliant to the abusive and aggressive). By using these five personalities with each of the 10 situations, we will end up with 50 variations.

To assess the competency of each test-taker, we will ask them to roleplay in the context of two randomly selected scenarios. To ensure approximately equal difficulty level among the random scenarios we will hire improv actors to make the telephone calls. These actors will be provided with the scenarios but not with scripts. They will be asked to improvise the conversation in response to the test-taker's questions, statements, and reactions. They will also be asked to push the participant to a medium level of difficulty (irrespective of the personality profile assigned to them).

What We Actually Did

Because of time and budget constraints, we did not implement these plans for our final performance test. However, we did adopt and adapt aspects of this test-construction plan to make our training design faster and better.

We used the planned test as the operational definition of our training goal. We created several scenarios according to the original plan and incorporated them in different parts of the training session. We created the checklist for rating participants' performances and used it to define the content and objectives of our training.

The Training Sequence

The facilitator walks participants through the rating checklist items and explains appropriate behaviors for demonstrating empathy during a conversation with an upset customer.

The facilitator conducts a roleplay (inviting a participant to play the role of an upset customer) to demonstrate appropriate empathic behaviors.

The facilitator assigns different scenarios to teams of participants. These teams analyze the scenarios and prepare an outline for a simulated telephone conversation. They enact this conversation as an audio drama. After each enactment, participants from the other teams (and the facilitator) provide feedback on which empathic behaviors were demonstrated effectively and which behaviors could be improved in future conversations.

Individual participants use other scenarios to conduct rapid roleplays to improve their fluency with empathic behaviors.

As a final activity, each participant is given a different scenario. A randomly selected participant roleplays a CSR while the facilitator roleplays an upset customer (as in the case of the improv actor in our original plan.)

Suggestion for Training Designers

Early in the training design process, construct an authentic performance test. Use components of this test to operationalize the training goal, objectives, and content. Design your training to enable participants to perform effectively in this test—even if you don't actually use it in the original form.

[Table of Contents]

Guest Gamer

Interview with William Hall

TGL: William, what's your area of specialty?

William: Improvisation training for the stage and using improvisational theories to promote creativity, teamwork, and communication. I also work in the area of business theater where we use the tools of theatre for business events and training.

TGL: How did you get into designing and using games?

William: My best friend tricked me into playing a game—and the results were life changing. She enticed me to guess a short story she'd written. After about 10 minutes, I had guessed the whole thing. All I remember now is that it was a story about Aliens. Afterwards she told me that I had actually written the story with my guesses. She simply answered “yes” when my questions ended with vowels and “no” when they ended with consonants. It was then that I began to understand that creativity could be effortless with the right game or frame.

TGL: How long have you been designing and using games?

William: Since 1986 when we started the improv company in San Francisco.

TGL: Where do you use games?

William: I use games with myself when I am trying to break into a different level of creativity. I also use them with improv classes and corporate training environments. I also use them with Business Theatre. People generally relax if you can introduce a game in the right way, and once relaxed, they can begin to go on a journey of training, of perspective, and of connection.

TGL: How do your clients respond when you start playing games?

William: Well, that's tricky. Once they see the game's ability to engage and produce results they love it. But if you simply tell a client that you're going to play games, they become nervous. Games are perceived as frivolous.

TGL: How do your participants respond?

William: The participants love the engagement and interaction. The way business associates interact with people can be confusing. The rules of engagement are elusive and, if misinterpreted, can be career ending. Games and exercises offer structure to engagement and interaction, making it safe. The safer we can make the room, the more possibilities there are.

TGL: What is the most horrible or embarrassing moment you've had in conducting games?

William: I'll tell this story only because it was a big lesson for me. I was hired to lead a teambuilding workshop for a development team in Silicon Valley. When we arrived on site, my client told me that the team was seriously dysfunctional and that she had to trick them into being there (Clue Number 1). “Don't tell them about the real purpose of the training”, she implored (Clue Number 2). When we led them on a simple exercise to examine communication style and workflow, they acted out and sabotaged the game. (Clue Number 3.) How many clues did I need back then! When we led them on a risk taking exercise, two of them accused us of doing theatre games and left, shouting, “We have real work to do”. That was a tough workshop.

TGL: What advice do you have for newcomers?

William: When you want to design games, get up out of the chair and do it. Avoid talking about it as much as possible. Either design the game or walk through the game you are designing. It will help the design process.

When you are thinking about using games, use them. Use games to engage all participants. Play with them. Encourage the participants to make the connection between the game and the desired outcome. Generally, they will.

To get acceptance of games, frame them as exercises. Frame them as a “starting point”. And use a really good quote such as this one from Albert Einstein: Games are the most elevated form of investigation.

TGL: What do you think is the most important characteristic of facilitators, training games, and participants?

William: Quiet confidence is an important characteristic of a facilitator. Quiet to allow participants to step forward and confidence to help participants relax.

An effective training game produces results that are not only felt and understood but also has a demonstrable return on investment (ROI).

I enjoy a participant who is both willing to try anything and willing to challenge anything.

TGL: What are things that you dislike about facilitators, training games, and participants?

William: I dislike facilitators who tell participants what they should have learned from the activity or ask them a set of rhetorical questions about what they learned.

I dislike training games that have overly long and complicated instructions.

I dislike apathetic participants.

TGL: What types of games do you use most frequently?

William: I like to use games in which small groups of people work together.

TGL: What is your most favorite game?

William: I am very fond of Thirty-Five (and all of its variations).

TGL: Who are your favorite game designers?

William: Thiagi, Keith Johnstone, and Viola Spolin. I also enjoy the openness that I find in the work of Del Close although it is harder to identify the “games” he has designed.

TGL: Do you have any book recommendations?

William: I think Made To Stick has a lot to offer our industry. I think there is a way to take their observations and turn them into exercises. And naturally I recommend The Playbook: Improv Games for Performers.

TGL: What is your prediction about the future of games?

William: I am going to go out on a limb here, and I think that the future of games looks very good. I predict they will be around and useful as long as humans are around and interacting. The brain is a very interesting thing. Sometimes it is able to function in a straightforward fashion (problem? -> solution!). And sometimes it works in an indirect method. Games are the best way to facilitate this very useful type of thinking.

[Table of Contents]

Control Groups

Alone And Together

As a researcher, I was always fascinated by control-group designs in which one group received an “experimental treatment” (for example, a debriefing discussion after an experiential activity) and the other group did not (for example, just participating in the experiential activity without the debriefing discussion). I have applied the control-group approach in my simulation games designed to explore interpersonal concepts.

Key Idea

Participants are divided into an even number of teams. During the first round, some teams solve a sudoku puzzle jointly while members of the other teams solve it individually. During the second round teams solve another sudoku puzzle, reversing the joint and individual approaches.

Index Tags

Teamwork. Independent work. Sudoku puzzles. Control group design.

Purpose

To explore the advantages and disadvantages of working in teams and working independently.

Participants

Minimum: 4
Maximum: Any number
Best: 16 to 40

Time Requirement

20 to 40 minutes

Handouts

• How To Solve a Mini Sudoku Puzzle (PDF; 81,928 bytes)
• First Sudoku: Solve in Teams (PDF; 46,102 bytes)
• First Sudoku: Solve Alone (PDF; 44,746 bytes)
• First Sudoku: Solution (PDF; 29,638 bytes)
• Second Sudoku: Solve in Teams (PDF; 45,734 bytes)
• Second Sudoku: Solve Alone (PDF; 43,516 bytes)
• Second Sudoku: Solution (PDF; 29,294 bytes)
• Debriefing Questions

Supplies

• Pencils
• Countdown timer
• Whistle

Flow

Brief participants. Distribute copies of How To Solve Mini Sudoku Puzzles. Pause while participants read the handout. Answer any questions.

Organize teams. Form an even number of teams (2, 4, 6, 8, 10, …) each with two to seven participants.

Distribute first sudoku for team solving. Give the First Sudoku: Solve in Teams handout to one half of the teams. Each team should receive a single copy of the handout to require team members to work together. Ask the teams to work jointly and solve the puzzle.

Distribute first sudoku for independent solving. Give the First Sudoku: Solve Alone handout to the remaining teams. Each team member should receive a personal copy of the handout to solve the puzzle independently. Ask team members not to talk to each other and solve the puzzle independently.

Monitor the solving activity. Announce a 3-minute time limit for solving the puzzle. At the end of this time, blow a whistle and ask everyone to stop solving the puzzle. Distribute copies of the solution.

Collect reaction data. After a short pause, ask each participant to write down a number from 1 to 5 to indicate her reaction to the puzzle solving experience. Explain that “1” stands for a very negative reaction (including frustration or boredom) and “5” for a very positive reaction (including enthusiasm and a feeling of achievement). Participants can write this number on a corner of the solution sheet.

Distribute second sudoku for team solving. Identify the teams that received the independent-solving handout during the previous round and give each team a single copy of the Second Sudoku: Solve in Teams handout. Ask members of each team to work jointly and solve the problem.

Distribute second sudoku for independent solving. Identify the teams that received the team-solving handout during the previous round and give each member of these teams a copy of the Second Sudoku: Solve Alone handout. Explain that each team member should solve the puzzle independently.

Monitor the solving activity. As before, announce a 3-minute time limit for solving the puzzle. At the end of this time, blow a whistle and ask everyone to stop solving the puzzle. Distribute copies of the solution.

Collect reaction data. As before, ask each participant to write down a number from 1 to 5 to indicate her reaction to the second puzzle solving experience.

Debrief in teams. Distribute copies of Debriefing Questions to each participant. Ask everyone to work with other members of their team and discuss their responses to the questions.

Debrief the whole group. After a suitable pause, get the attention of all participants. Conduct a debriefing discussion using the same set of questions.

[Table of Contents]

Handout 8

Debriefing Questions

Was your reaction to working alone different from your reaction to working in a team?

Why do you think people have different reactions to working alone and working in a team?

Why would some people prefer teamwork to independent work? Why would others have the opposite preference?

What are the advantages of teamwork? What are the advantages of independent work?

In which situations would you prefer to use teamwork? Independent work?

If you were an expert sudoku puzzle solver, how would you feel about working in a team? Working alone?

If you were a newcomer to solving sudoku puzzles, how would you feel about working in a team? Working alone?

[Table of Contents]

Structured Sharing

Presentation Success
by Dominique Fredregill, William Hall, Phil Incorvia, Dan Klein, and Doug Randall

Recently, my friends William Hall and Rebecca Stockley organized a wonderful workshop for improv theater people. We had a great time designing, modifying, and facilitating training games. In one of the workshop activities, we took a list-making framegame and redesigned it to suit different training needs. The team listed above created this game on presentation skills. Since the workshop, I have tweaked (and messed up) the game slightly and used it with a few groups. Here's the current version.

Key Idea

Participants rate their skill levels on different key factors that contribute to the successful presentations. On each of these factors, the low-scoring participants have a conversation with the high-scoring participants on how to improve their skill level.

Index Tags

Structured sharing. Presentation skills. Peer coaching.

Purpose

To acquire and apply practical guidelines for improving your presentations.

Participants

This activity is best suited for participants who have some previous experience in making presentations.

Minimum: 6
Maximum: 50
Best: 12 to 30

Time Requirement

30 to 60 minutes

Handout

Seven Factors That Contribute to Presentation Success

Supplies

• Paper
• Pens or pencils
• Timer

Flow

Make a list. Ask each participant to recall successful presentations that they—and other speakers—have made. Invite them to analyze these presentations and make a list of seven factors that have contributed to their success. Ask participants to do this individually and announce a time limit of 3 minutes.

Replace with the “official” list. At the end of 3 minutes, collect the lists from each participant. Explain that you will review and create a consolidated list of the seven most frequently listed items. To save time, you will use a consolidated list from the items supplied by previous participants. Distribute copies of the handout, Seven Factors That Contribute to Presentation Success.

Score your skill level. Ask each participant to go through the list of seven success factors and rate her current skill level on a 5-point scale (1 - very low to 5 - very high) on each factor. Pause while participants complete this self-assessment task.

Assemble peer-coaching groups. Begin with the first success factor in the list. Ask participants who scored their skill level at 1 (very low skill) and at 5 (very high skill) to come to the front of the room and stand facing each other. Ask members of the low-skill group to ask members of the high-skill group questions related to this success factor. (Example: What types of closing have you used successfully in your presentations?) Ask different members of the high-scoring group to share different practical guidelines in response to the questions.

Repeat the process. Assemble low and high-scoring teams for each of the other success factors. If necessary, use these modifications:

• If there are no very high (5) or very low (1) scores, then use participants who scored themselves at high (4) or low (2) levels.
• If there are no high or very high scores, you act as a high-scoring coach and respond to questions from the very low or low group.

Conclude the session. After working through all seven factors, ask all participants to select one of the practical guidelines from the any of the high- scoring groups. Encourage participants to implement this idea in the near future.

[Table of Contents]

Handout 9

Seven Factors That Contribute to Presentation Success

Purpose. Focus on a clear purpose for your presentation.

Opening. Have a dramatic opening that is relevant to your topic.

Closing. End the presentation on a memorable note.

Interactivity. Keep audience members interacting with each other, with you, and with the topic of your presentation.

Storytelling. Tell relevant stories.

Props. Use different objects to explain and illustrate your topic.

Slides. Use slides sparingly and appropriately.

[Table of Contents]

Jolt

Enjoy and Learn

Here's a jolt that I have been using in my training workshops and presentations. It's based on a technique that I borrowed from my nephew Raja Sooriamoorthi, who is a professor of computer science.

Let me describe the jolt the way I use it. Of course, you must modify it to suit your style.

Sometime during the first half-hour of a workshop (or the first 10 minutes of a presentation), I pause suddenly as if struck by startling epiphany. I say:

I have been carefully observing everyone's behaviors and reading all your minds. I have analyzed the data and discovered the person who is going to learn the most from this training session and enjoy it the most. Before I identify this person, why don't you look around and locate this person. When I count to three, please point to this person.

I pause briefly and count, “one, two, three”. I take a quick look at the different people being pointed at. I also make a note of participants (if any) who point to themselves. I comment:

Your choices are definitely in the right direction. But let me point to the one person who is going to enjoy the session the most and learn the most.

I build up some suspense. I count to three once more and point to—myself. I explain my choice:

I am going to enjoy this workshop and I am going to learn the most. I have the choice to decide what I want to do. I set high expectations and choose to behave in ways that will enable me to transform these expectations into reality.

You too have the choice. I hope you too will choose to behave in ways that make this the most enjoyable and the most useful training experience you have ever had.

Think about that for a few moments.

I pause for about 15 seconds and return to what I was doing before the interlude.

[Table of Contents]

Reading More with Les

Books for Learning About How Adults Learn

To order a book from Amazon, click its cover art below. We receive a small commission if you do this.

Galbraith, Michael W. (editor) (2004). Adult Learning Methods: A Guide for Effective Instruction. Malabar, FL: Krieger Publishing Company. (ISBN: 978-1575242323)

As a disclaimer, let me say that I don't tolerate abstract, academic, theoretical texts. I would rather have a dozen books that are light in theory but strong in application than a single book of the opposite stripe. However, when I discover a book that successfully blends theory and application, I delight in it.

Galbraith has collected nearly two dozen chapters for just such a book. The book is divided into two parts , the first one titled Understanding and Facilitating Adult Learning and the second one titled Methods and Techniques.

The eight chapters of Part One provide enough adult learning theory that enable trainers to update themselves on contemporary thoughts. Self-scoring assessments give readers an opportunity to identify and more fully understand their philosophies regarding adult learning and their training styles. Daniel Pratt's chapter on ethics in our field represents an especially welcome exploration of a frequently ignored element in the training field. Raymond Wlodkowski's chapter on creating motivating learning environments has something for everyone who trains.

The 14 chapters in Part Two provide the thoughtful practitioner a set of best practices and ideas for different instructional methods. Gary Conti and Rita Kolody open Part Two with practical advice for selecting among various methods. Several other chapters are related to each other: Patricia Maslin-Ostrowski and Richard Ackerman have contributed a chapter on the Case Story for example, while Victoria Marsick's chapter is on Case Study. James and Maria Witte have contributed a chapter on Interactive Television, while Barbara White and Cathy Bridwell's chapter focuses more generally on Distance Learning Techniques, and Michael Day focuses more specifically on the topic of Enhancing Learning Communities in Cyberspace. More traditional chapters cover concepts such as learning portfolios, mentoring, learning contracts, demonstration and simulation, and panel discussions. Each of these chapters builds on the traditional foundations, but provides the reader with solid practical tips based on the best thinking in the field.

This book straddles theory and practice well. My copy is pretty marked up in the margins. That alone tells me that the book has pushed the way I think about my training approaches and helped me improve them.

Kroehnert, Gary (2000). Basic Training for Trainers, Third Edition. (ISBN: 978-0074709139)

Gary Kroehnert writes simply, focusing on ideas that can be adapted and applied. I have appreciated it about each of his books I have read. This book follows the same pattern.

Although designed for the novice trainer, Basic Training is useful for the experienced trainer as well. Novices will discover that the book provides useful worksheets and enough information to get started on the basic elements of needs analysis, learning objectives, instructional methods, lesson planning, testing learner comprehension, and dealing with difficult in training rooms. Tenured trainers will find this book a useful reality check, bringing their attention to solid, usable processes and techniques and helping them refocus on good, effective training practice that moves them away from non-value added habits or ideas they may have acquired.

Kroehnert ably demonstrates the creation and management of training budgets, methods for conducting training needs surveys, and handling both overhead and data projectors. Each page contains real-world suggestions and examples.

Kroehnert's book is written at a different level, for a different purpose, than the earlier chapters that Galbraith collected in Adult Learning Methods. Yet these two books complement each other very effectively.

[Table of Contents]

Brian's Words

Built To Last
by Brian Remer

Built To Last

The Incan ruins of Machu Pichu are famous for their massive stonewalls fitted together without mortar. Even after several hundreds of years, the blocks are so tight that there's not the slightest gap between them. What's more, each block has a unique, irregular shape. Some are larger than an SUV!

Why build with such irregular stones when clearly any shape could have been fashioned? Because these walls have withstood earthquakes when walls of standardized block have crumbled.

Helping people fit into the place that's best for them builds a stronger organization than forcing people to conform.

[Table of Contents]

Check It Out

Will Thalheimer's Blog ( http://www.willatworklearning.com/ )

Will is my favorite scholar. Steeped in instructional research, he is on a mission for identifying and correcting myths and lies about the training business.

Will's blog is a collection of research-based commentary on learning, performance, and the industry thereof. Warning: Will has challenged many of my cherished (and stupid) beliefs about training and learning. He is a grown-up version of the know-it-all-kid who sat next to me in seventh grade.

Want an example of Will's delightfully sarcastic style? Check out his commentary, “People Remember 10%, 20%…Oh Really?” Then track the sequels.

[Table of Contents]

Quick Poll

Synonyms for Game

At a recent conference, David Metcalf presented a story of a client canceling a training game design project. The developers resubmitted the project and it was accepted. The only change they made was to change the title of the activity from game to simulation!

In the September issue of TGL, we invited readers to submit a suitable replacement for the term “game”. From the submissions we received, we selected 15 candidates.

Help us select the best alternative by participating in this poll.

[Table of Contents]