Our mission statement, copyright notice, and cast of characters.

Do It Anyway: Level 4 Evaluation Design
Plan for an evaluation that you are NOT going to do.

Communication Styles
Two sides to every issue.

Hand Game
The winners get beaten by the losers.

Three Useful Books
Sticky ideas, email essentials, and training results.

Cryptic Cluster Puzzle
Obvious Facts About Training
Pay attention to them to make your training more effective.

Brian's Words
Trash or Treasure by Brian Remer
Well-organized garbage.

Check It Out
Rock, Paper, Scissors
All about a simple game.

Single Item Survey
What Do You Call It?
A game by any other name….





To increase and improve the use of interactive, experiential strategies to improve human performance in an effective, efficient, and enjoyable way.

Editorial Roster

Editor: Sivasailam (Thiagi) Thiagarajan

Assistant Editor: Raja Thiagarajan

Associate Editor: Jean Reese

Contributing Editors: Brian Remer and Les Lauber

Editorial Advisory Board: Bill Wake, Matthew Richter, Samuel van den Bergh, and <type your name here>

Copyright Info

The materials in this newsletter are copyright 2007 by The Thiagi Group. However, they may be freely reproduced for educational/training activities. There is no need to obtain special permission for such use as long as you do not reproduce more than 100 copies per year. Please include the following statement on all reproductions:

Reprinted from THIAGI GAMELETTER. Copyright © 2007 by The Thiagi Group, Inc.

For any other use of the content, please contact us ( for permission.

Subscription Info

All registered subscribers receive Thiagi GameLetter free of charge.

However, to prevent us from becoming bankrupt, we have decided to adopt a Busker Protocol. If you like what you read, if you find it useful, and if you'd like us to continue publishing the newsletter, please feel free to chip in with any financial contribution. Our estimated annual cost for this newsletter is $30,000. So we suggest an annual contribution of $30 (which is less than one-third the subscription cost of Thiagi's earlier paper-based newsletter). We would appreciate any amount that you send us, but make sure it is less than $30,000 (since we don't want to make a profit). You can mail your check to Thiagi, 4423 East Trailridge Road, Bloomington, IN 47408 or call us at (812) 332-1478 to charge the amount to a credit card. Or you can charge your credit card online, through The Thiagi Group, Inc. Please let us know if you need an invoice for financial record keeping.

Feedback Request

Thiagi believes in practicing what he preaches. This is an interactive newsletter, so interact already! Send us your feedback, sarcastic remarks, and gratuitous advice through email to . Thanks!


Do It Anyway: Level 4 Evaluation Design

At the beginning of all my training design projects, I quickly and carefully prepare a Level 4 evaluation plan. (This is the type of evaluation that relates training to business results.) I know that conducting Level 4 evaluation is tricky, expensive, and time consuming. Few clients are interested in this type of evaluation, but I plan for it anyway.

Why do I do this? I will explain my reasons after I give you details of a recent Level 4 evaluation design.

Evaluating a recent training design

Recently I did some training for Customer Service Representatives (CSRs) for an insurance company on how to show concern and empathy during telephone conversations with customers. Let me describe my evaluation plan.

WARNING: I have a Ph. D. in research design but I am trying to talk to you in coffee-shop English. If I get too technical and if you get confused, send me an email note ( ) and I will respond to you. If you are a research expert and want to point out the flaws in my design, send me an email. I will not respond to you.

Evaluation strategy

We will use an interview protocol to collect information from customers who had a conversation with a specific CSR during the previous 48 hours. This protocol will contain a set of standardized instructions on how to ask a series of open-ended questions that begin with “Tell me about your conversation with the CSR”. There will be additional follow-up questions. The interview will end with a request for the customer to rate the CSR's performance along the dimensions of courtesy, friendliness, competency, and promptness.

All interviews will be recorded. They will be analyzed by a trained group of listeners who will rate the CSRs' performance based on the customers' comments.

The evaluation design will use a time series design with the interviews being conducted at three different times. We selected this evaluation design because the training workshops will be offered to different groups of employees, 20 employees at a time, once a week during a 10-week period. Customers who spoke to the same 10 CSRs will be involved in each of the three sets of interviews.

Here are some additional details about the timing of these interviews:

Baseline interviews. These interviews will be conducted with two customers each who had talked with each of the 10 CSRs. This interview will be conducted before any CSR received their training.

Mid-course interviews. These interviews will be conducted with two new customers each who had talked with each of the 10 CSRs. Of these 10 CSRs, five would have received training and the other five would not have received training.

Final interviews. These interviews will be conducted with two new customers who had talked with each of the 10 CSRs. At this time, all 10 CSRs would have received training.

Analysis and conclusions

At the end of all three sets of interviews, we will have a total of 60 recorded conversations, 20 each from the baseline, midstream, and final interviews. The analysts will listen to all interviews and rate the CSRs' performance as reflected in the customer's comments. The recordings will be randomly presented to analysts and they will not be told whether or not the CSR had received training before the interview.

Here's what we hope to learn from the analyses:

Comparison between the baseline and the final interviews. Positive differences will show the probable impact of training. If there is no difference (or if there is a negative difference), we will have to question the validity and the effectiveness of the training.

Comparison between the trained and the untrained CSRs during the mid-course interviews. The trained group scoring higher than the untrained group suggests the effectiveness of the training. If there is no difference (or if there is a difference in the opposite direction) we will have to question the validity and effectiveness of the training.

Comparison between the mid-course and final interviews. If the first group shows an improvement in the score, we may attribute it to the impact of additional real-world practice. If there is deterioration in the scores, we may suspect that the novelty effect is wearing off or that there is not enough incentive or management support to sustain the performance improvement.

If the client is willing to provide the resources and time, we will implement this Level 4 evaluation plan. However, in my experience, less than 5 percent of the clients are willing to support this type of evaluation.

So why am I doing it?

I don't think that planning a rigorous Level 4 evaluation is a waste of time, even if it never gets implemented. In the long run, I think that this planning enables the training design to proceed at a faster, cheaper, and better fashion by focusing our efforts on business-related outcomes and reducing the need for revisions.

Planning Level 4 Evaluation provides a concrete goal for the training design team. It helps the SMEs and writers to better understand the rationale for the training project. We can align our training content and objectives to the business results by repeatedly asking ourselves, “How would this help us perform better on our Level 4 evaluation?”

We can share the interview and the analysis procedure with participants in our training session to concretely explain the overall training goal. We can create training activities that prepare participants to get ready for this type of evaluation. All of these activities speed up our instructional design process.

Why not Level 3?

Why don't I just plan for Level 3 Evaluation (measuring transfer of training to the workplace), which has a greater probability of being implemented? Why not just record actual CSR conversations with customers and analyze them to the see how the CSRs are applying their new skills and knowledge? To me, this approach appears to be the instructional equivalent of inviting the fox to guard the chicken coop. Using Level 3 data to predict business results assumes that the behaviors that we are training for will produce desirable business results. By planning for Level 4 Evaluation, we go beyond just what we taught our trainees and actually validating the training content.

Let me repeat

Here's my main message: At the beginning of all training design activities, plan for a rigorous Level 4 Evaluation. Even if your plans do not get implemented, you can use it to align, improve, and speed up your training design activities.


Communication Styles

Different cultures and groups are marked by their preferences along various dichotomies that represent communication styles. Understanding and acceptance of diversity requires the ability to see both sides of a dichotomy. That's what this activity attempts to do.

Key Idea

In groups of three, two participants take turns to present a case for one side or the other of a dichotomy. The third participant acts as the judge and awards points for the presentations.

Index Tags

Diversity. Dichotomy. Three-player games. Presentations. Communication styles.


To take a more objective look at the advantages and disadvantages of both sides of different dichotomies related to communication styles.


Minimum: 3
Maximum: Any number
Best: 12-24

Time Requirement

10 to 30 minutes for the activity (depending on the number of dichotomies explored)

5 to 15 minutes for debriefing


Room Setup

This activity requires plenty of open space, since it is best conducted as a stand-up exercise. If there are chairs in the room, push them along the walls or arrange them into informal clusters of three.


Select a set of dichotomies. Review the reference list of dichotomies associated with communication styles. Select five to 13 of these dichotomies that are likely to influence interpersonal interaction and teamwork among participants. Feel free to select dichotomies that are not included in the list or to modify the words used to identify different dichotomies.


Brief the participants. Explain that differences among groups of people are typically marked by marked preferences among pairs of opposites such as

Explain that you are going to call the communication concepts represented by such pairs of words “dichotomies” and conduct and activity to explore the nature of such dichotomies.

Explore a dichotomy. Ask participants to raise their hands if they personally prefer direct mode of communication to indirect mode. Point out that most people's preference is shaped partly by the groups they belong to and the people they associate with rather than by logical analysis and decision making. Start a short discussion of this dichotomy by asking these four questions:

Some participants may say, “It all depends on the situation.” Accept this response and point out that both modes of communication have some advantages and disadvantages.

Here are some of the statements from a recent play of this game:

Advantages of the direct mode of communication:

Disadvantages of the direct mode of communication:

Advantages of the indirect mode of communication:

Disadvantages of the indirect mode of communication:

Organize triads. Ask participants to form themselves into groups of three.

Distribute the supplies. Give a packet of about 10 index cards to each person. Make sure that everyone has a pen or pencil.

Get ready for the first debate. Announce a pair of opposite words (example: playful — serious) to specify your first dichotomy. Ask all participants to silently think of the advantages and disadvantages (or the positive and negative aspects) of the two sides. Announce a 30-second time limit for this preparation activity.

Assign roles. Blow the whistle at the end of 30 seconds. Identify the tallest person in each triad and ask her to take on the role of a neutral judge. Ask the judge to point to one of the other two people in the triad and give her one of the two words in the dichotomy. (Example: seriousness)

Listen to the first side. Explain that the selected person will have 60 seconds to present a case supporting the concept given to her and attacking the opposite concept. Tell the person to explain the advantages on her side and the disadvantages of the other side. Remind the judge and the other participant in the triad to listen carefully. Start the timer.

Listen to the other side. Blow the whistle to indicate the end of the 60 seconds. Now ask the other person in each triad to present the case for the other side (and against the previous side) for the next 60 seconds. As before, ask the judge and the other person to listen carefully to this presentation.

Score the presentations. Blow the whistle at the end of a minute. Ask the judge to secretly distribute 13 points between the two presentations to reflect their relative credibility and persuasiveness. Instruct the judge to write the appropriate numbers secretly on two index cards (no fractions or negative numbers, please) and place them, written side down, near the feet of each presenter.

Reassign the roles. Explain that for the next round of the game, the person to the left of the previous judge now becomes the next judge.

Repeat the activity. Announce another dichotomy (example: concrete - abstract). Proceed with the game as before: 30 seconds for preparation, 1 minute each for presenting the case for both sides, and time for the judge to secretly score the presentations.

Conclude the game. Conduct the game for at least three rounds so every member of the triad has a chance to play the role of judge. If time permits, continue as many rounds as possible with different dichotomies, making sure that every person has an equal number of opportunities to be the judge.

Determine the winner. After the last round, ask each person to collect the index cards at her feet and the scores. The maximum possible for each person is 13 times the number of index cards; the average score is 7 times the number of index cards. Depending on whether the participants are closer to the competitive or cooperative end of the dichotomy, ask them to announce their scores and identify the winner.

Debrief the participants. Hold a discussion of the dichotomies and dialogue process by asking questions such as the following:

Variations and Adjustments

Too many participants? This should not be a problem with this game because any large group of participants can be divided into groups of three and play the game simultaneously.

Too few participants? No problem. If you have only two participants, you join them to form the triad. If there is only one participant, you and she can present the case for different sides of the dichotomy. Let the participant decide which side she wants to take.

Some participants are left over after you have organized themselves into groups of three. If one participant is left over, ask her to operate as an observer and eavesdrop on the conversations at different triads. If two participants are left over, join them to form a triad.

Want to increase the level of interaction? Instead of playing the game among three individuals, try playing it among three teams. Divide participants into three teams of two to five members each. After announcing the dichotomy, ask team members to spend three minutes discussing among themselves to identify the advantages and disadvantages of both sides of the dichotomy. Appoint one team to be the panel of judges and let them select one of the other teams to make a case for one side in 2 minutes. The team may appoint a spokesperson to present its case or all members of the team may take turns to argue their case. Follow up by giving 2 minutes to the other team to presents its case for the other side.

The selected dichotomies are complex. Explain the two sides of the dichotomy with a mini-lecture that incorporates plenty of examples. Alternatively, give participants a glossary of the dichotomies with examples.

Not enough time? Speed up the process by having participants present their cases immediately after you announce the dichotomy (without any preparation time). Reduce the presentation times to 30 seconds each.

Not enough time to debate all dichotomies? Assign a homework activity for the participants that require them to complete a 2 x 2 matrix with the left-over dichotomies.

Here's a sample homework sheet:

Individualistic Team-oriented

Game Plan for Communication Styles

Step Facilitator Participants
Brief the participants.
(3 minutes)
Introduce the concept of dichotomies in communication styles. Give examples of dichotomies and discuss the two sides of a sample dichotomy. Listen and ask questions. Participate in the discussion.
Organize triads.
(2 minutes)
Ask participants to form themselves into groups of three. Introduce yourself to the other members of the triad.
Prepare for the first set of presentations.
(2 minutes)
Specify a dichotomy. Give instructions. Prepare a presentation on the advantages and disadvantages of both sides of the dichotomy.
Assign roles
(1 minute)
Select a judge in each triad. The judge points to one of the other players and specifies one side of the dichotomy.
Present the cases.
(3 minutes)
Give instructions. Keep time. The selected participant presents her case in support of the side assigned to her. Then the other participant presents the case for the other side.
Score the presentations.
(1 minute)
Give instructions. The judge secretly distributes 13 points between the two participants who presented their cases.
Reassign the roles.
(1 minute)
Give instructions. The participant to the left of the previous judge becomes the judge for the next round.
Repeat the activity.
(3 minutes for each round.)
Identify another dichotomy. Give instructions. Repeat the procedure with new dichotomies. Two participants take turns to present their cases for the two sides of the dichotomy. Judge secretly distributes 13 points between the presentations during each round.
Conclude the game.
(3 minutes)
Announce the end of the game. Give instructions. Identify the winners. Collect all score cards and find the total score.
Debrief participants.
(5 minutes)
Conduct a discussion of the earlier activities and the nature of dichotomies. Participate in the discussion.


Dichotomies in Communication Style

abstract - concrete
accommodating - competing
active - passive
affective - neutral
animated - lethargic
arrogant - humble
assertive - meek
attached - detached
big picture - details
command and control - collaborative and consensual
concrete - abstract
conformist - nonconformist
decisive - indecisive
deductive - inductive
deep - shallow
direct - indirect
egalitarian - hierarchical
either A or B - both A and B
energetic - weak
Exaggerating, hyperbolic - understated, low-key
expressive gestures - no expression or gestures
factual - intuitive
fast - slow
formal - informal
Glitzy, flashy - low key, subdued
harmonious - disruptive
hesitant - unhesitant
individualistic - team oriented
linear - circular
listening - talking
low context - high context
low scan - high scan
materialistic - spiritual
much touch - avoiding touch
objective - subjective
optimistic - pessimistic
organized - chaotic
patient - impatient
personal - impersonal
planned - emerging
playful - serious
rational - emotional
realistic - idealistic
reflective - impulsive
self divulging - secretive
short - long
simple - complex
structured - flexible
taciturn - talkative
taking turns - interrupting
tense - relaxed
terse - long-winded
theoretical - practical
tolerant - intolerant
trusting - paranoid
yes, and - yes, but

Hand Game


Have you ever played the game called Rock, Paper, Scissors? If you haven't I strongly recommend that you try this game. It will take you only a few minutes. Visit this month's Check It Out section and learn how to play Rock, Paper, Scissors. If you don't have any friends to play with, then play the game against a computer opponent.

Rock, Paper, Scissors delivers an important message about the relative nature of power and influence. CEO is a version of Rock, Paper, Scissors that highlights the interesting relationship among customers, employees, and owners.

Key Idea

Players select one of the three types of people (employees, owners, or customers) during each round and determine who won during that round by using the following formula:

Employees “beat” customers because their behaviors can make the customers happy or unhappy (for example, by giving a large discount or by adamantly refusing to work around a meaningless policy).

Owners beat employees because, among other things, they can hire and fire employees.

Customers beat owners because unless customers are satisfied, the owners will go out of business.

During each round of the game, players simultaneously choose one of these three types of people and determine who is the winner.


Minimum: 2
Maximum: Any number, divided into pairs.
Best: 10 to 30


3 to 10 minutes

Room Setup

CEO is primarily a stand-up game. Make sure that pairs of participants can stand up facing each other.


Brief the players. Explain the cycle of power among employees, owners, and customers.

Teach the three letters. Show participants how to display the letter C (for customers), E (for employees), and O (for owners). Here's how to finger spell these three letters:


Demonstrate how to play each round of the game. Ask for a volunteer to play the game with you. For the first round, you and the other player make a fist and hold it about shoulder height. Count “One, two, three”, and during each count lower and raise your fist. After “three”, lower your fist one more time, say “Show” and display one of the three letters. It is important that you and the other player display the letters at the same time so that neither of you can cheat by changing your mind after discovering what letter the letter the other player has chosen.

Explain that the game is to be played in sets of three. At the end of each round, determine the winner according to the formula explained earlier. (If both players spell the same letter, the round ends in a tie.) At the end of each set of three rounds, whoever has won most of the rounds, wins the set.

Conduct the activity. Ask participants to pair up. If you have an odd number of participants, pair up with the person left over. Play the game for a specific period of time (for example, 5 minutes). At the end of this time, whoever has the most points in each pair is the winner. Find out the highest score player among all pairs in the room and congratulate this champion CEO player.

Debrief. Discuss the cyclical nature of power and authority among employees, owners, and customers. Ask participants to come up examples of how each of these groups have influence over the others.


Three Useful Books

To order a book from Amazon, click its cover art below. We receive a small commission if you do this.

[Book Cover] Heath, Chip and Dan Heath (2007). Made to stick: Why some ideas survive and others die. New York, NY: Random House. (ISBN: 978-1400064281)

This practical book presents six key qualities that make ideas stick. These qualities (that almost spell “success”) are simplicity, unexpectedness, concreteness, credibility, emotional, and stories. The brothers Heath walk the talk and provide guidelines based on research results by using a memorable and readable style. The book is interspersed with idea clinics that display “before-and-after” samples of messages to demonstrate the process of making ideas stickier. A useful epilogue lists symptoms of problems and suitable solutions. An easy-to-use reference guide at the back of the book reviews the main ideas from each chapter.

Sample practical suggestion from the book: Use stories for giving people energy to act. To create or select a suitable story, use one of these three templates: Challenge plots to overcome obstacles (as in the case of David and Goliath), connection plots to connect or reconnect with people (as in the story of the Good Samaritan), and creativity plots that inspire us to use a new way of thinking (as in the story of a falling apple leading Newton on to the discovery of the laws of gravitation).

[Book Cover] Shipley, David and Will Schwalbe (2007). SEND: The essential guide to email for office and home. New York, NY: Alfred A. Knopf (ISBN: 978-0-307-26364-3>

Considering that I spend several hours every day composing and sending email, this book is definitely on a relevant topic. It has significantly changed (and definitely improved) my emailing behaviors in a couple of days. The authors are journalists and editors and they have an extremely readable style. The book goes beyond the mechanics of emails and provides excellent guidelines to effective writing. All chapters contain practical, ready-to-use suggestions on important topics including when to—and when not to—use emails, how to make the best use of different parts of an email, and how to write emails to ask, to answer, to inform, to thank, to apologize, and to connect. The chapter on the emotional email explains why people tend to start flaming wars. The chapter gives valuable advice on how to detect and control your anger, sarcasm, and meanness.

Sample practical tip from the book: The subject line is the most important part of an email note. Make sure it is informative and does not sound like spam. Use specific names that are identifiable to the recipient: Don't say “Meeting”; say, “Thiagi's meeting”.

[Book Cover] Wick, Calhoun, Roy Pollock, Andrew Jefferson, and Richard Flanagan. (2006). The six disciplines of breakthrough learning: How to turn training and development into business results. San Francisco, Pfieffer (ISBN: 978-0-7879-8254-6).

This book clearly explains how trainers and instructional designers can partner with business managers to achieve tangible results. Here are the six disciplines: define outcomes in business terms, design the complete experience, deliver for application, drive follow-through, deploy active support, and document results. Throughout the book, the emphasis is on transfer and application of learning to achieve sustainable business results. Useful features of the book include a variety of real-life examples and action points for learning leaders.

Sample practical suggestion from the book: Redefine the trainer's role to include post-workshop responsibilities and results. Pay particular attention to the support from the participants' managers.

Cryptic Cluster Puzzle

Obvious Facts About Training

A cryptic cluster puzzle is a combination of a word association test and a cryptogram. The puzzle displays a list of items that belong to the same category. The items are coded with a substitution code in which every letter of the alphabet is consistently replaced by another letter.

See an example of a cryptic cluster puzzle.

Read how to solve cryptic clusters.

Here's a new cryptic cluster puzzle based on your inputs to last month's single item survey. All the items in this list are obvious facts about training. Try your hand at decoding the 10 items.

10 Suggestions For Becoming A Facilitative Trainer

------- -- --- --------.
----- ----- -- ---------.
-------- -- --- ---------.
- ------- -- ----- - -------.
-------- ------ ----- -- -------.
----- ---- -- ----- ------- ---- ------.
---'- ----- --- ---- ------- ------ - -----.
-------- -- --- ----- --- -------, --- --- -------.
--- ---- -- -------- -- -- ---- --- ------- ----------.
------- ------- ----------- -- ------; ----------- ------- ------- -- -----.

A PDF version (17,958 bytes) for printing out

A hint

Send Your Solution

Instead of including the solution in this issue, we decided to invite you to email your solution to . Be sure to include your name in this email.

We will print the names of the first ten people who send us the solution. We will also print the correct solution in the next issue of TGL.

Brian's Words

Brian Remer is Creative Learning Director at The Firefly Group ( ). In addition to writing 99 powerful words every month for this column, Brian invents games and interactive strategies to expand learning and deepen insights.

Trash or Treasure
by Brian Remer

Trash or Treasure

“Used bicycle parts—Some junk”

That's the label on a typical cardboard box in Ralph's garage. This barn next to his home where he has lived (and accumulated) for 50 years used to have space for three cars. Today there is barely room for one. The building is stuffed with treasures only Ralph would value. Bent yard tools, charred clothes from a house fire, a child's kindergarten project—all boxed and labeled—fill the space from floor to ceiling. You can find anything.

Unfortunately, it's not enough to be organized if you're still surrounded by junk.


Cryptic Cluster Hint

The first item in the list is “Telling is not training”.

Check It Out

Rock, Paper, Scissors

Rock, Paper, Scissors (RPS) is a game that you will enjoy playing.

If you want to learn how to play the game or if you want to improve your current level of expertise, here are three places to visit on the web:

Wikipedia. The Wikipedia article on RPS can be found at,_Paper,_Scissors . Here, you can learn its name in other languages, and about mating preferences among lizards, the mathematics concept of nontransitivity, Canadian domination at international RPS tournaments during recent years, and use of RPS as an alternative dispute resolution technique.

Play RPS. If you don't have enough friends to play the game with you, visit this page ( ), pick a character, and play RPS with an online opponent.

Rock Paper Scissors Strategy Guide. This page ( ) explains why RPS is not a game of pure chance and presents a standard method for playing tournaments. Read about the preferred opening moves among men and women and statistics about the most frequent choice among tournament winners.

Single Item Survey

What Do You Call It?

A few months ago, we introduced the concept of single item surveys. Read more about this approach in the February 2007 issue of TGL.

The word game has so many trivial, frivolous, and superficial elements associated with it that many trainers do not use it while talking to their clients and their participants.

Here's the single item survey question for this month:

In a training context, what is a more acceptable synonym for the word "game"?

Be selective. Of course, you can identify several synonyms. But limit yourself to a single important alternative.

Here are a few responses that we have already received:

To contribute your response to this question, visit this survey page (opens in a new window) and type your short answer.

Along with your contribution, you may include your name or keep your response anonymous.