THIAGI GAMELETTER: September 2007

Evaluation

Do It Anyway: Level 4 Evaluation Design

At the beginning of all my training design projects, I quickly and carefully prepare a Level 4 evaluation plan. (This is the type of evaluation that relates training to business results.) I know that conducting Level 4 evaluation is tricky, expensive, and time consuming. Few clients are interested in this type of evaluation, but I plan for it anyway.

Why do I do this? I will explain my reasons after I give you details of a recent Level 4 evaluation design.

Evaluating a recent training design

Recently I did some training for Customer Service Representatives (CSRs) for an insurance company on how to show concern and empathy during telephone conversations with customers. Let me describe my evaluation plan.

WARNING: I have a Ph. D. in research design but I am trying to talk to you in coffee-shop English. If I get too technical and if you get confused, send me an email note ( thiagi@thiagi.com ) and I will respond to you. If you are a research expert and want to point out the flaws in my design, send me an email. I will not respond to you.

Evaluation strategy

We will use an interview protocol to collect information from customers who had a conversation with a specific CSR during the previous 48 hours. This protocol will contain a set of standardized instructions on how to ask a series of open-ended questions that begin with “Tell me about your conversation with the CSR”. There will be additional follow-up questions. The interview will end with a request for the customer to rate the CSR's performance along the dimensions of courtesy, friendliness, competency, and promptness.

All interviews will be recorded. They will be analyzed by a trained group of listeners who will rate the CSRs' performance based on the customers' comments.

The evaluation design will use a time series design with the interviews being conducted at three different times. We selected this evaluation design because the training workshops will be offered to different groups of employees, 20 employees at a time, once a week during a 10-week period. Customers who spoke to the same 10 CSRs will be involved in each of the three sets of interviews.

Here are some additional details about the timing of these interviews:

Baseline interviews. These interviews will be conducted with two customers each who had talked with each of the 10 CSRs. This interview will be conducted before any CSR received their training.

Mid-course interviews. These interviews will be conducted with two new customers each who had talked with each of the 10 CSRs. Of these 10 CSRs, five would have received training and the other five would not have received training.

Final interviews. These interviews will be conducted with two new customers who had talked with each of the 10 CSRs. At this time, all 10 CSRs would have received training.

Analysis and conclusions

At the end of all three sets of interviews, we will have a total of 60 recorded conversations, 20 each from the baseline, midstream, and final interviews. The analysts will listen to all interviews and rate the CSRs' performance as reflected in the customer's comments. The recordings will be randomly presented to analysts and they will not be told whether or not the CSR had received training before the interview.

Here's what we hope to learn from the analyses:

Comparison between the baseline and the final interviews. Positive differences will show the probable impact of training. If there is no difference (or if there is a negative difference), we will have to question the validity and the effectiveness of the training.

Comparison between the trained and the untrained CSRs during the mid-course interviews. The trained group scoring higher than the untrained group suggests the effectiveness of the training. If there is no difference (or if there is a difference in the opposite direction) we will have to question the validity and effectiveness of the training.

Comparison between the mid-course and final interviews. If the first group shows an improvement in the score, we may attribute it to the impact of additional real-world practice. If there is deterioration in the scores, we may suspect that the novelty effect is wearing off or that there is not enough incentive or management support to sustain the performance improvement.

If the client is willing to provide the resources and time, we will implement this Level 4 evaluation plan. However, in my experience, less than 5 percent of the clients are willing to support this type of evaluation.

So why am I doing it?

I don't think that planning a rigorous Level 4 evaluation is a waste of time, even if it never gets implemented. In the long run, I think that this planning enables the training design to proceed at a faster, cheaper, and better fashion by focusing our efforts on business-related outcomes and reducing the need for revisions.

Planning Level 4 Evaluation provides a concrete goal for the training design team. It helps the SMEs and writers to better understand the rationale for the training project. We can align our training content and objectives to the business results by repeatedly asking ourselves, “How would this help us perform better on our Level 4 evaluation?”

We can share the interview and the analysis procedure with participants in our training session to concretely explain the overall training goal. We can create training activities that prepare participants to get ready for this type of evaluation. All of these activities speed up our instructional design process.

Why not Level 3?

Why don't I just plan for Level 3 Evaluation (measuring transfer of training to the workplace), which has a greater probability of being implemented? Why not just record actual CSR conversations with customers and analyze them to the see how the CSRs are applying their new skills and knowledge? To me, this approach appears to be the instructional equivalent of inviting the fox to guard the chicken coop. Using Level 3 data to predict business results assumes that the behaviors that we are training for will produce desirable business results. By planning for Level 4 Evaluation, we go beyond just what we taught our trainees and actually validating the training content.

Let me repeat

Here's my main message: At the beginning of all training design activities, plan for a rigorous Level 4 Evaluation. Even if your plans do not get implemented, you can use it to align, improve, and speed up your training design activities.

[Table of Contents]

Framegame

Communication Styles

Different cultures and groups are marked by their preferences along various dichotomies that represent communication styles. Understanding and acceptance of diversity requires the ability to see both sides of a dichotomy. That's what this activity attempts to do.

Key Idea

In groups of three, two participants take turns to present a case for one side or the other of a dichotomy. The third participant acts as the judge and awards points for the presentations.

Index Tags

Diversity. Dichotomy. Three-player games. Presentations. Communication styles.

Purpose

To take a more objective look at the advantages and disadvantages of both sides of different dichotomies related to communication styles.

Participants

Minimum: 3
Maximum: Any number
Best: 12-24

Time Requirement

10 to 30 minutes for the activity (depending on the number of dichotomies explored)

5 to 15 minutes for debriefing

Supplies

• Index cards
• Pens or pencils
• Timer
• Whistle

Room Setup

This activity requires plenty of open space, since it is best conducted as a stand-up exercise. If there are chairs in the room, push them along the walls or arrange them into informal clusters of three.

Preparation

Select a set of dichotomies. Review the reference list of dichotomies associated with communication styles. Select five to 13 of these dichotomies that are likely to influence interpersonal interaction and teamwork among participants. Feel free to select dichotomies that are not included in the list or to modify the words used to identify different dichotomies.

Flow

Brief the participants. Explain that differences among groups of people are typically marked by marked preferences among pairs of opposites such as

• Cooperative - competitive
• Assertive - meek
• Direct - indirect
• Individualistic - team-oriented
• Optimistic - pessimistic

Explain that you are going to call the communication concepts represented by such pairs of words “dichotomies” and conduct and activity to explore the nature of such dichotomies.

Explore a dichotomy. Ask participants to raise their hands if they personally prefer direct mode of communication to indirect mode. Point out that most people's preference is shaped partly by the groups they belong to and the people they associate with rather than by logical analysis and decision making. Start a short discussion of this dichotomy by asking these four questions:

• What are the advantages of the direct mode of communication?
• What are the disadvantages of the direct mode of communication?
• What are the advantages of the indirect mode of communication?
• What are the disadvantages of the indirect mode of communication?

Some participants may say, “It all depends on the situation.” Accept this response and point out that both modes of communication have some advantages and disadvantages.

Organize triads. Ask participants to form themselves into groups of three.

Distribute the supplies. Give a packet of about 10 index cards to each person. Make sure that everyone has a pen or pencil.

Get ready for the first debate. Announce a pair of opposite words (example: playful — serious) to specify your first dichotomy. Ask all participants to silently think of the advantages and disadvantages (or the positive and negative aspects) of the two sides. Announce a 30-second time limit for this preparation activity.

Assign roles. Blow the whistle at the end of 30 seconds. Identify the tallest person in each triad and ask her to take on the role of a neutral judge. Ask the judge to point to one of the other two people in the triad and give her one of the two words in the dichotomy. (Example: seriousness)

Listen to the first side. Explain that the selected person will have 60 seconds to present a case supporting the concept given to her and attacking the opposite concept. Tell the person to explain the advantages on her side and the disadvantages of the other side. Remind the judge and the other participant in the triad to listen carefully. Start the timer.

Listen to the other side. Blow the whistle to indicate the end of the 60 seconds. Now ask the other person in each triad to present the case for the other side (and against the previous side) for the next 60 seconds. As before, ask the judge and the other person to listen carefully to this presentation.

Score the presentations. Blow the whistle at the end of a minute. Ask the judge to secretly distribute 13 points between the two presentations to reflect their relative credibility and persuasiveness. Instruct the judge to write the appropriate numbers secretly on two index cards (no fractions or negative numbers, please) and place them, written side down, near the feet of each presenter.

Reassign the roles. Explain that for the next round of the game, the person to the left of the previous judge now becomes the next judge.

Repeat the activity. Announce another dichotomy (example: concrete - abstract). Proceed with the game as before: 30 seconds for preparation, 1 minute each for presenting the case for both sides, and time for the judge to secretly score the presentations.

Conclude the game. Conduct the game for at least three rounds so every member of the triad has a chance to play the role of judge. If time permits, continue as many rounds as possible with different dichotomies, making sure that every person has an equal number of opportunities to be the judge.

Determine the winner. After the last round, ask each person to collect the index cards at her feet and the scores. The maximum possible for each person is 13 times the number of index cards; the average score is 7 times the number of index cards. Depending on whether the participants are closer to the competitive or cooperative end of the dichotomy, ask them to announce their scores and identify the winner.

Debrief the participants. Hold a discussion of the dichotomies and dialogue process by asking questions such as the following:

• Do you believe that people who belong to different cultures and different groups may look at each of these dichotomies differently?
• Are there any dichotomies about which all human beings will have the same preference?
• How did your personal preferences impact on the way you prepared and presented your case?
• How difficult was it to come up with positive points about the side that you personally dislike? Similarly, how difficult was it to come up with negative points about the side that you personally like?
• Which was easier: Making a case for a side or making a case against a side?
• Which dichotomy was the most difficult for you to think about and prepare your case?
• Did you get some new insights as a result of this activity? Which of these insights occurred during the preparation time and which ones while listening to the other person?
• Do you think that people's preferences for one side or the other of a dichotomy is due to logical thinking or due to social pressure? Why do you think so?
• How would you feel if you keep reflecting on both sides of each dichotomy without taking a stand? What would other people feel about you?
• What are some other dichotomies that could be used during future rounds of this game?

Variations and Adjustments

Too many participants? This should not be a problem with this game because any large group of participants can be divided into groups of three and play the game simultaneously.

Too few participants? No problem. If you have only two participants, you join them to form the triad. If there is only one participant, you and she can present the case for different sides of the dichotomy. Let the participant decide which side she wants to take.

Some participants are left over after you have organized themselves into groups of three. If one participant is left over, ask her to operate as an observer and eavesdrop on the conversations at different triads. If two participants are left over, join them to form a triad.

Want to increase the level of interaction? Instead of playing the game among three individuals, try playing it among three teams. Divide participants into three teams of two to five members each. After announcing the dichotomy, ask team members to spend three minutes discussing among themselves to identify the advantages and disadvantages of both sides of the dichotomy. Appoint one team to be the panel of judges and let them select one of the other teams to make a case for one side in 2 minutes. The team may appoint a spokesperson to present its case or all members of the team may take turns to argue their case. Follow up by giving 2 minutes to the other team to presents its case for the other side.

The selected dichotomies are complex. Explain the two sides of the dichotomy with a mini-lecture that incorporates plenty of examples. Alternatively, give participants a glossary of the dichotomies with examples.

Not enough time? Speed up the process by having participants present their cases immediately after you announce the dichotomy (without any preparation time). Reduce the presentation times to 30 seconds each.

Not enough time to debate all dichotomies? Assign a homework activity for the participants that require them to complete a 2 x 2 matrix with the left-over dichotomies.

Here's a sample homework sheet:

	Individualistic	Team-oriented
Advantages
Disadvantages

Game Plan for Communication Styles

Step	Facilitator	Participants
Brief the participants. (3 minutes)	Introduce the concept of dichotomies in communication styles. Give examples of dichotomies and discuss the two sides of a sample dichotomy.	Listen and ask questions. Participate in the discussion.
Organize triads. (2 minutes)	Ask participants to form themselves into groups of three.	Introduce yourself to the other members of the triad.
Prepare for the first set of presentations. (2 minutes)	Specify a dichotomy. Give instructions.	Prepare a presentation on the advantages and disadvantages of both sides of the dichotomy.
Assign roles (1 minute)	Select a judge in each triad.	The judge points to one of the other players and specifies one side of the dichotomy.
Present the cases. (3 minutes)	Give instructions. Keep time.	The selected participant presents her case in support of the side assigned to her. Then the other participant presents the case for the other side.
Score the presentations. (1 minute)	Give instructions.	The judge secretly distributes 13 points between the two participants who presented their cases.
Reassign the roles. (1 minute)	Give instructions.	The participant to the left of the previous judge becomes the judge for the next round.
Repeat the activity. (3 minutes for each round.)	Identify another dichotomy. Give instructions. Repeat the procedure with new dichotomies.	Two participants take turns to present their cases for the two sides of the dichotomy. Judge secretly distributes 13 points between the presentations during each round.
Conclude the game. (3 minutes)	Announce the end of the game. Give instructions. Identify the winners.	Collect all score cards and find the total score.
Debrief participants. (5 minutes)	Conduct a discussion of the earlier activities and the nature of dichotomies.	Participate in the discussion.

[Table of Contents]

List1

Dichotomies in Communication Style

abstract - concrete
accommodating - competing
active - passive
affective - neutral
animated - lethargic
arrogant - humble
assertive - meek
attached - detached
big picture - details
command and control - collaborative and consensual
concrete - abstract
conformist - nonconformist
decisive - indecisive
deductive - inductive
deep - shallow
direct - indirect
egalitarian - hierarchical
either A or B - both A and B
energetic - weak
Exaggerating, hyperbolic - understated, low-key
expressive gestures - no expression or gestures
factual - intuitive
fast - slow
formal - informal
Glitzy, flashy - low key, subdued
harmonious - disruptive
hesitant - unhesitant
individualistic - team oriented
linear - circular
listening - talking
low context - high context
low scan - high scan
materialistic - spiritual
much touch - avoiding touch
objective - subjective
optimistic - pessimistic
organized - chaotic
patient - impatient
personal - impersonal
planned - emerging
playful - serious
rational - emotional
realistic - idealistic
reflective - impulsive
self divulging - secretive
short - long
simple - complex
structured - flexible
taciturn - talkative
taking turns - interrupting
tense - relaxed
terse - long-winded
theoretical - practical
tolerant - intolerant
trusting - paranoid
yes, and - yes, but

[Table of Contents]

Hand Game

CEO

Have you ever played the game called Rock, Paper, Scissors? If you haven't I strongly recommend that you try this game. It will take you only a few minutes. Visit this month's Check It Out section and learn how to play Rock, Paper, Scissors. If you don't have any friends to play with, then play the game against a computer opponent.

Rock, Paper, Scissors delivers an important message about the relative nature of power and influence. CEO is a version of Rock, Paper, Scissors that highlights the interesting relationship among customers, employees, and owners.

Key Idea

Players select one of the three types of people (employees, owners, or customers) during each round and determine who won during that round by using the following formula:

Employees “beat” customers because their behaviors can make the customers happy or unhappy (for example, by giving a large discount or by adamantly refusing to work around a meaningless policy).

Owners beat employees because, among other things, they can hire and fire employees.

Customers beat owners because unless customers are satisfied, the owners will go out of business.

During each round of the game, players simultaneously choose one of these three types of people and determine who is the winner.

Players

Minimum: 2
Maximum: Any number, divided into pairs.
Best: 10 to 30

Time

3 to 10 minutes

Room Setup

CEO is primarily a stand-up game. Make sure that pairs of participants can stand up facing each other.

Flow

Brief the players. Explain the cycle of power among employees, owners, and customers.

Teach the three letters. Show participants how to display the letter C (for customers), E (for employees), and O (for owners). Here's how to finger spell these three letters:

C
E
O

Demonstrate how to play each round of the game. Ask for a volunteer to play the game with you. For the first round, you and the other player make a fist and hold it about shoulder height. Count “One, two, three”, and during each count lower and raise your fist. After “three”, lower your fist one more time, say “Show” and display one of the three letters. It is important that you and the other player display the letters at the same time so that neither of you can cheat by changing your mind after discovering what letter the letter the other player has chosen.

Explain that the game is to be played in sets of three. At the end of each round, determine the winner according to the formula explained earlier. (If both players spell the same letter, the round ends in a tie.) At the end of each set of three rounds, whoever has won most of the rounds, wins the set.

Conduct the activity. Ask participants to pair up. If you have an odd number of participants, pair up with the person left over. Play the game for a specific period of time (for example, 5 minutes). At the end of this time, whoever has the most points in each pair is the winner. Find out the highest score player among all pairs in the room and congratulate this champion CEO player.

Debrief. Discuss the cyclical nature of power and authority among employees, owners, and customers. Ask participants to come up examples of how each of these groups have influence over the others.

[Table of Contents]

Bookshelf

Three Useful Books

To order a book from Amazon, click its cover art below. We receive a small commission if you do this.

Heath, Chip and Dan Heath (2007). Made to stick: Why some ideas survive and others die. New York, NY: Random House. (ISBN: 978-1400064281)

This practical book presents six key qualities that make ideas stick. These qualities (that almost spell “success”) are simplicity, unexpectedness, concreteness, credibility, emotional, and stories. The brothers Heath walk the talk and provide guidelines based on research results by using a memorable and readable style. The book is interspersed with idea clinics that display “before-and-after” samples of messages to demonstrate the process of making ideas stickier. A useful epilogue lists symptoms of problems and suitable solutions. An easy-to-use reference guide at the back of the book reviews the main ideas from each chapter.

Sample practical suggestion from the book: Use stories for giving people energy to act. To create or select a suitable story, use one of these three templates: Challenge plots to overcome obstacles (as in the case of David and Goliath), connection plots to connect or reconnect with people (as in the story of the Good Samaritan), and creativity plots that inspire us to use a new way of thinking (as in the story of a falling apple leading Newton on to the discovery of the laws of gravitation).

Shipley, David and Will Schwalbe (2007). SEND: The essential guide to email for office and home. New York, NY: Alfred A. Knopf (ISBN: 978-0-307-26364-3>

Considering that I spend several hours every day composing and sending email, this book is definitely on a relevant topic. It has significantly changed (and definitely improved) my emailing behaviors in a couple of days. The authors are journalists and editors and they have an extremely readable style. The book goes beyond the mechanics of emails and provides excellent guidelines to effective writing. All chapters contain practical, ready-to-use suggestions on important topics including when to—and when not to—use emails, how to make the best use of different parts of an email, and how to write emails to ask, to answer, to inform, to thank, to apologize, and to connect. The chapter on the emotional email explains why people tend to start flaming wars. The chapter gives valuable advice on how to detect and control your anger, sarcasm, and meanness.

Sample practical tip from the book: The subject line is the most important part of an email note. Make sure it is informative and does not sound like spam. Use specific names that are identifiable to the recipient: Don't say “Meeting”; say, “Thiagi's meeting”.

Wick, Calhoun, Roy Pollock, Andrew Jefferson, and Richard Flanagan. (2006). The six disciplines of breakthrough learning: How to turn training and development into business results. San Francisco, Pfieffer (ISBN: 978-0-7879-8254-6).

This book clearly explains how trainers and instructional designers can partner with business managers to achieve tangible results. Here are the six disciplines: define outcomes in business terms, design the complete experience, deliver for application, drive follow-through, deploy active support, and document results. Throughout the book, the emphasis is on transfer and application of learning to achieve sustainable business results. Useful features of the book include a variety of real-life examples and action points for learning leaders.

Sample practical suggestion from the book: Redefine the trainer's role to include post-workshop responsibilities and results. Pay particular attention to the support from the participants' managers.

[Table of Contents]

Cryptic Cluster Puzzle

Obvious Facts About Training

A cryptic cluster puzzle is a combination of a word association test and a cryptogram. The puzzle displays a list of items that belong to the same category. The items are coded with a substitution code in which every letter of the alphabet is consistently replaced by another letter.

See an example of a cryptic cluster puzzle.

Read how to solve cryptic clusters.

Here's a new cryptic cluster puzzle based on your inputs to last month's single item survey. All the items in this list are obvious facts about training. Try your hand at decoding the 10 items.

10 Suggestions For Becoming A Facilitative Trainer

YDHHNUC NM UQY YZPNUNUC.
------- -- --- --------.
DWDZV CZQOK NM RNJJDZDUY.
----- ----- -- ---------.
YZPNUNUC NM UQY KZDPIANUC.
-------- -- --- ---------.
P YZPNUDZ NM JNZMY P MYORDUY.
- ------- -- ----- - -------.
YZPNUNUC MAQOHR JQIOM QU ZDMOHYM.
-------- ------ ----- -- -------.
UDWDZ FPLD PU PROHY HDPZUDZ JDDH MYOKNR.
----- ---- -- ----- ------- ---- ------.
RQU'Y YZPNU YQQ HQUC TNYAQOY YPLNUC P SZDPL.
---'- ----- --- ---- ------- ------ - -----.
YZPNUNUC NM PHH PSQOY YAD HDPZUDZ, UQY YAD YZPNUDZ.
-------- -- --- ----- --- -------, --- --- -------.
YAD CQPH QJ YZPNUNUC NM YQ FPLD YAD HDPZUDZ MOIIDMMJOH.
--- ---- -- -------- -- -- ---- --- ------- ----------.
IQUYDUY TNYAQOY NUWQHWDFDUY NM SQZNUC; NUWQHWDFDUY TNYAQOY IQUYDUY NM JHOJJ.
------- ------- ----------- -- ------; ----------- ------- ------- -- -----.

A PDF version (17,958 bytes) for printing out

A hint

Send Your Solution

Instead of including the solution in this issue, we decided to invite you to email your solution to thiagi@thiagi.com . Be sure to include your name in this email.

We will print the names of the first ten people who send us the solution. We will also print the correct solution in the next issue of TGL.

[Table of Contents]

Brian's Words

Trash or Treasure
by Brian Remer

Trash or Treasure

“Used bicycle parts—Some junk”

That's the label on a typical cardboard box in Ralph's garage. This barn next to his home where he has lived (and accumulated) for 50 years used to have space for three cars. Today there is barely room for one. The building is stuffed with treasures only Ralph would value. Bent yard tools, charred clothes from a house fire, a child's kindergarten project—all boxed and labeled—fill the space from floor to ceiling. You can find anything.

Unfortunately, it's not enough to be organized if you're still surrounded by junk.

[Table of Contents]

Hint

Cryptic Cluster Hint

The first item in the list is “Telling is not training”.

[Table of Contents]

Check It Out

Rock, Paper, Scissors

Rock, Paper, Scissors (RPS) is a game that you will enjoy playing.

If you want to learn how to play the game or if you want to improve your current level of expertise, here are three places to visit on the web:

Wikipedia. The Wikipedia article on RPS can be found at http://en.wikipedia.org/wiki/Rock,_Paper,_Scissors . Here, you can learn its name in other languages, and about mating preferences among lizards, the mathematics concept of nontransitivity, Canadian domination at international RPS tournaments during recent years, and use of RPS as an alternative dispute resolution technique.

Play RPS. If you don't have enough friends to play the game with you, visit this page ( http://www.playrps.com/index.html ), pick a character, and play RPS with an online opponent.

Rock Paper Scissors Strategy Guide. This page ( http://www.chessandpoker.com/rps_rock_paper_scissors_strategy.html ) explains why RPS is not a game of pure chance and presents a standard method for playing tournaments. Read about the preferred opening moves among men and women and statistics about the most frequent choice among tournament winners.

[Table of Contents]

Single Item Survey

What Do You Call It?

A few months ago, we introduced the concept of single item surveys. Read more about this approach in the February 2007 issue of TGL.

The word game has so many trivial, frivolous, and superficial elements associated with it that many trainers do not use it while talking to their clients and their participants.

Here's the single item survey question for this month:

In a training context, what is a more acceptable synonym for the word "game"?

Be selective. Of course, you can identify several synonyms. But limit yourself to a single important alternative.

Here are a few responses that we have already received:

• Training activities
• Non-computerized group decision support systems
• Modified Delphi technique

To contribute your response to this question, visit this survey page (opens in a new window) and type your short answer.

Along with your contribution, you may include your name or keep your response anonymous.

[Table of Contents]