THIAGI GAMELETTER: July 2011

Structured Sharing

One, Two, Four, All

Structured Sharing activities help participants to share and analyze their best practices, knowledge, and opinions. The primary source of information is the participant group; the facilitator's role is merely that of a guide.

In One, Two, Four, All, participants begin by working individually (One). Later, they form partnerships (Two), create a group (Four), and share their ideas with everyone (All).

One, Two, Four, All is a framegame. You can easily pull out the content provided here and plug in your own content. In the Flow section below, the generic rules for Top Tips are printed in regular type. A specific example is printed in italics.

Participants

Any number can play. The best game is for 12-20 participants.

Time

Generally, 10-15 minutes. The exact time requirement depends on the number of participants and the amount of time allotted to each round.

Supplies

• Countdown timer
• Whistle

Preparation

Prior to your training session, come up with an open question dealing with a common challenge usually faced by the participants.

Flow

Work individually on the open question. Announce the question and assign a suitable time limit. Ask each participant to recall successful strategies that she has used in the past. Also invite participants to recall tips that they have heard about or read about. Ask each participant to select the top tip.

Work with a partner. After a suitable pause, blow a whistle and ask each participant to find a partner to share the tips. Warn the partners that they will have to recall and present the other person's tips at a later time. Roam among the partners, eavesdropping on their conversations.

Work in quads. After a suitable pause to permit the sharing of tips between partners, blow the whistle again and ask each pair to join another pair to form a team of four called a quad. Ask all four members of the quad to share the tips from the previous round, each person recalling and reporting her partner's tip.

Select the best tips. Blow the whistle again and ask each quad to select one tip for presentation to the entire group. Recommend that the participants select a practical tip that is unique.

Present to the whole group. Blow the whistle again and randomly select one of the quads. Ask the spokesperson from this quad to present its best tip. Repeat the procedure with a few other quads.

Take your turn. Comment briefly on the tips, adding suitable caveats about their use. Also, if you have a useful tip that is different from those presented by the participants, share them with the group.

Conclude the session. Ask each participant to take a few minutes to recall the different tips and to jot down personal notes about one or two of them for immediate application. Thank all participants for their contributions.

Adjustment

Inconvenient number of participants? The game involves the participants working with a partner and, later, in teams of four. If you have an odd number of participants, join the group or ask one partnership to invite the extra participant to join them. Make similar adjustments when each pair joins another pair.

[Table of Contents]

Review Game

Frenzy

This fast-moving review game was inspired by my visit to the trading floor of the Chicago Board of Trade. I use this game as a review activity near the end of my workshops. It is especially effective with factual content as in the case of product-knowledge presentations.

Participants

6 to 25. Best game is for 12 to 25 participants, organized into groups of four or five.

Time

15-30 minutes.

Supplies

• A large number of question cards, each with a number and a question that requires a short answer.
• An answer sheet with the question card numbers and the correct answers.
• Copies of the handout How to Play Frenzy
• Timer

Flow

Demonstrate the play of the game. Distribute copies of the handout, How To Play Frenzy. Pause briefly while participants read this handout. Ask for three volunteers to come to the front. Explain that you will be the Game Master for this round. Give four question cards to each player. Start the timer and ask your players to yell out card numbers and answers. Demonstrate the procedure by referring to the answer sheet, giving feedback, and replacing the question cards. Stop the game after 2 minutes, help the players to computer their scores, and identify the winner.

Organize groups. Divide participants into two to five groups. Each group should have three to five players. Explain that these groups are not teams: The players in each group compete with each other. In each group, identify a player to be the first Game Master.

Distribute supplies. Give a deck of question cards and a copy of the answer sheet to each Game Master.

Begin the first round. Start the timer and blow the whistle. Ask Game Masters to conduct the game with their group.

Conclude the first round. At the end of 2 minutes, blow the whistle again and ask players to stop. Instruct each player to count the number of correctly answered cards. This is the player's score for the first round. Congratulate the highest scorer in each group for winning the first round.

Conduct the second round. Ask Game Masters to collect all cards, shuffle them, and give the deck to the player on the left. This person is now the new Game Master. Conduct the game as before.

Repeat the procedure. Continue playing additional rounds of the game until every member of each group has had a turn at being the Game Master.

Conclude the game. After the final round, identify the player (or players) in each group who won most rounds. Congratulate these winners.

[Table of Contents]

Handout1

How to Play Frenzy

Receive cards. One of you will be selected to be the Game Master. Get four question cards from this person. Review the questions on these cards and come up with answers.

Answer a question. If you know the answer to any of the questions, yell out the card number and the answer. If you don't know any answer, make a guess. Keep yelling the card number and the answer until you get the attention of the Game Master.

Replace the card. If the Game Master says that your answer is correct, place the card in front of you. If the Game Master says that your answer is incorrect, give the card to the Game Master. In either case, get another question card from the Game Master.

Continue playing. Repeat the process, trying to answer as many question cards as possible within the 2-minute period.

Calculate your score. When the Game Master concludes the round, count the number of question cards that you answered correctly. This is your score for the round. If you have the highest score, you win this round.

Continue playing. New Game Masters will conduct other 2-minute rounds. Repeat the same procedure during each game (except when you are the Game Master).

[Table of Contents]

Jolt

Miss — Understanding
by Tracy Tagliati

Making assumptions is something all of us do everyday. Most of time, it is efficient to make such assumptions based on our prior experiences. However, sometimes these assumptions produce undesirable consequences. This jolt demonstrates what we might miss when we jump to conclusions.

Synopsis

Participants are given a handout and asked to circle each time the letter “e” appears in the text. Most people will circle the letter “e” when it appears at the beginning of a word, and miss the letter “e” when it appears at the end of the word, or in a familiar word like “the”.

Topic

• Communnication
• Making assumptions

Purpose

To demonstrate that the more familiar we are with something the more likely we are to make assumptions and possibly miss important information.

Participants

One or more.

Time

3 minutes for the activity and 5 minutes for the debriefing.

Supplies

• Handout (one for each participant)
• Pen or pencil
• Powerpoint slide that highlights each time the letter “e” appears in the text

Flow

Distribute the handout. Ask each participant to independently read through the handout and circle each time the letter “e” appears in the text.

Stop the activity. Call time after 1 minute. Ask the participants to count the number of times they found the letter “e”.

Display the answer. Show the PowerPoint slide to reveal that the letter “e” appeared in for a total of 58 times. Ask if anyone found all 58 appearances. Congratulate the participants who found the most occurrences of the letter “e”.

Review the handout. Ask the participants to check their handout and count the number of times they circled the letter “e” when it appeared at the beginning of a word. Ask them to count each time they missed finding the letter “e” when it appeared at the end of the word. Also ask them to count the number of times they missed the “e” in a familiar word like “the”. (Most participants would have missed the letter “e” more often when in appeared towards the end of the word or in familiar words like “the”. )

Debriefing

Explain to the participants that the more familiar we are with a word, the less likely we are to read it letter by letter. Instead, we tend to pay attention to the first few letters, consider the context, scan and look for patterns, and then make assumptions about how the word ends.

Ask and discuss the following types of questions:

Sample responses:

Here are some other debriefing questions:

[Table of Contents]

Handout 3

Handout

Instructions: Draw a circle around the letter “e” each time it appears in the following paragraph. Work independently. You have one minute.

As educators we often rely on text-based resources, and this often requires the need for a professional proofreader. Proofreaders are sometimes called typesetters or copywriters, but whatever you call them you will quickly come to realize that these people are professionals who take pride in their work. Your trusted proofreader will do their level best to catch every spelling or grammar mistake and make sure your work is error free.

[Table of Contents]

Slide 1

Slide

As educators we often rely on text-based resources, and this often requires the need for a professional proofreader. Proofreaders are sometimes called typesetters or copywriters, but whatever you call them you will quickly come to realize that these people are professionals who take pride in their work. Your trusted proofreader will do their level best to catch every spelling or grammar mistake and make sure your work is error free.

[Table of Contents]

Improv

Function Junction
by Tracy Tagliati

It is important to have original ideas; it is equally important build on the original ideas of others. This improv activity provides the opportunity for appreciating and building on the ideas of others.

Purpose

• To practice supporting, connecting, and extending the ideas of others.
• To foster cooperation and collaboration.

Participants

Best with groups of four or more

Time

5-10 minutes.

Flow

Form groups. Divide the participants into groups, each with four to five members.

Brief the participants. Explain that one member in the group will begin by identifying an object and its function. Give an example: I have a pair a scissors and I use it to cut paper. The second member of the group will jump in and say a sentence about another object with a similar function. Example, I have a knife and I use it to cut vegetables. The other members of the group will then jump in one-by-one and contribute to the established pattern:

Conduct the activity. Ask a group member to say a sentence about an object and its function. Ask the other members to take turns making statements until they seem to have exhausted all ideas.

Begin a new round. If time allows, ask a group member to start a new round with a new object and function.

Here's another example of group members creating a string of functional sentences:

Variation. Provide participants with prompts to begin the round. Write these prompts on individual pieces of paper. Ask a participant to pick out and read a prompt. Ask the other members to jump in with ideas and build upon it. Here are some examples of prompts:

[Table of Contents]

Guest Gamer

An Interview with Gary Rush

TGL: Gary, how did you get into designing training games?

Gary: What got me into designing games was using them in facilitated workshops. Later, I began teaching these games to my participants. I found that games are critical with groups to help participants grapple with complex issues, conceptual ideas, and sometimes, just teambuilding.

TGL: How long have you been using training games?

Gary: I have been using games since the mid 1980s when I began to teach participants how to facilitate. I began with case study and roleplay games, adding additional games as I became more comfortable with them.

TGL: When do you use training games?

Gary: I use games in my training as well as in my facilitation workshops. The games are similar except that in my training sessions, I want the participants to learn something. In my workshops, the participants need to discover something they already know.

TGL: How do your clients react to training games?

Gary: My clients love the games—just as long as I don't call them games. When I was told that a group of executives wouldn't want to participate in anything that smacked of the warm-and-fuzzy, I guessed that it was because the activities that they had been exposed to previously were facilitated poorly. I also knew that the executives needed games. I had them participate in games (without calling them games) by making them directly relevant to the topic at hand. The executives participated and enjoyed the games. So, it's all in the presentation.

TGL: How do your participants react to the use of games?

Gary: My participants love the games. But they worry about the reaction of their participants. So I recite the story about the reaction of executives. My participants realize that properly framed and if relevant, everyone enjoys games—and they add value.

TGL: What advice do you have to newcomers about the use of games?

Gary: My best advice is to go for it. Try it. When you do, make absolutely certain that the game is relevant and it adds value to what you are doing. Avoid fun for the sake of fun and frame your games properly.

TGL: What types of games do you use frequently?

Gary: My most common games are roleplaying activities and case studies. I use them in all of my training sessions. I also use graphical games in my strategic planning workshops. They really help to simplify complex concepts such as creating mission statements.

TGL: Who are your favorite game designers?

Gary: My favorite game designer has to be Thiagi. It's really because of the thought behind the games that enables me to take his games and adapt them for different situations.

TGL: What book recommendations do you have?

Gary: Besides Thiagi's books, I'd have to recommend the Facilitator's Guide to Participatory Decision-Making, by Sam Kaner. It's an excellent book for group facilitators on how to engage participants. The book includes numerous activities with complete instructions on how to facilitate them.

TGL: What is your prediction about the future of games?

Gary: I think that the future for games is bright. The younger generations have grown up with more games than I did. Our games were low tech and often just for fun. Today's generations are learning from games. These types of games are easily used in face-to-face learning as well as Internet based learning. They give us more options and the audience is more apt to accept them.

[Table of Contents]

Interactive Story

Data Mining
by Gary Rush

Here's a game that can be used in a content-heavy training session that relies on reference manuals. It is a combination of a textra game and an interactive storytelling activity.

Participants

This game is normally used in a class of between 2 and 12 students. Just about any sized group will work.

Time

20-40 minutes, depending on the number of groups.

Supplies

• Reference manuals related to the training topic
• Copies of an unfinished story: Create a story that requires the students to search through the entire reference manual to fill in the missing parts. (See a sample unfinished story that incorporates a reference manual for facilitators.)

Flow

Organize groups. Divide the participants into groups of 1 to 4 people.

Distribute the unfinished story. Give each group a copy of the story. You may use different versions of the story with different groups.

Brief the participants. Explain that the story contains some missing elements. Ask the members of each group to find suitable information from the reference manual to complete the story. Emphasize that there are no wrong answers, but some answers are not as right as others. Ask the groups to note down page references.

Announce a time limit. 15 minutes should be sufficient most of the time. Some groups may finish earlier, but that should not present a problem.

Conclude the session. Call time at the end of the 15 minutes. Convene the groups back into the overall group. Ask each group to read their completed story. At the end of the presentation, conduct a poll to decide which story is the best one.

[Table of Contents]

Handout 2

How I Conducted My Process Modification Workshop

Last week, I was assigned to facilitate a workshop with a group of salespeople in our company.

Here's a profile of the sales team that attended the workshop:

The goal for my workshop was to redo the process that the sales team followed to set sales goals. I planned to implement this type of workshop process:

<Describe your workshop agenda here.>

In preparation for the workshop, I interviewed members of the sales team. Here are the questions that I asked them:

<List the interview questions here.>

Given the profile of the participants, I planned to incorporate these exercises to help them come together as a more cohesive team.

<Describe three or more team exercises.>

I estimated that the workshop will require _______________ hours.

Once I had the agenda planned out, I needed to set up the room. I came up with this request for the furniture, equipment, and room set up:

<Describe your requirements here.>

Given all of this preparation, the workshop went smoothly and produced a new sales process that everyone agreed to.

[Table of Contents]

Say It Quick

Self-Distraction
by Brian Remer

When giving a presentation, your effectiveness begins with clarity about why you are standing in front of a group in the first place—as illustrated in this 99-Word Story.

Self-Distraction

At a Chamber of Commerce workshop, the presenter mentioned several times how relieved he was to have a small class so that he wouldn't have to use PowerPoint. But his presentation consisted of walking us through a paper version of his slides! When questioned later, he explained that he finds PowerPoint “distracting.” “People look up at the screen and not at me!” he said.

Why does he need people watching him? Recognition, affirmation, reinforcement, attention? The workshop shouldn't be about him but about learning.

Take your eyes off the goal and all sorts of self doubt creeps in.

[Table of Contents]

Discoveries

Make a Powerful Point ( http://Prezi.com )
by Brian Remer

Has PowerPoint changed your life? It has mine—and not for the better. Somehow we've gotten to the state where no one seems able to make their point without the “power” of Microsoft. Now, in full disclosure, I am biased against this type of presentation to begin with. I'd rather create opportunities for learning by using metaphors, games, simulations, and discussion. That said, lecturing while showing slides does have its place—if done well. But I've probably seen two decent presentations, maybe.

Fortunately, I don't have to complain about bad PowerPoint presentations any more now that I've discovered Prezi.com, an on-line tool that emphasizes the visual dimension of a presentation.

Using its simple interface, you can easily convert an existing presentation or invent something from scratch. Prezi begins with an infinite canvas to construct your presentation. Of course you can add text in various fonts and sizes, but you can also insert photos, images, video, audio, or PDF documents. Then add motion with panning and zooming capabilities. Combine all these gadgets and you've got the potential to create something useful and memorable.

As with any technology, there are some cautions when using Prezi. You can easily overdo the motion effects and make your viewers seasick with too much zooming and panning. But Prezi will force you to think more visually. You'll find yourself putting thoughts into more succinct bullet lists, adding graphics, and using relationships of size and space to emphasize certain information. Prezi presentations have a lot of similarity to mind maps so they are a nice fit for people who think visually.

Raja says: For more about Prezi, and some sample presentations by Brian, see his original article from the May 2011 Firefly News Flash

[Table of Contents]

Ideas

Hogworts Theory of Learning
by Brian Remer

In the Harry Potter series of books and movies, I believe the professors at Hogworts School of Witchcraft and Wizardry do a poor job of teaching. Typically, they demonstrate a spell then expect their students to replicate it. The teachers never give any real instruction. There are no hints about how to pronounce a spell or wave a wand. Harry and his friends are left to blunder through their classes defending themselves from dangerous creatures while creating comic relief for the rest of us.

The Hogworts Theory of Learning is not all fiction. When my daughter entered high school, her Social Studies teacher insisted his students use PowerPoint for their first group presentation. But students were never given any instruction in how to use this tool without "endangering" everyone in the class. In the end, my daughter waved a laser pointer at pages and pages of text that she had dumped straight into a few slides.

Whenever you dump something whether its rocks, socks, or information, you end up with a mess. Someone needs to sort, categorize, and put everything in its place. That's what learning is: sorting, making linkages, and finding just the right place for new information among everything one already knows. Instead of dumping, set the scene for success.

A friend wanted help designing a day-long workshop. For one section, the client insisted that project evaluators be given time to present their findings. But my friend was concerned that people would get bored during 90 minutes of data presentation.

We began to talk about why this evaluation data was important. What did people need to know? Why did we think they should know it? What did we expect them to do after knowing it? How would participants apply what they learned? With answers to these questions, my friend realized she couldn't simply turn class time over to the evaluators. The participants needed to analyze the data themselves to derive meaning from it.

As teachers, we can't just dump data on students and expect them to learn. But, when we give them the opportunity to make comparisons, ask them to manipulate new concepts, or invite them to play with unfamiliar ideas then we begin to see the sparks fly!

[Table of Contents]

Activities

Choose the Right Wand
by Brian Remer

The 99-Word Story in this issue suggests that one really needs to think about a lot of factors before standing in front of a group. First, what is your purpose? Are you selling, teaching, generating ideas, solving problems, entertaining, or preaching? Then, once you are clear, choose the technology, tactics, methods, media, activities, and discussion questions that will help achieve that purpose.

Which wand will you pull from your training wizard robe and what spell will you cast? Do you have a complex topic to share? Maybe a book club makes the most sense so people can take plenty of time to study and reflect. Are you selling something? Facts and technical specs will be important but so is a good story that gives a compelling reason to purchase your widget. Is learning your objective? Provide information but tie it to a practical application so your participants will use it back on the job.

Whatever your reason for standing in front of a group, you'll always be better off with more than one method of engagement. If you must use PowerPoint, at least take advantage of its visual nature and embed it with memorable graphics that can act as metaphors to stimulate deeper thinking.

Better still, make a Prezi presentation and take advantage of spatial relationships, size, color, and motion to increase the “stickiness” of your message. Start from scratch or convert your pet PowerPoint to a Prezi presentation. When you do, please share it (email Brian)! Good luck and have fun!

[Table of Contents]

Single Topic Survey

To Tell the Truth
by Tracy Tagliati

Do you always tell the truth during training sessions? If you answered yes, you are probably lying. Don't be embarrassed; you're not alone. Studies show that most people lie on average two times a day, and my guess is that some of that is happening during training sessions. It might take on the form of a simple exaggeration to grab the participant's attention, a little white lie to spare others' feelings, a research inexactitude to drive home a point, or a big ol' whopper to get a laugh.

While there may be legitimate reasons for lying, there are some concerns. Experts remind us that as trainers we are often viewed as authority figures, and that title carries with it additional responsibilities. They remind us that that even the occasional act of straying from the truth may have the potential for eroding trust and damaging our credibility with our participants.

What are your thoughts?

Poll Question

As a trainer, is it acceptable to lie during a training session?

• Yes
• No
• It depends

(The poll opens in a new window.)

Open Question

(The survey opens in a new window.)

You may include your name along with your response, or if you prefer, keep it anonymous.

We asked some of our friends and here is what they had to say:

Beth: As a trainer, I am never let the truth get in the way of making a good learning point.

Habib: I am guilty of exaggerating during sessions. For example at the close of my session yesterday I said, “You're the best group I've ever worked with.” Most of time, I do it with a wink, and the participants have fun with it.

Rod: During training sessions, I stay as close to the truth as possible at all times. However, I have been known to grab the participant's attention by taking advantage of the exaggerations of other's. For example, on the topic of “Lying” I might say, “There's some research out there that claims we all lie on average two times an hour. That might be an exaggeration, but it still is worth looking into.”

[Table of Contents]

Survey Results

Road Warrior
by Tracy Tagliati

Last month we asked if you think the life of a road warrior is getting easier.

Here are the results:

(Percentages reflect 47 votes received by June 23, 2011.)

Of those of you who responded, 45% said, “Yes” and 55% said “No”.

We also asked you for your favorite tips, tricks, and tools to make the life of a road warrior easier, and you had some great ones. Here are a few.

Response 5) My favorite on-line site for travel tips is “WebFlyer”. You will find everything you ever wanted to know about how to snag seats, upgrades and get the best deals.

Response 6) Send your luggage ahead by UPS or FedEx. (esp if you fly on an airline where you cannot check for free). It saves time and is less work.
John McDermott
[Note: John wrote several other good suggestions as well.]

Response 7) Try to travel as light as possible — one bag for the hold and 1 item of hand luggage only.
I seldom achieve it, but when I do …. what a difference!
Paul Stuart, Singapore

Response 12) I have to work collaboratively with my team on the road. I found the shared online workspace Huddle is very useful. It allows for have a centralized way to continue our work. Programs like this offer are useful because they offer whiteboards, file sharing, telephone conferencing, and can integrate with calendar programs.

See more readers' responses (including the rest of John McDermott's) or add your own.

Thank you for your responses.

[Table of Contents]

Topical Tweets

Evaluation

Here is a collection of our tweets (that began a couple of months ago) on the topic of evaluation of training products and programs. Follow us (@thiagi and @tracytagliati) for daily tweets on different aspects related to the design and facilitation of training activities.

1. I am re-living my career as an evaluator — and planning to share some useful insights.
2. My preferred definition of evaluation: The process of determining the worth of something.
3. Measurement is a part of evaluation, but it is not the same as evaluation.
4. What is being evaluated can be a product (example: a keyboard), process (example: a game to teach touch typing), or a person (eample: Thiagarajan).
5. All evaluations involve a combination of product, process, and person. They are all inter-related. Don't try isolation techniques.
6. Identify the primary object of evaluation (product, process, or person). Then identify associated elements to be included.
7. Two purposes of evaluation: Formative (to improve something) and summative (to prove something.
8. Formative evaluation is for continuous improvement. It's for tweaking the product, process, or person.
9. Example of formative questions: How can the keyboard be modified for left- handers? How can this process be used with non-native speakers? How can I be happier?
10. Summative evaluation is designed to prove something, to make yes/no, go/no-go decisions.
11. Summative questions: Does this keyboard meet our needs? Does this process produce a significant return on investment? Am I okay?
12. All evaluation has both formative and summative uses. It's dysfunctional to separate them.
13. In evaluation, include both goal-free and goal-based approaches. Measure all types of costs and outcomes.
14. Goal-free approach identifies unanticipated outcomes (positive or negative) and costs.
15. Goal-based and goal-free examples: New incentive system increases sales—and decreases colllaboration.
16. Another example: Participants solve quadratic equations—and hate algebra.
17. Goals and objectives for an intervention should be measurable. Appropriate metrics make evaluation objective and effective.
18. Evaluation compares actual results with the goals for the intervention. This goal-based approach is objective and useful.
19. Sometimes unanticipated outcomes outweigh goal-based results. Got to identify these unanticipated—and relevant—effects.
20. Thoughts on goal-free evaluation: an interesting area, but ignored by evaluators. Michael Scriven wrote about it long time ago.
21. Systems approach supports goal-free evaluation. Sometimes, you fix a problem in one element of the system—and produce negative consequences in others
22. Politicians thrive on goal-free evaluation: “Interventions suggested by the other side will produce a lot of negative consequences.”
23. Goal-free evaluation is critical in large-scale policy interventions. Examples: gun control, capital punishment, assassinations.
24. Example: Evaluation of a 10-million dollar primary education intervention in Liberia. Significant results NOT due to programmed instruction.
25. In the primary education project, significant results were due to free breakfast. This intervention was an afterthought in the project.
26. Hire an evaluator to measure ALL relevant outcomes associated with the intervention. This is the best way to conduct goal-free evaluation.
27. Three models for evaluation: scientific (controlled studies), ethnographic (participatory observation), and legal (jurisprudential).
28. Advantages of the scientific method: The studies are replicable. The results are generalizable.
29. Disadvantage of scientific method. Variables that influence most human performance interventions are not easily controllable.
30. Disadvantage of scientific method: It is difficult to generalize from controlled studies to the chaotic real world.
31. Scientific evaluation uses controls for isolating effects of intervention. Example: One group works receives intervention; other group does not.
32. The ethnographic model for evaluation is derived from anthropology and other social sciences. It uses qualitative techniques.
33. The legal model for evaluation is based o the assumption that the truth emerges through a debate between two opposing points of view.
34. In legal evaluation, different parties collect data in support of an intervention and opposed to it. They conduct a hearing.
35. Legal evaluation can incorporate the scientific and ethnographic models. It follows the rules of conducting administrative inquiries.
36. The legal model for evaluation is the method of choice for evaluating large-scale social interventions.
37. Measurement is a necessary—but not sufficient—element of evaluation.
38. Measurement: This stick is 3.79m long. Evaluation: I cannot use the stick to crush the spider on the ceiling.
39. Measurement gives useful data for determining the worth of an object being evaluated. But it is not the same as evaluation.
40. Measurement: Average score on the test is 64. Evaluation: This course does not produce certifiable mastery of the objectives.
41. Measurement: He is 7' 2" tall. Evaluation 1: He will make a good basketball player. Evaluation 2: He will NOT make a good steeplechase jockey.
42. Systematic observation is a measurement technique. You can observe in real time or through audio or video recordings.
43. Use observation systems to count, classify, and organize observational data.
44. Make better use of observation as a measuring tool through rating scales.
45. Make better use of observation as a measuring strategy through the use of rubrics. Visit http://bit.ly/gCsgmj .
46. Performance tests require actual perform in an authentic context. Objectively score with rating scales. Advantage: Highly valid.
47. Disadvantage of performance tests: Time required for construction, administration, and scoring.
48. Use scenario-based questions for testing. Search for Will Thalheimer's brilliant paper on “simulation-like questions”.
49. Confirmation bias = you see what you expect to see. This reduces objectivity of observation as a measurement strategy.
50. Disconfirmation bias = you ignore what you don't expect to see. Another threat to the validity of observation as a measurement strategy.
51. Are you familiar with the invisible gorilla video? Watch it: http://bit.ly/lqceq8 . Clearly illustrates disconfirmation bias.
52. Invisible gorilla video http://bit.ly/lqceq8 also explains the limitations of goal-based evaluation.
53. Instead of observing someone's performance, you can count or rate the products of such performance.
54. Rating participant's portfolios is a very valid measurement technique.
55. Rating work products provides valid data in a manufacturing situation—whether the product is a hamburger or an automobile.
56. In a service industry, ratings can be applied to work-related products such as complaints or complements or financial records.
57. Rating of work-related products. Advantages: authenticity. Disadvantage: subjectivity.
58. Another measurement strategy: review and analysis of existing reports and records.
59. Examples of review and analysis of existing records as a measurement technique: Back orders, customer complaints, and tweets.

[Table of Contents]