Talk:TRC Gimmick Rallye Evaluation Cards
I don't want to "mess up" the draft, but am thinking we could separate (into separate sections)
* scoring, and awards presentation efficient? (moved close to end) * material clear and legible? * registration and CP workers competent?
- Dean
The rallyemaster doesn't have much control over registration, so we shouldn't count that as part of the "best rallye" awards (although it may still be useful feedback). It would be good to ask about the beginners school too, although we shouldn't expect upper division rallyists to answer that question. And we shouldn't count it when someone other than the rallyemaster gives the class.
I like the idea of putting the questions in rough chronological order (i.e., moving scoring and awards presentation towards the end).
And as we mentioned last night, very few people can offer meaningful feedback on whether protests were handled fairly. Perhaps a better indicator would be how often the protest committee overrules the rallyemaster.
- Darin McGrew 10:51, 14 November 2007 (PST)
I did many of the tasks that we had been talking about. See what you think.
- Dean
I recommend leaving the existing form alone, except for the changes to the "Length". 10 Categories is sufficient for the feedback that we need.
I am not sure if many rallyemasters use the form as it exists today as a vehicle to improve or change rallies. As a consequence, making changes to the form is likely not going to do much. If we add more categories, rallyists may be less likely to fill out the form.
A better use of time might be for rallyemasters to use the existing form to create a post-mortem of their event that includes 2 topics:
a) What worked well b) What can be improved upon.
Such a post-mortem should be optional, but would be nice to include in the TRC minutes (if it is done).
What to Cut?
Okay, we've got 13 sections now. What do we cut/consolidate to get back to 10 (or fewer)?
Darin McGrew 18:20, 21 November 2007 (PST)
I think 13 sections and 15 easy-to-answer questions might not be too much, if it fits nicely on a half page.
- Dean
Since I was involved in the creation of this form in the '80s, I have some bias. I like the 10 category/100 point total format. In fact, the form was developed to fit this format. Also, I do not know how you will get overall scores if you go with a disagree/agree format.
I do agree that without written comments, the completed forms do not provide much value. Since these forms do not seem to be used by TRC for quality control or user feedback (just isn't working out), TRC might consider getting rid of the evaluations altogether. Are they really useful?
Using Incomplete Evaluations
I also propose that each question get scored by those who answer that question, so partial evaluations are used.
- Dean
If we include topics that not everyone will be able to answer, then we'll need to do something reasonable with "incomplete" evaluations. So, how should we handle the math for a rallye's total score while counting incomplete evaluations?
It's easy enough to average the responses we get in each category, and it's
okay if some categories have more data points than others. But do we calculate
the total score by adding the average scores for each category? I can't think
of a better way, but this still seems clunky to me.
- Darin McGrew 10:50, 28 November 2007 (PST)
If we end up adopting the agree/disagree format, then we could assume that blank statements are either "N/A" or "neither agree nor disagree".
I can see arguments for either, depending on the specific statement.
- Darin McGrew 12:11, 6 December 2007 (PST)
Making Evaluations Useful
Personally, I don't find the evaluation forms useful, except for the comments written on them (in the margins, or on the back). The numbers are purely arbitrary, and apparently even our worst rallyes are well above average (the Lake Wobegon effect), since they score above a 5 in all categories.
Maybe it would be more useful to have a number of statements (e.g., "Traffic was not a problem.") and ask whether they agree strongly, agree somewhat, neither agree nor disagree, disagree somewhat, or disagree strongly? Or maybe we could use a 7-point scale: agree strongly, agree, agree somewhat, etc.
- Darin McGrew 10:57, 28 November 2007 (PST)
I think this might be the best approach of all. It has the benefit of simplifying the questions, too. One of the complaints I've heard about the present eval card is that the questions cover slightly disjoint subjects, so the competitor has to average out things in their head.
- Steve 12:24, 2 December 2007 (PST)
Okay, I took a stab at creating a new evaluation card that uses the agree/disagree format. What do you think?
- Darin McGrew 13:26, 3 December 2007 (PST)
Email Feedback
Teresa suggested rephrasing the "clear, concise, and accurate" question so we get less negative feedback from rallyists who just missed an expert gimmick. I noted that gimmick rallyes often deliberately use unclear, verbose language, and that perhaps "clear" and "concise" are not criteria we should rate.
Teresa also emphasized that each rating should cover only one thing, since rallyists enter ratings like "7/8/10" on the current form when a single rating covers multiple topics.
I'd like a way to assess how well we're handling protests, but the only metrics I can think of are the total number of protests, the number of protests granted by the rallyemaster, and the number of protests granted by the committee. Including a question on the evaluation form seems pointless, because most people have no idea how well protests were handled.
A general question is whether this is a rallye evaluation, or a rallyemaster evaluation. Consensus appears to be that it is a rallye evaluation, and that we can use it for other purposes (e.g., rallyemaster awards) by ignoring the data that is irrelevant for those other purposes (e.g., questions about things like registration that the rallyemaster has no control over).
We need to figure out what to do with incomplete forms. Maybe we can just treat the incomplete questions as "N/A" or as "neither agree nor disagree".
We also need to decide whether to toss the top 10% and bottom 10%. I think this will be less necessary with the Agree/Disagree format.
For a "best rallyemaster" award, we could have a scale of 1-5 (strongly disagree to strongly agree), with 20 questions that are counted.
That gives a maximum score of 100 points.
Then we could round out the evaluation form to 25 or 30 questions, adding questions that won't count towards the award, but which we as a club care about.
- Darin McGrew 12:21, 12 December 2007 (PST)
Purpose
There seems to be concern that the evalation forms unfairly penalize the rallyemaster or rate them unfairly for either events out of their control (e.g. a parade in the middle of a rallye), or registration (pretty much a standard operation the past few years).
A few comments:
1) Unforeseeable events are in fact part of the rallye experience, and events such as parties, parades, rain, and wind should be evaluated, even if they are out of the rallyemaster control. I don't think the rallyemaster is being evaluated, but the event is being evaluated.
2) With Steve and Nancy leaving the registration duties, a rallyemaster interested in getting ranked high at the end of the year will need to consider how to ensure registration is handled efficiently.
3) The Registration is really a small part of Administration. Administration includes the rallye school, the attitude of rallye personnel, the finish process, protests, etc - all of which are under control of the rallyemaster.
Clear & Concise!
I see a movement to eliminate the term "clear, concise and accurate". Actually, I think "clear, concise and accurate" is what differentiates the good rallyes from the rest and states the issue quite well. A small, tight, accurate, and unambiguous set of instructions makes for a better rallye and results in less protests.
I think in cases where rallyists put down several numbers on the eval form (e.g. 7/8/10) we can create a procedure to just average the numbers.
Process
With the increasing concern over the use of evaluation forms as a way to rate the quality of rallye for a year-end awardc, and also given we are no longer a part of NCSCC (and therefore subject to their processes), I think we might need to look at some of the evaluation processes such as:
1) Throw out top 10% and bottom 10% of the forms. Let's keep them all.
2) Incomplete forms (rallyists only fill-out a subset of the categories). I recall seeing this as the biggest issue. In the past we just threw out the evaluation. I am a little mixed on this as we do have some useful information. We could use what we do get and average it in with the rest of the numbers.
3) Mixed numbers for a single category - I say we just average the numbers. I don't recall ever seeing this occur on an evaluation form, but I could imagine it occurring.
I remember historically, a non TRC rallyemaster doing the scoring each year for NCSCC events, and his events always won top-dog, sometimes by a few hundredths of a point. His events were good, but I did always question the transparency of his processes.
Current Status
At last night's business meeting, we decided to go with the agree/disagree format, and to finalize the details before the January rallye via email.
- Darin McGrew 12:53, 19 December 2007 (PST)
I was happy with almost all edits, but sorry to see this one go:
- The maps in the critique were easy to understand.
Map technology (hence, maps) vary widely.
I hope to use my notes to make my edits in the next day or so.
- Dean
Overall, I really like the edits Cris made, although I did a bit of copy editing myself, and restored a couple of the questions he cut. The questions I restored are ones that we can ignore for the "best rallyemaster" award, but I'd like to include something about the rallye school at the start, and something about protests.
I'd still like to include something about the driving distance, but I think 2 questions about the length of the rallye are sufficient, and I'd rather distinguish between "too long" and "too short" than between time required and distance driven.
It might be nice to add something about maps, but I think
- After reading the critique, I understand the gimmicks I missed.
pretty much covers it. The maps aren't an end to themselves; their purpose (if they are used at all) is to help rallyists understand the rallye, especially the parts they missed.
-Darin McGrew 17:16, 19 December 2007 (PST)
FWIW, the current version has 24 questions, including the following could be ignored for the purposes of a "best rallyemaster" award:
- Registration was efficient.
- What we learned at tonight's beginners introduction (rallye school) helped us finish and enjoy the rallye.
- After reading the protest committee's comments, I understand their decisions.
I don't see another question to cut, but if we really want a max score of 100, then we could pick one to ignore for the "best rallyemaster" award.
But overall, 2 dozen questions seems about right.
Purpose of Form
A few times I have heard comments to the effect the rallyemaster is being rated. Let's remember, it's the rallye and rallye experience that is being rated and not the rallyemaster.
This rating may include items seemingly out of the rallymasters control, such as weather, parades, parties, registration, emergencies, etc. However, what IS in the rallyemaster's control is how they handle such unplanned events.
Yes, the form is gathering information about the entire rallye.
However, for the purposes of a "best rallyemaster" award, we may (or may not) wish to ignore information about parts of the rallye that the rallyemaster cannot control.
- Darin McGrew 17:42, 19 December 2007 (PST)
Final Format?
What is the final format of the evaluation card going to be like?
I expected a half page like our current one, with each statement on its own line. Either before or after the statement, would be the numbers 0-5, like this:
| Overall Impressions | |
|---|---|
| The rallye was fun. | 0 1 2 3 4 5 |
| There was a good balance of easy and difficult gimmicks. | 0 1 2 3 4 5 |
| The rallye included new, innovative ideas and/or an interesting theme. | 0 1 2 3 4 5 |
| The rallye was well planned and "polished". | 0 1 2 3 4 5 |
or like this:
| Overall Impressions | |
|---|---|
| 0 1 2 3 4 5 | The rallye was fun. |
| 0 1 2 3 4 5 | There was a good balance of easy and difficult gimmicks. |
| 0 1 2 3 4 5 | The rallye included new, innovative ideas and/or an interesting theme. |
| 0 1 2 3 4 5 | The rallye was well planned and "polished". |
Do we want to have them write a number instead, like this:
| Overall Impressions | |
|---|---|
| The rallye was fun. | ______ |
| There was a good balance of easy and difficult gimmicks. | ______ |
| The rallye included new, innovative ideas and/or an interesting theme. | ______ |
| The rallye was well planned and "polished". | ______ |
- Darin McGrew 17:38, 19 December 2007 (PST)
I think numbers would be better
Let's Leave the Evaluation Form: "As is"
After trying to reformat and adjust the evaluation form, my recommendation is to leave it as it is today (e.g. no changes).
As we try to get more information from rallyists, or ask more questions, the form becomes more complex or at a minimum is more time-consuming to fill-out. The more complex or time-consuming any survey, the less likely you are to get *any* useful information. If you've ever tried to fill out the surveys that hotels send you after a stay you will know what I mean: I'll answer 5 - 10 multiple choice questions, but not the 25 - 50 questions they want me to answer. In a sense - less is more.
It's simply not worth it on the part of the rallyist to spend a lot of time providing the detailed information we think might be useful. At the end of the rallye, I think most rallyists want to spend 20 - 50 seconds on an evaluation, and then adding what they consider important information to the back.
As I was experimenting with the evaluation form to make it more "rallyist friendly", I began approaching the current form in terms of simplicity and clarity. As a result, I began to question whether or not changing the evaluation form would really provide any more benefits.
Do rallyemasters really not know what they would do better in their next rallye based upon the evaluation forms, their experience talking to rallyists, and the general "buzz at the finish? Also, when you look at the year-end results, would anyone have any fundamental disagreements with the rallye-of-the-year placements and rankings such that a new form would improve upon?
We decided on Tuesday to switch to the new agree/disagree format. I think we should do that.
I think we can have more than 10 agree/disagree statements, and still have an evaluation card that is easier/faster to complete than our current evaluation card. Deciding whether you agree or disagree with a simple statement is easier than picking a number between 1 and 10 for a category described by several statements.
I've thought about your comments, and I've realized that part of the problem is that everyone is holding on to their pet questions. Dean wants something about maps, I want something about the rallye school and protest committee, and on and on.
Cris, the edits that you made yesterday were great!
You pruned 35 questions down to about 20 questions.
We need to let go of our pet questions and our pet distinctions, and be willing to merge statements that are too closely related, and be be willing to drop statements that aren't going to give us useful information anyway.
- Darin McGrew 16:00, 20 December 2007 (PST)
I pretty much agree with Darin that we've agreed to change the eval card format. I think circling an answer is easier than writing my own, and is easier to read, even if it has been altered in ink after a change of mind.
I would also not replace two Qs about distance/duration with one. If it gets bad marks from different scorers, one might think the rallye too long and other too short.
- I/We had enough time to finish the rallye without feeling rushed.
- I/We had enough to do without finishing too early.
Also sorry to see a Q about the rallye school go. This is normally performed by the rallye master (or someone the RM picks).
- What we learned at tonight's beginners introduction (rallye school) helped us finish and enjoy the rallye.
I know there isn't much point in one editor ripping things out and another putting them back in. What we need is a concensus, but the wiki doesn't lend itself to voting. We could make a table with a line for each Q and a place for names or initials on each side of the issue.
- Dean
I like your edits of the top of the form. Thanks!
I like circling answers myself. Circling also helps us avoid answers like 3.5 or 1.75. But numbers would require more horizontal space than a blank to write a number, and the current layout is pretty tight.
I think we need to be pretty brutal about editing this down. We can easily end up with 40+ questions, and as Cris pointed out, less is more. We need to keep it short enough that rallyists will fill it out. Otherwise we get no feedback, rather than more feedback.
I think if the rallye is the wrong length, then we'll know whether it's too long or too short. And rallyists can write their answer on the back (or in the margins on the front). If we include both the phrase "too long" and the phrase "too short", then they can circle the appropriate phrase.
And I think the "start personnel" question covers the rallye school and the expert Q&A as appropriate. I think we need to avoid questions that a lot of rallyists won't be able to answer meaningfully.
- Darin McGrew 19:31, 22 December 2007 (PST)
Final Version
I updated the page with the text from the final version that was used at the January rallye. Some of the edits I made were specifically to get all the items onto a single line, so I could use as large a font as possible.
- Darin McGrew 19:06, 7 January 2008 (PST)