Talk:TRC Gimmick Rallye Evaluation Cards
I don't want to "mess up" the draft, but am thinking we could separate (into separate sections)
* scoring, and awards presentation efficient? (moved close to end) * material clear and legible? * registration and CP workers competent?
- Dean
The rallyemaster doesn't have much control over registration, so we shouldn't count that as part of the "best rallye" awards (although it may still be useful feedback). It would be good to ask about the beginners school too, although we shouldn't expect upper division rallyists to answer that question. And we shouldn't count it when someone other than the rallyemaster gives the class.
I like the idea of putting the questions in rough chronological order (i.e., moving scoring and awards presentation towards the end).
And as we mentioned last night, very few people can offer meaningful feedback on whether protests were handled fairly. Perhaps a better indicator would be how often the protest committee overrules the rallyemaster.
- Darin McGrew 10:51, 14 November 2007 (PST)
I did many of the tasks that we had been talking about. See what you think.
- Dean
I recommend leaving the existing form alone, except for the changes to the "Length". 10 Categories is sufficient for the feedback that we need.
I am not sure if many rallyemasters use the form as it exists today as a vehicle to improve or change rallies. As a consequence, making changes to the form is likely not going to do much. If we add more categories, rallyists may be less likely to fill out the form.
A better use of time might be for rallyemasters to use the existing form to create a post-mortem of their event that includes 2 topics:
a) What worked well b) What can be improved upon.
Such a post-mortem should be optional, but would be nice to include in the TRC minutes (if it is done).
What to Cut?
Okay, we've got 13 sections now. What do we cut/consolidate to get back to 10 (or fewer)?
Darin McGrew 18:20, 21 November 2007 (PST)
I think 13 sections and 15 easy-to-answer questions might not be too much, if it fits nicely on a half page.
- Dean Since I was involved in the creation of this form in the '80s, I have some bias. I like the 10 category/100 point total format. In fact, the form was developed to fit this format. Also, I do not know how you will get overall scores if you go with a disagree/agree format.
I do agree that without written comments, the completed forms do not provide much value. Since these forms do not seem to be used by TRC for quality control or user feedback (just isn't working out), TRC might consider getting rid of the evaluations altogether. Are they really useful?
Using Incomplete Evaluations
I also propose that each question get scored by those who answer that question, so partial evaluations are used.
- Dean
If we include topics that not everyone will be able to answer, then we'll need to do something reasonable with "incomplete" evaluations. So, how should we handle the math for a rallye's total score while counting incomplete evaluations?
It's easy enough to average the responses we get in each category, and it's
okay if some categories have more data points than others. But do we calculate
the total score by adding the average scores for each category? I can't think
of a better way, but this still seems clunky to me.
- Darin McGrew 10:50, 28 November 2007 (PST)
If we end up adopting the agree/disagree format, then we could assume that blank statements are either "N/A" or "neither agree nor disagree".
I can see arguments for either, depending on the specific statement.
- Darin McGrew 12:11, 6 December 2007 (PST)
Making Evaluations Useful
Personally, I don't find the evaluation forms useful, except for the comments written on them (in the margins, or on the back). The numbers are purely arbitrary, and apparently even our worst rallyes are well above average (the Lake Wobegon effect), since they score above a 5 in all categories.
Maybe it would be more useful to have a number of statements (e.g., "Traffic was not a problem.") and ask whether they agree strongly, agree somewhat, neither agree nor disagree, disagree somewhat, or disagree strongly? Or maybe we could use a 7-point scale: agree strongly, agree, agree somewhat, etc.
- Darin McGrew 10:57, 28 November 2007 (PST)
I think this might be the best approach of all. It has the benefit of simplifying the questions, too. One of the complaints I've heard about the present eval card is that the questions cover slightly disjoint subjects, so the competitor has to average out things in their head.
- Steve 12:24, 2 December 2007 (PST)
Okay, I took a stab at creating a new evaluation card that uses the agree/disagree format. What do you think?
- Darin McGrew 13:26, 3 December 2007 (PST)
Email Feedback
Teresa suggested rephrasing the "clear, concise, and accurate" question so we get less negative feedback from rallyists who just missed an expert gimmick. I noted that gimmick rallyes often deliberately use unclear, verbose language, and that perhaps "clear" and "concise" are not criteria we should rate.
Teresa also emphasized that each rating should cover only one thing, since rallyists enter ratings like "7/8/10" on the current form when a single rating covers multiple topics.
I'd like a way to assess how well we're handling protests, but the only metrics I can think of are the total number of protests, the number of protests granted by the rallyemaster, and the number of protests granted by the committee. Including a question on the evaluation form seems pointless, because most people have no idea how well protests were handled.
A general question is whether this is a rallye evaluation, or a rallyemaster evaluation. Consensus appears to be that it is a rallye evaluation, and that we can use it for other purposes (e.g., rallyemaster awards) by ignoring the data that is irrelevant for those other purposes (e.g., questions about things like registration that the rallyemaster has no control over).
We need to figure out what to do with incomplete forms. Maybe we can just treat the incomplete questions as "N/A" or as "neither agree nor disagree".
We also need to decide whether to toss the top 10% and bottom 10%. I think this will be less necessary with the Agree/Disagree format.
For a "best rallyemaster" award, we could have a scale of 1-5 (strongly disagree to strongly agree), with 20 questions that are counted. That gives a maximum score of 100 points. Then we could round out the evaluation form to 25 or 30 questions, adding questions that won't count towards the award, but which we as a club care about.
- Darin McGrew 12:21, 12 December 2007 (PST)