Talk:TRC Gimmick Rallye Evaluation Cards

I don't want to "mess up" the draft, but am thinking we could separate (into separate sections)

* scoring, and awards presentation efficient? (moved close to end)
* material clear and legible?
* registration and CP workers competent?

- Dean

---

The rallyemaster doesn't have much control over registration, so we shouldn't count that as part of the "best rallye" awards (although it may still be useful feedback). It would be good to ask about the beginners school too, although we shouldn't expect upper division rallyists to answer that question. And we shouldn't count it when someone other than the rallyemaster gives the class.

I like the idea of putting the questions in rough chronological order (i.e., moving scoring and awards presentation towards the end).

And as we mentioned last night, very few people can offer meaningful feedback on whether protests were handled fairly. Perhaps a better indicator would be how often the protest committee overrules the rallyemaster.

- Darin McGrew 10:51, 14 November 2007 (PST)

---

I did many of the tasks that we had been talking about. See what you think.

- Dean

---

I recommend leaving the existing form alone, except for the changes to the "Length". 10 Categories is sufficient for the feedback that we need.

I am not sure if many rallyemasters use the form as it exists today as a vehicle to improve or change rallies. As a consequence, making changes to the form is likely not going to do much. If we add more categories, rallyists may be less likely to fill out the form.

A better use of time might be for rallyemasters to use the existing form to create a post-mortem of their event that includes 2 topics:

a) What worked well b) What can be improved upon.

Such a post-mortem should be optional, but would be nice to include in the TRC minutes (if it is done).

- Cris Wendt

What to Cut?

Okay, we've got 13 sections now. What do we cut/consolidate to get back to 10 (or fewer)?

Darin McGrew 18:20, 21 November 2007 (PST)

---

I think 13 sections and 15 easy-to-answer questions might not be too much, if it fits nicely on a half page.

- Dean Since I was involved in the creation of this form in the '80s, I have some bias. I like the 10 category/100 point total format. In fact, the form was developed to fit this format. Also, I do not know how you will get overall scores if you go with a disagree/agree format.

I do agree that without written comments, the completed forms do not provide much value. Since these forms do not seem to be used by TRC for quality control or user feedback (just isn't working out), TRC might consider getting rid of the evaluations altogether. Are they really useful?

Using Incomplete Evaluations

I also propose that each question get scored by those who answer that question, so partial evaluations are used.
- Dean

If we include topics that not everyone will be able to answer, then we'll need to do something reasonable with "incomplete" evaluations. So, how should we handle the math for a rallye's total score while counting incomplete evaluations?

It's easy enough to average the responses we get in each category, and it's okay if some categories have more data points than others. But do we calculate the total score by adding the average scores for each category? I can't think of a better way, but this still seems clunky to me.
- Darin McGrew 10:50, 28 November 2007 (PST)

If we end up adopting the agree/disagree format, then we could assume that blank statements are either "N/A" or "neither agree nor disagree". I can see arguments for either, depending on the specific statement.
- Darin McGrew 12:11, 6 December 2007 (PST)

Making Evaluations Useful

Personally, I don't find the evaluation forms useful, except for the comments written on them (in the margins, or on the back). The numbers are purely arbitrary, and apparently even our worst rallyes are well above average (the Lake Wobegon effect), since they score above a 5 in all categories.

Maybe it would be more useful to have a number of statements (e.g., "Traffic was not a problem.") and ask whether they agree strongly, agree somewhat, neither agree nor disagree, disagree somewhat, or disagree strongly? Or maybe we could use a 7-point scale: agree strongly, agree, agree somewhat, etc.
- Darin McGrew 10:57, 28 November 2007 (PST)

I think this might be the best approach of all. It has the benefit of simplifying the questions, too. One of the complaints I've heard about the present eval card is that the questions cover slightly disjoint subjects, so the competitor has to average out things in their head.
- Steve 12:24, 2 December 2007 (PST)

Okay, I took a stab at creating a new evaluation card that uses the agree/disagree format. What do you think?
- Darin McGrew 13:26, 3 December 2007 (PST)