Resumen de Effect of rubric rating scale on the evaluation of engineering design projects
Mary Kathryn Thompson, Line Harder Clemmensen, Beung-uk Ahn
This paper explores the impact of the rubric rating scale on the evaluation of projects from a first year engineering design course.A small experiment was conducted in which twenty-one experienced graders scored five technical posters using one of four ratingscales. All rating scales tested produced excellent results in terms of inter-rater reliability and validity. However, there weresignificant differences in the performance of each of the scales. Based on the experiment’s results and past experience, we concludethat increasing the opportunities for raters to deduct points results in greater point deductions and lower overall scores. Increasingthe granularity of the scale can reduce this effect. Rating scales that use letter grades are less reliable than other types of scale.Assigning weights to individual criteria can lead to problems with validity if the weights are improperly balanced. Thus, heavilyweighted rubrics should be avoided if viable alternatives exist. Placing more responsibility for the final score on the grader insteadof the rubric seems to increase the validity at the cost of rater satisfaction. Finally, rater discomfort can lead to intentional misuseof a rating scale. This, in turn, increases the need to perform outlier detection on the final scores. Based on these findings, we recommend rating scale rubrics that use simple 3 or 4-point ordinal rating scales (augmented checks) for individual criteria and thatassign numerical scores to groups of criteria.
