SEPARATION OF SYSTEMATIC AND RANDOM ERRORS IN ORDINAL RATING SCALES

Abstract

The aim of this paper is to introduce a new rank method which enables us to separate the inconsistency of repeated measurements into random and systematic differences and to quantify this lack of consistency in a few measures. The key of the separation approach is to make a particular type of ranking of the repeated judgements in the same experimental unit. It means that cases which have the same classification from one rater will be internally ranked according to the classifications from the other. This enables us to extract the random variation. The variance of the rank differences between the judgements is a suitable measure of the random interrater variability. The systematic differences are described by empirical measures of relative position and of relative concentration. These measures are normed into the interval [-1,1]. Our method has been applied to several medical rating scales both for construction and analysis. We use one of the data sets as an illustration.