Inter-rater reliability with heterogeneous observational data: a bayesian semi-parametric approach

In various observational contexts in which many evaluators judge a set of elements, it is common to assume that, although there is variability in the judging process among them, they are all part of the same population. In statistical models this assumption is expressed by specifying a normal distribution for random effects. However, due to the possible heterogeneity between different evaluators, it is possible to assume the existence of several subgroups of evaluators. The present work aims to overcome the strong assumption of a single rater population in estimating inter-rater agreement by specifying a Dirichlet process as the distribution of random effects. The following are proposed: a new semi- parametric index to quantify rater bias in the presence of inter-intra group heterogeneity and an approximate ICC index. An application case of the method to the emergency context of assignment of colour codes is discussed. The paramedics are asked to classify the request for help received at 118 on a scale of severity. It is therefore particularly important that the assignment of the codes takes place in a consistent manner among the paramedics and is as free as possible from their individual bias. It is shown that the proposed method allows to refine the estimate with respect to alternative methods.