Measurement of Rater Consistency by Chance-Corrected Agreement Coefficients

Xie, Zheng orcid iconORCID: 0000-0001-8649-6235, Gadepalli, Chaitanya and Cheetham, Barry M.G. (2018) Measurement of Rater Consistency by Chance-Corrected Agreement Coefficients. In: UKSim-AMSS 20th International Conference on Modelling & Simulation, 27 - 29 March 2018, Cambridge, UK.

Full text not available from this repository.

Official URL: http://uksim.info/uksim2018/CD1/pdfs/UKSim2018-40v...

Abstract

Measurement of consistency in the decisions made by observers or raters is an important problem in clinical medicine. Chance corrected agreement coefficients such as the Cohen and Fleiss Kappas are commonly used for this purpose, though the way that they estimate the probability of agreement 'by chance' has been strongly questioned. Alternatives have been proposed, such as the Aickin Alpha coefficient and the Gwet AC1 and AC2 coefficients which are gaining currency. A well known paradox illustrates deficiencies of the Kappa coefficients which, it is claimed, are remedied by an approach which grades the subjects according to their probability of being hard to score. The AC1 and AC2 coefficients result from the application of this grading to the Brennan-Prediger coefficient which may be considered a simplified form of Kappa. This paper questions the rationale of the hardness probability used by AC1 and proposes an alternative approach that may be applied to weighted and unweighted multi-rater Cohen and Fleiss Kappas and also Intra-Class Correlation (ICC) coefficients.


Repository Staff Only: item control page