Reformulation and Generalisation of the Cohen and Fleiss Kappas

Xie, Zheng orcid iconORCID: 0000-0001-8649-6235, Gadepalli, Chaitanya and Cheetham, Barry M.G. (2017) Reformulation and Generalisation of the Cohen and Fleiss Kappas. LIFE: International Journal of Health and Life-Sciences, 3 (3). pp. 1-15. ISSN 2454-5872

[thumbnail of Version of Record]
PDF (Version of Record) - Published Version
Available under License Creative Commons Attribution Non-commercial.


Official URL:


The assessment of consistency in the categorical or ordinal decisions made by observers or raters is an important problem especially in the medical field. The Fleiss Kappa, Cohen Kappa and Intra-class Correlation (ICC), as commonly used for this purpose, are compared and a generalised approach to these measurements is presented. Differences between the Fleiss Kappa and multi-rater versions of the Cohen Kappa are explained and it is shown how both may be applied to ordinal scoring with linear, quadratic or other weighting. The relationship between quadratically weighted Fleiss and Cohen Kappa and pair-wise ICC is clarified and generalised to multi-rater assessments. The AC1 coefficient is considered as an alternative measure of consistency and the relevance of the Kappas and AC1 to measuring content validity is explored

Repository Staff Only: item control page