Agreement between an isolated rater and a group of raters

Statistica Neerlandica

Volumn 63, Issue 1, 2009, Pages 82-100

  Vanbelle, S ^a   Albert, A ^a  

^a UNIVERSITY OF LIÈGE   (Belgium)

Author keywords

Expert group; Kappa coefficient; Nominal scale; Ordinal scale

Indexed keywords

EID: 58449131599     PISSN: 00390402     EISSN: 14679574     Source Type: Journal    
DOI: 10.1111/j.1467-9574.2008.00412.x     Document Type: Article

References (21)

1. Charlin, B., R. Gagnon, L. Sibert C. Van der Veluten (2002 Le test de concordance de script : un instrument d'évaluation du raisonnement clinique, Pédagogie Médicale 3, 135 144.
2. Cicchetti, D. V. T. Allison (1971 A new procedure for assessing reliability of scoring EEG sleep recordings, American Journal of EEG Technology 11, 101 109.
3. Cohen, J. (1960 A coefficient of agreement for nominal scales, Educational and Psychological Measurement 20, 37 46.
4. Cohen, J. (1968 Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit, Psychological Bulletin 70, 213 220.
5. Eckstein, M. P., T. D. Wickens, G. Aharonov, G. Ruan, C. A. Morioka J. S. Whiting (1998 Quantifying the limitations of the use of consensus expert commitees in ROC studies, Proceedings SPIE: Medical Imaging 1998: Image Perception 3340, 128 134.
6. Efron, B. R. J. Tibshirani (1993 An introduction to the bootstrap, Chapman and Hall, New York.
7. Fleiss, J. L. (1981 Statistical methods for rates and proportions, 2nd edn, John Wiley, New York.
8. Fleiss, J. L. J. Cohen (1973 The equivalence of weighted kappa and the intraclass correlation coefficient as measure of reliability, Educational and Psychological Measurement 33, 613 619.
9. Kalant, N., M. Berlinguet, J. G. Diodati, L. Dragatakis F. Marcotte (2000 How valid are utilization review tools in assessing appropriate use of acute care beds? Canadian Medical Association Journal 162, 1809 1813.
10. Kraemer, H. C. (1979 Ramifications of a population model for κ as a coefficient of reliability, Psychometrika 44, 461 472.
11. Kraemer, H. C., V. S. Periyakoil A. Noda (2002 Kappa coefficient in medical research Tutorials in Biostatistics 1, 85 105.
12. Landis, J. R. G. G. Koch (1977 An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers, Biometrics 33, 363 374.
13. Light, R. J. (1971 Measures of response agreement for qualitative data: some generalizations and alternatives, Psychological Bulletin 76, 365 377.
14. Miller, D. P., K. F. O'Shaughnessy, S. A. Wood R. A. Castellino (2004 Gold standards and expert panels: a pulmonary nodule case study with challenges and solutions, Proceedings SPIE: Medical Imaging 1998: Image Perception, Observer Performance and Technology Assessment 5372, 173 184.
15. Ruperto, N., A. Ravelli, S. Oliveira, M. Alessio, D. Mihaylova, S. Pasic, E. Cortis, M. Apaz, R. Burgos-Vargas, F. Kanakoudi-Tsakalidou, X. Norambuena, F. Corona, V. Gerloni, S. Hagelberg, A. Aggarwal, P. Dolezalova, C. M. Saad, S. C. Bae, R. Vesely, T. Avcin, H. Foster, C. Duarte, T. Herlin, G. Horneff, L. Lepore, M. van Rossum, L. Trail, A. Pistorio, B. Andersson-Gare, E. H. Giannini A. Martini (2006 Pediatric Rheumatology International Trials Organization. The Pediatric Rheumatology International Trials Organization/American College of Response to Therapy in Juvenile Systemic Lupus Erythematosus: prospective validation of the definition of improvement, Arthritis and Rheumatism (Arthritis Care and Research) 55, 355 363.
16. Salerno, S. M., P. C. Alguire S. W. Waxman (2003 Competency in interpretation of 12-lead electrocardiograms: a summary and appraisal of published evidence, Annals of Internal Medicine 138, 751 760.
17. Schouten, H. J. A. (1982 Measuring pairwise interobserver agreement when all subjects are judged by the same observers, Statistica Neerlandica 36, 45 61.
18. Smith, R., A. J. Copas, M. Prince, B. George, A. S. Walker S. T. Sadiq (2003 Poor sensitivity and consistency of microscopy in the diagnosis of low grade non-gonococcal urethritis, Sexually Transmitted Infections 79, 487 490.
19. Soeken, K. L. P. A. Prescott (1986 Issues in the use of kappa to estimate reliability, Medical care 24, 733 741.
20. Vanbelle, S., V. Massart, D. Giet A. Albert (2007 Test de concordance de script: un nouveau mode d'établissement des scores limitant l'effet du hasard, Pédagogie Médicale 8, 71 81.
21. Williams, G. W. (1976 Comparing the joint agreement of several raters with another rater, Biometrics 32, 619 627.