Assessing agreement between multiple raters with missing rating information, applied to breast cancer tumour grading

PLoS ONE

Volumn 3, Issue 8, 2008, Pages

  Fanshawe, Thomas R ^a   Lynch, Andrew G ^b   Ellis, Ian O ^c   Green, Andrew R ^c   Hanka, Rudolf ^d  

^a LANCASTER UNIVERSITY   (United Kingdom)

^b UNIVERSITY OF CAMBRIDGE   (United Kingdom)

^c UNIVERSITY OF NOTTINGHAM   (United Kingdom)

^d UNIVERSITY OF CAMBRIDGE   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; BAYES THEOREM; BREAST CANCER; CANCER GRADING; CONSENSUS; CONTROLLED STUDY; DISEASE SEVERITY; HUMAN; PATHOLOGIST; PROBABILITY; SCORING SYSTEM; UNCERTAINTY; BREAST TUMOR; CANCER STAGING; CELL NUCLEUS; FEMALE; METHODOLOGY; MITOSIS INDEX; OBSERVER VARIATION; PATHOLOGY; REPRODUCIBILITY;

BAYES THEOREM; BREAST NEOPLASMS; CELL NUCLEUS; FEMALE; HUMANS; MITOTIC INDEX; NEOPLASM STAGING; OBSERVER VARIATION; PATHOLOGY; PROBABILITY; REPRODUCIBILITY OF RESULTS;

EID: 51449094360     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0002925     Document Type: Article

References (40)

1. Nelson J, Pepe M (2000) Statistical description of interrater variability in ordinal ratings. Statistical Methods in Medical Research 9: 475-496.
2. Zhou X, Obuchowski N, McClish D (2002) Statistical Methods in Diagnostic Medicine. New York: Wiley.
3. Pepe M (2003) The Statistical Evaluation of Medical Tests for Classification and Prediction. Oxford: Oxford University Press.
4. Uebersax J, Grove W (1993) A latent trait finite mixture model for the analysis of rating agreement. Biometrics 49: 823-835.
5. Cohen J (1960) A coefficient of agreement for nominal scales. Education and Psychological Measurement 20: 37-46.
6. Feinstein A, Cicchetti D (1990) High agreement but low Kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology 43: 543-549.
7. Agresti A (1990) Categorical Data Analysis. New Work: Wiley.
8. Langeheine R, Rost J (1988) Latent Trait and Latent Class Models. New York: Plenum Press.
9. Galea M, Blamey R, Elston C, Ellis I (1992) The Nottingham Prognostic Index in primary breast cancer. Breast Cancer Research and Treatment 22: 207-219.
10. Patey D, Scarff R (1928) The position of histology in the prognosis of carcinoma of the breast. Lancet 211: 801-804.
11. Elston C, Ellis I (1991) Pathological prognostic factors in breast cancer. I. The value of histological grade in breast cancer: experience from a large study with long-term follow-up. Histopathology 19: 403-410.
12. Hartman W (1984) Minimal breast cancer: an update. Cancer 53: 681-684.
13. Ellis I, Elston C (1995) Tumours of the breast. In: Fletcher C, ed (1995) Diagnostic Histopathology of Tumors. pp 635-689.
14. Ellis I, Galea M, Broughton N, Locker A, Blamey R, et al. (1992) Pathological prognostic factors in breast cancer. II. Histological type. Relationship with survival in a large study with long-term follow-up. Histopathology 20: 479-489.
15. National Co-ordinating Group for Breast Screening Pathology (1995) Pathology reporting in breast cancer screening: NHS BSP Publications.
16. Robbins P, Pinder S, de Klerk N, Dawkins H, Harvey J, et al. (1995) Histological grading of breast carcinomas: a study of interobserver agreement. Human Pathology 26: 873-879.
17. Meyer J, Alvarez C, Milikowski C, Olsen N, Russo I, et al. (2005) Breast carcinoma malignancy grading by Bloom-Richardson system vs proliferation index: reproducibility of grade and advantages of proliferation index. Modern Pathology 18: 1067-1078.
18. Ellis I, Coleman D, Wells C, Kodikara S, Paish E, et al. (2006) Impact of a national external quality assessment scheme for breast pathology in the U.K. Journal of Clinical Pathology 59: 138-145.
19. Siegel S, Castellan N (1988) Nonparametric Statistics for the Behavioural Sciences. London: McGraw Hill.
20. Landis J, Koch G (1977) The measurement of observer agreement for categorical data. Biometrics 33: 159-174.
21. Fleiss J (1971) Measuring nominal scale agreement among many raters. Psychological Bulletin 76: 378-382.
22. Cicchetti D, Feinstein A (1990) High agreement but low Kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology 43: 551-558.
23. Guggenmoos-Holzmann I (1993) How reliable are chance-corrected measures of agreement. Statistics in Medicine 12: 2191-2205.
24. Cohen J (1968) Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin 70: 213-220.
25. Landis J, Koch G (1977) An application of hierarchical Kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics 33: 363-374.
26. Skrondal A, Rabe-Hesketh S (2008) Latent variable modelling. Statistical Methods in Medical Research. pp 17.
27. Formann A, Kohlmann T (1996) Latent class analysis in medical research. Statistical Methods in Medical Research 5: 179-211.
28. Yang I, Becker M (1997) Latent variable modelling of diagnostic accuracy. Biometrics 53: 948-958.
29. Uebersax J, Grove W (1990) Latent class analysis of diagnostic agreement. Statistics in Medicine 9: 559-572.
30. Agresti A, Lang J (1993) Quasi-symmetric latent class models, with application to rater agreement. Biometrics 49: 131-139.
31. Uebersax J (1993) Statistical modeling of expert ratings on medical treatment appropriateness. Journal of the American Statistical Association 88: 421-427.
32. Rogel A, Boelle P, Mary J (1998) Global and partial agreement among several observers. Statistics in Medicine 17: 489-501.
33. Martin Andres A, Femia Marzo P (2004) Delta: a new measure of agreement between two raters. British Journal of Mathematical and Statistical Psychology 57: 1-19.
34. Landis J, Koch G (1977) A one-way components of variance model for categorical data. Biometrics 33: 671-679.
35. James I (1983) Analysis of nonagreements among multiple raters. Biometrics 39: 651-657.
36. Altaye M, Dormer A, Klar N (2001) Inference procedures for assessing interobserver agreement among multiple raters. Biometrics 57: 584-588.
37. Spiegelhalter D (2005) Funnel plots for comparing institutional performance. Statistics in Medicine 24: 1185-1202.
38. Spiegelhalter D, Thomas A, Best N, Lunn D (2003) WinBUGS User Manual.
39. Lunn D, Thomas A, Best N, Spiegelhalter D (2000) WinBUGS - a Bayesian modelling framework: concepts, structure and extensibility. Statistics and Computing 10: 321-333.
40. R Development Core Team (2005) R: A language and environment for statistical computing. Vienna, Austria: R Foundation For Statistical Computing.