Measuring inter-observer agreement in contour delineation of medical imaging in a dummy run using fleiss' kappa

Methods of Information in Medicine

Volumn 51, Issue 6, 2012, Pages 489-494

  Rücker, Gerta ^a   Schimek Jasch, T ^a   Nestle, U ^a  

^a UNIVERSITY OF FREIBURG   (Germany)

Author keywords

Contour delineation; Fleiss' kappa; Medical imaging; Observer agreement; PET CT

Indexed keywords

ALGORITHM; ARTICLE; COMPUTER ASSISTED RADIOTHERAPY; HUMAN; METHODOLOGY; MULTICENTER STUDY (TOPIC); NEOPLASM; OBSERVER VARIATION; PATTERN RECOGNITION; PROSPECTIVE STUDY; RANDOMIZED CONTROLLED TRIAL (TOPIC); STATISTICS;

ALGORITHMS; FORM PERCEPTION; HUMANS; MULTICENTER STUDIES AS TOPIC; NEOPLASMS; OBSERVER VARIATION; PROSPECTIVE STUDIES; RADIOTHERAPY PLANNING, COMPUTER-ASSISTED; RANDOMIZED CONTROLLED TRIALS AS TOPIC;

EID: 84870840370     PISSN: 00261270     EISSN: None     Source Type: Journal    
DOI: 10.3414/ME12-01-0005     Document Type: Article

References (21)

1. Michalski JM, Lawton C, Naqa IE, Ritter M, O'Meara E, Seider MJ, et al. Development of RTOG consensus guidelines for the definition of the clinical target volume for postoperative conformal radiation therapy for prostate cancer. International Journal of Radiation Oncology, Biology, Physics 2010; 76 (2): 361-368.
2. Lim K, Small J William, Portelance L, Creutzberg C, Jürgenliemk-Schulz IM, Mundt A, et al. Consensus Guidelines for delineation of clinical target volume for intensity-modulated pelvic radiotherapy for the definitive treatment of cervix cancer. International Journal of Radiation Oncology, Biology, Physics 2011; 79 (2): 348-355.
3. Zijdenbos AP, Dawant BM, Margolin RA, Palmer AC. Morphometric analysis of white matter lesions in MR images: Method and Validation. IEEE Transactions of Medical Imaging 1994; 13 (4): 716-724.
4. Gibon D, Viard R, Rodrigues C. Contour comparison metrics. Loos Les Lille, France: AQUILAB SAS; 2009. Available from: http://www.aquilab.com.
5. Nestle U, Fleckenstein J, Kremp S, Schaefer-Schuler A, Hellwig D, Groeschel A, et al. PET-Plan NSCLC: first impressions from the pilot study. European Journal of Nuclear Medicine and Molecular Imaging 2007; 34: 141.
6. Schimek-Jasch T, Gibon D, Astner ST, Badakhshi H, Bultel YP, Hehr T, et al. Target volume delineation in locally advanced NSCLC: Multicenter contouring dummy run. Strahlentherapie und Onkologie 2010; 186: 144.
7. Schimek-Jasch T, Rücker G, Vach W, König J, Jacob V, Götz T, et al. Contour comparison metrics for two or more observers-Optimizing the evaluation of agreement in a contouring dummy run by identifying and resolving statistical pitfalls in the framework of the a multicenter study. Strahlentherapie und Onkologie 2011; 187: 132.
8. Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 1960; 20 (1):3 7-46.
9. Cohen J. Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin 1968; 70: 213-220.
10. Gefeller O, Brenner H. How to correct for chance agreement in the estimation of sensitivity and specificity of diagnostic tests. Methods Inf Med 1994; 33: 180-186.
11. Neveu D, Aubas P, Seguret F, Kramar A, Dujols P. Measuring agreement for ordered ratings in 3M3 tables. Methods Inf Med 2006; 45 (5): 541-547.
12. Fleiss JL. Measuring nominal scale agreement among many raters. Psychological Bulletin 1971; 76 (5): 378-382.
13. Fleiss JL, Cohen J, Everitt BS. Large sample standard errors of kappa and weighted kappa. Psychological Bulletin 1969; 72: 323-327.
14. Fleiss JL, Nee JCM, Landis JR. Large sample variance of kappa in the case of different sets of raters. Psychological Bulletin 1979; 86 (5): 974-977.
15. Landis JR, Koch GG. The Measurement of Ob-server Agreement for Categorical Data. Biometrics 1977; 33 (1): 159-174.
16. Wikipedia; 2011. Accessed October 18, 2011. Available from: http://en. wikipedia.org/wiki/Fleiss_kappa.
17. Krummenauer F. Methoden zur Evaluation bildgebender Verfahren von begrenzter Reproduzierbarkeit. Aachen: Shaker; 2005.
18. Albert PS, Dodd LE. A cautionary note on the robustness of latent class models for estimating diagnostic error without a gold standard. Biometrics 2004; 60 (2): 427-435.
19. Pepe MS, Janes H. Insights into latent class analysis of diagnostic test performance. Biostatistics 2007; 8 (2): 474-484.
20. Warfield SK, Zou KH, Wells WM. Simultaneous truth and performance level estimation (STAPLE): An algorithm for the validation of image segmentation. IEEE Transactions on Medical Imaging 2004; 23 (7): 903-921.
21. Dempster AP, Laird NM, Rubin DB. Maximum likelihood from incomplete data via EM algorithm. Journal of the Royal Statistical Society Series BMethodological 1977; 39 (1): 1-38.