Measurement, analysis and interpretation of examiner reliability in caries experience surveys: Some methodological thoughts

Clinical Oral Investigations

Volumn 16, Issue 1, 2012, Pages 117-127

  Agbaje, Jimoh Olubanwo ^a   Mutsvari, Timothy ^b   Lesaffre, Emannuel ^b,c   Declerck, Dominique ^a  

^a DEPARTMENT OF IMAGING AND PATHOLOGY   (Belgium)

^b KU LEUVEN   (Belgium)

^c ERASMUS MEDICAL CENTER   (Netherlands)

Author keywords

Agreement measurement; Caries experience; Dice coefficient; Kappa; Percentage agreement; Sensitivity; Specificity

Indexed keywords

ALGORITHM; ARTICLE; CALIBRATION; CONFIDENCE INTERVAL; DENTAL CARIES; HEALTH SURVEY; HUMAN; QUALITY CONTROL; REPRODUCIBILITY; SENSITIVITY AND SPECIFICITY; VALIDATION STUDY;

ALGORITHMS; BENCHMARKING; CALIBRATION; CONFIDENCE INTERVALS; DENTAL CARIES; DMF INDEX; HUMANS; REPRODUCIBILITY OF RESULTS; SENSITIVITY AND SPECIFICITY; VALIDATION STUDIES AS TOPIC;

EID: 84855961758     PISSN: 14326981     EISSN: 14363771     Source Type: Journal    
DOI: 10.1007/s00784-010-0475-x     Document Type: Article

References (44)

1. Pitts NB, Evans DJ, Pine CM (1997) British Association for the Study of Community Dentistry (BASCD) diagnostic criteria for caries prevalence surveys-1996/97. Community Dent Health 14(Suppl 1): 6-9.
2. Ismail AI, Sohn W, Tellez M, Amaya A, Sen A, Hasson H, Pitts NB (2007) The International Caries Detection and Assessment System (ICDAS): an integrated system for measuring dental caries. Community Dent Oral Epidemiol 35: 170-178.
3. World Health Organization (1997) Oral health surveys. Basic methods. World Health Organization, Geneva.
4. Pine CM, Pitts NB, Nugent ZJ (1997) British Association for the Study of Community Dentistry (BASCD) guidance on the statistical aspects of training and calibration of examiners for surveys of child dental health. A BASCD coordinated dental epidemiology programme quality standard 2. Community Dent Health 14(Suppl 1): 18-29.
5. International Caries Detection and Assessment System Coordinating Committee (2005) Criteria manual - International Caries Detection and Assessment System (ICDAS II).
6. Maxwell AE (1970) Comparing the classification of subjects by two independent judges. Br J Psychiatry 116: 651-655.
7. McNemar Q (1947) Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12: 153-157.
8. Stuart AA (1955) A test for homogeneity of the marginal distributions in a two-way classification. Biometrika 42: 412-416.
9. Cicchetti DV, Allison T (1971) A new procedure for assessing reliability of scoring EEG sleep recordings. Am J EEG Technol 11: 101-109.
10. Cohen J (1968) Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychol Bull 70: 213-220.
11. Kingman A (1986) A procedure for evaluating the reliability of a gingivitis index. J Clin Periodontol 13: 385-391.
12. Uebersax JS (1993) Statistical modeling of expert ratings on medical treatment appropriateness. J Am Stat Assoc 88: 421-427.
13. Uebersax JS (1987) Diversity of decision-making models and the measurement of interrater agreement. Psychol Bull 101: 140-146.
14. Stamm JW, Stewart PW, Bohannan HM, Disney JA, Graves RC, Abernathy JR (1991) Risk assessment for oral diseases. Adv Dent Res 5: 4-17.
15. Dice LR (1945) Measures of the amount of ecologic association between species. Ecology 26: 297-302.
16. Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20: 37-46.
17. Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33: 159-174.
18. Tooth LR, Ottenbacher KJ (2004) The kappa statistic in rehabilitation research: an examination. Arch Phys Med Rehabil 85: 1371-1376.
19. Gwet KL (2008) Computing inter-rater reliability and its variance in the presence of high agreement. Br J Math Stat Psychol 61: 29-48.
20. Uebersax JS (2005) Statistical methods for rater agreement: the tetrachoric and polychoric correlation coefficient. http://www. john-uebersax. com/stat/tetra. htm. Accessed 10 May 2010.
21. Hutchinson TP (1993) Focus on psychometrics. Kappa muddles together two sources of disagreement: tetrachoric correlation is preferable. Res Nurs Health 16: 313-316.
22. Byrt T, Bishop J, Carlin JB (1993) Bias, prevalence and kappa. J Clin Epidemiol 46: 423-429.
23. Declerck D, Leroy R, Martens L, Lesaffre E, Garcia-Zattera MJ, Vanden BS, Debyser M, Hoppenbrouwers K (2008) Factors associated with prevalence and severity of caries experience in preschool children. Community Dent Oral Epidemiol 36: 168-178.
24. Assaf AV, de Castro MM, Zanin L, Tengan C, Pereira AC (2006) Effect of different diagnostic thresholds on dental caries calibration-a 12-month evaluation. Community Dent Oral Epidemiol 34: 213-219.
25. Assaf AV, Tagliaferro EP, Meneghim MC, Tengan C, Pereira AC, Ambrosano GM, Mialhe FL (2007) A new approach for interexaminer reliability data analysis on dental caries calibration. J Appl Oral Sci 15: 480-485.
26. Vanobbergen J, Lesaffre E, Garcia-Zattera MJ, Jara A, Martens L, Declerck D (2007) Caries patterns in primary dentition in 3-, 5- and 7-year-old children: spatial correlation and preventive consequences. Caries Res 41: 16-25.
27. Williamson JM, Datta S, Satten GA (2003) Marginal analyses of clustered data when cluster size is informative. Biometrics 59: 36-42.
28. Hoehler FK (2000) Bias and prevalence effects on kappa viewed in terms of sensitivity and specificity. J Clin Epidemiol 53: 499-503.
29. Spitznagel EL, Helzer JE (1985) A proposed solution to the base rate problem in the kappa statistic. Arch Gen Psychiatry 42: 725-728.
30. Maclure M, Willett WC (1987) Misinterpretation and misuse of the kappa statistic. Am J Epidemiol 126: 161-169.
31. Braga MM, Oliveira LB, Bonini GA, Bonecker M, Mendes FM (2009) Feasibility of the International Caries Detection and Assessment System (ICDAS-II) in epidemiological surveys and comparability with standard World Health Organization criteria. Caries Res 43: 245-249.
32. Fyffe HE, Deery C, Nugent ZJ, Nuttall NM, Pitts NB (2000) Effect of diagnostic threshold on the validity and reliability of epidemiological caries diagnosis using the Dundee Selectable Threshold Method for caries diagnosis (DSTM). Community Dent Oral Epidemiol 28: 42-51.
33. Gilthorpe MS, Griffiths GS, Maddick IH, Zamzuri AT (2000) The application of multilevel modelling to periodontal research data. Community Dent Health 17: 227-235.
34. Tu YK, Gilthorpe MS, Griffiths GS, Maddick IH, Eaton KA, Johnson NW (2004) The application of multilevel modeling in the analysis of longitudinal periodontal data-part I: absolute levels of disease. J Periodontol 75: 127-136.
35. Williams FM, Nan G (2006) Estimation of sensitivity and specificity of clustered binary data. Statistics and data analysis, SUGI 31 Proceedings, SAS Proceedings.
36. Burnside G, Pine CM, Williamson PR (2007) The application of multilevel modelling to dental caries data. Stat Med 26: 4139-4149.
37. Mutsvari T, Lesaffre E, Garcia-Zattera MJ, Diya L, Declerck D (2010) Factors that influence data quality in caries experience detection: a multilevel modeling approach. Caries Res 44: 438-444.
38. Williamson JM, Lipsitz SR, Manatunga AK (2000) Modeling kappa for measuring dependent categorical agreement data. Biostatistics 1: 191-202.
39. Liang KY, Zeger SL (1986) Longitudinal data analysis using generalized linear models. Biometrika 73: 13-22.
40. Lesaffre E, Mwalili SM, Declerck D (2004) Analysis of caries experience taking inter-observer bias and variability into account. J Dent Res 83: 951-955.
41. Barnhart HX, Williamson JM (2002) Weighted least-squares approach for comparing correlated kappa. Biometrics 58: 1012-1019.
42. Lin HM, Williamson JM, Lipsitz SR (2003) Calculating power for the comparison of dependent κ-coefficients. J R Stat Soc C Appl Stat 52: 391-404.
43. Wacholder S, Armstrong B, Hartge P (1993) Validation studies using an alloyed gold standard. Am J Epidemiol 137: 1251-1258.
44. Brenner H (1996) Correcting for exposure misclassification using an alloyed gold standard. Epidemiology 7: 406-410.