Hierarchical modeling of agreement

Statistics in Medicine

Volumn 31, Issue 28, 2012, Pages 3667-3680

  Vanbelle, Sophie ^a   Mutsvari, Timothy ^b   Declerck, Dominique ^c   Lesaffre, Emmanuel ^a,d  

^a MAASTRICHT UNIVERSITY   (Netherlands)

^b KU LEUVEN   (Belgium)

^c UNIVERSITY OF LEUVEN   (Belgium)

^d ERASMUS UNIVERSITY ROTTERDAM   (Netherlands)

Author keywords

Cohen's kappa; Intraclass; Markov chain Monte Carlo; Multilevel; Nested; Rater; Reliability

Indexed keywords

ARTICLE; BAYES THEOREM; CALCULATION; CALIBRATION; CANINE TOOTH; CORRELATION COEFFICIENT; COVARIANCE; DECIDUOUS TOOTH; HUMAN; INCISOR; INTERRATER RELIABILITY; KAPPA STATISTICS; LINEAR SYSTEM; MATHEMATICAL COMPUTING; MATHEMATICAL MODEL; MAXIMUM LIKELIHOOD METHOD; PREMOLAR TOOTH; PROBABILITY;

BAYES THEOREM; CHILD; DENTAL CARE FOR CHILDREN; DENTAL CARIES; FEMALE; HUMANS; LIKELIHOOD FUNCTIONS; MALE; MARKOV CHAINS; MONTE CARLO METHOD; OBSERVER VARIATION; REPRODUCIBILITY OF RESULTS;

EID: 84870249498     PISSN: 02776715     EISSN: 10970258     Source Type: Journal    
DOI: 10.1002/sim.5424     Document Type: Article

References (62)

1. Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 1960; 20:37-46.
2. Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin 1968; 70:213-220.
3. Kraemer HC. Ramifications of a population model for κ as a coefficient of reliability. Psychometrika 1979; 44:461-472.
4. Conger AJ. Integration and generalization of kappas for multiple raters. Psychological Bulletin 1980; 88:322-328.
5. Fleiss JL. Statistical Methods for Rates and Proportions, 2nded. John Wiley: New York, 1981.
6. Davies M, Fleiss JL. Measuring agreement for multinomial data. Biometrics 1982; 38:1047-1051.
7. Schuster CS, Smith DA. Dispersion-weighted kappa: an integrative framework for metric and nominal scale agreement coefficients. Psychometrika 2005; 70:135-146.
8. Schouten HJA. Measuring pairwise interobserver agreement when all subjects are judged by the same observers. Statistica Neerlandica 1982; 36:45-61.
9. Vanbelle S, Albert A. Agreement between an isolated rater and a group of raters. Statistica Neerlandica 2009; 63:82-100.
10. Vanbelle S, Albert A. Agreement between two independent groups of raters. Psychometrika 2009; 74:477-491.
11. Graham P. Modelling covariate effects in observer agreement studies: the case of nominal scale agreement. Statistics in Medicine 1995; 14:299-310.
12. Perkins SM, Becker MP. Assessing rater agreement using marginal association models. Statistics in Medicine 2002; 21:1743-1760.
13. Uebersax JS. Validity inferences from interobserver agreement. Psychological Bulletin 1988; 104:405-416.
14. Agresti A. Modelling patterns of agreement and disagreement. Statistical Methods in Medical Research 1992; 1:201-218.
15. Schuster C, Smith DA. Indexing systematic rater agreement with a latent-class model. Psychological Methods 2002; 7:384-395.
16. Schuster C, Smith DA. Estimating with a latent class model the reliability of nominal judgments upon which two raters agree. Educational and Psychological Measurement 2006; 66:739-747.
17. Coughlin SS, Pickle LW, Goodman MT, Wilkens LR. The logistic modeling of interobserver agreement. Journal of Clinical Epidemiology 1992; 45:1237-1241.
18. Shoukri MM, Martin SW, Mian IUH. Maximum likelihood estimation of the kappa coefficient from models of matched binary responses. Statistics in Medicine 1995; 14:83-99.
19. Shoukri MM, Mian IUH. Maximum likelihood estimation of the kappa coefficient from bivariate logistic regression. Statistics in Medicine 1996; 15:1409-1419.
20. Barlow W. Measurement of interrater agreement with adjustment for covariates. Biometrics 1996; 52:695-702.
21. Lipsitz SR, Parzen M, Fitzmaurice GM, Klar N. A two-stage logistic regression model for analyzing inter-rater agreement. Psychometrika 2003; 68:289-298.
22. Lipsitz SR, Stuart R, Williamson J, Klar N, Ibrahim J, Parzen M. A simple method for estimating a regression model for κ between a pair of raters. Journal of the Royal Statistical Society, Series A 2001; 164:449-465.
23. Thomson JR. Estimating equations for kappa statistics. Statistics in Medicine 2001; 20:2895-2906.
24. Williamson JM, Manatunga AK. Assessing interrater agreement from dependent data. Biometrics 1997; 53:707-714.
25. Klar N, Lipsitz SR, Ibrahim JG. An estimating equations approach for modelling kappa. Biometrical Journal 2000; 42:45-58.
26. Williamson JM, Lipsitz SR, Manatunga AK. Modeling kappa for measuring dependent categorical agreement data. Biostatistics 2000; 1:191-202.
27. Gonin R, Lipsitz SR, Fitzmaurice GM, Molenberghs G. Regression modelling of weighted κ by using generalized estimating equations. Journal of the Royal Statistical Society, Series C 2000; 49:1-18.
28. Barnhart HX, Williamson JM. Weighted least-squares approach for comparing correlated kappa. Biometrics 2002; 58:1012-1019.
29. Thompson WD, Walter SD. A reappraisal of the kappa coefficient. Journal of Clinical Epidemiology 1988; 41:949-958.
30. Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problem of two paradoxes. Journal of Clinical Epidemiology 1990; 43:543-549.
31. Cicchetti DV, Feinstein AR. High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology 1990; 43:551-558.
32. Byrt T, Bishop J, Carlin JB. Bias, prevalence and kappa. Journal of Clinical Epidemiology 1993; 46:423-429.
33. Bloch DA, Kraemer HC. 2×2 kappa coefficients: measures of agreement or association. Biometrics 1989; 45:269-287.
34. Vach W. The dependence of Cohen's kappa on the prevalence does not matter. Journal of Clinical Epidemiology 2005; 58:655-661.
35. Hoehler FK. Bias and prevalence effects on kappa viewed in terms of sensitivity and specificity. Journal of Clinical Epidemiology 2000; 53:499-503.
36. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33:159-174.
37. Bishop YYM, Fienberg SE, Holland PW. Discrete Multivariate Analysis. Theory and Practice. MIT Press: Cambridge, 1975.
38. Fleiss JL, Davies M. Jackknifing functions of multinomial frequencies, with an application to a measure of concordance. American Journal of Epidemiology 1982; 115:841-845.
39. Garner JB. The standard error of Cohen's kappa. Statistics in Medicine 1991; 10:767-775.
40. Blackman NJ, Koval JJ. Interval estimation for Cohen's kappa as a measure of agreement. Statistics in Medicine 2000; 19:723-741.
41. Lesaffre E, Mwalili SM, Declerck D. Analysis of caries experience taking inter-observer bias and variability into account. Journal of Dental Research 2004; 83:951-955.
42. Mielke PW, Berry KJ, Johnston JE. Resampling probability values for weighted kappa with multiple raters. Psychological Reports 2008; 102:606-613.
43. Oden NL. Estimating kappa from binocular data. Statistics in Medicine 1991; 10:1303-1311.
44. Lesaffre E, Küchenhoff H, Mwalili SM, Declerck D. On the estimation of the misclassification table for finite count data with an application in caries research. Statistical Modelling 2009; 9:99-118.
45. Agbaje JO, Mutsvari T, Lesaffre E, Declerck D. Measurement, analysis and interpretation of examiner reliability in caries experience surveys: some methodological thoughts. Clinical Oral Investigations 2012; 16:117-127.
46. Mutsvari T, Lesaffre E, García-Zattera MJ, Diya L, Declerck D. Factors that influence data quality in caries experience detection: a multilevel modeling approach. Caries research 2010; 44:438-444.
47. Landis JR, Koch GG. An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics 1977; 33:363-374.
48. Schouten HJA. Estimating kappa from binocular data and comparing marginal probabilities. Statistics in Medicine 1993; 12:2207-2217.
49. Ren S, Yang S, Lai S. Intraclass correlation coefficients and bootstrap methods of hierarchical binary outcomes. Statistics in Medicine 2006; 25:3576-3588.
50. Gajewski BJ, Hart S, Bergquist-Beringer S, Dunton N. Inter-rater reliability of pressure ulcer staging: ordinal probit Bayesian hierarchical model that allows for uncertain rater response. Statistics in Medicine 2007; 26:4602-4618.
51. Zhang X, Cutter G. A Bayesian method of estimating kappa coefficient with application to a rheumatoid arthritis study. Communications in Statistics - Theory and Methods 2009; 38:3432-3444.
52. Hsiao CK, Chen PC, Kao WH. Bayesian random effects for interrater and test-retest reliability with nested clinical observations. Journal of Clinical Epidemiology 2011; 64:808-814.
53. WHO. Oral Health Surveys: Basic Methods, 4thed. World Health Organization: Geneva, 1997.
54. Pine CM, Pitts NB, Nugent ZJ. British Association for the Study of Community Dentistry (BASCD) guidance on the statistical aspects of training and calibration of examiners for surveys of child dental health. A BASCD coordinated dental epidemiology programme quality standard. Community Dental Health 1997; 14 (Suppl 1):18-29.
55. ® project, a longitudinal intervention health promotion study in Flanders (Belgium): baseline and first year results. European Journal of Paediatric Dentistry 2000; 2:87-96.
56. Scott WA. Reliability of content analysis: the case of nominal scale coding. Public Opinion Quarterly 1955; 19:321-325.
57. Spiegelhalter D, Thomas A, Best N, Gilks W. BUGS 0.5: Bayesian Inference Using Gibbs Sampling - Manual (versionii). Medical Research Council Biostatistics Unit: Cambridge, 1996.
58. Lunn DJ, Thomas A, Best N, Spiegelhalter D. WinBUGS - a Bayesian modelling framework: concepts, structure, and extensibility. Statistics and Computing 2000; 10:325-337.
59. Plummer M, Best N, Cowles K, Vines K. coda: Output Analysis and Diagnostics for MCMC. R package version 0.13-3, 2007. (Available from: http://CRAN.Rproject.org/).
60. Neuhaus JM, Kalbfleisch JD. Between- and within-cluster covariate effects in the analysis of clustered data. Biometrics 1998; 54:638-645.
61. Mancl LA, Leroux BG, DeRouen TA. Between-subject and within-subject statistical information in dental research. Journal of Dental Research 2000; 79:1778-1781.
62. Goldstein H, Browne W, Rasbash J. Partitioning variation in multilevel models. Understanding Statistics 2002; 1:223-231.