Interrater Agreement Evaluation: A Latent Variable Modeling Approach

Educational and Psychological Measurement

Volumn 73, Issue 3, 2013, Pages 512-531

  Raykov, Tenko ^a   Dimitrov, Dimiter M ^b   von Eye, Alexander ^a   Marcoulides, George A ^c  

^a MICHIGAN STATE UNIVERSITY   (United States)

^b GEORGE MASON UNIVERSITY   (United States)

^c UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

interrater agreement; latent variable modeling; raters; reliability; targets

Indexed keywords

EID: 84876977079     PISSN: 00131644     EISSN: 15523888     Source Type: Journal    
DOI: 10.1177/0013164412449016     Document Type: Article

References (35)

1. Burke M. J.,Dunlap W. P.Estimating interrater agreement with the average deviation index: A user's guide.Organizational Research Methods. 2002;5:159-172.
2. Cohen J.A coefficient of agreement for nominal scales.Educational and Psychological Measurement. 1960;20:37-46.
3. Cohen J.Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit.Psychological Bulletin. 1968;70:213-220.
4. Congdon P. J.,McQueen J.The stability of rater severity in large-scale assessment programs.Journal of Educational Measurement. 2000;37:163-178.
5. Dimitrov D. M.Statistical methods for validation of assessment scale data in counseling and related fields. Alexandria, VA: American Counseling Association; 2012:.
6. Enders C. A.Applied missing data analysis. New York: Guilford; 2010:.
7. Goldstein H.Multilevel modeling. New York, NY: Wiley; 2011:.
8. Jöreskog K. G.,Sörbom D.LISREL8 user's reference guide. Lincolnwood, IL: Scientific Software International; 1996:.
9. Laenen A.,Alonso A.,Molenberghs G.Reliability of a longitudinal sequence of scale ratings.Psychometrika. 2009;74:49-64.
10. LeBreton J. M.,Senter J. L.Answers to 20 questions on inter-rater agreement and inter-rater reliability.Organizational Research and Methods. 2008;11:815-852.
11. Lindell M. K.Assessing and testing interrater agreement in multi-item rating scales.Applied Psychological Measurement. 2001;25:89-99.
12. Little R. J.,Rubin D. B.Statistical analysis with missing data. New York, NY: Wiley; 2002:.
13. Muthén B.A general structural equation model with dichotomous, ordered categorical, and continuous latent variable indicators.Psychometrika. 1984;49:115-132.
14. Muthén L. K.,Muthén B.Mplus user's guide. Los Angeles, CA: Muthén & Muthén; 2010:.
15. Raykov T.Intra-class correlation coefficients in hierarchical designs: Evaluation using latent variable modeling.Structural Equation Modeling. 2011;18:73-90.
16. Raykov T.Handbook of structural equation modeling. Hoyle R., ed. New York, NY: Guilford; 2012:472-492.
17. Raykov T.,Marcoulides G. A.Using the delta method for approximate interval estimation of parametric functions in covariance structure models.Structural Equation Modeling. 2004;11:659-675.
18. Raykov T.,Marcoulides G. A.Introduction to psychometric theory. New York, NY: Taylor & Francis; 2011:.
19. Raykov T.,Marcoulides G. A.On examining the underlying normal variable assumption in latent variable models. 2012;:.
20. Raykov T.,Shrout P.Reliability of Scales with General Structure: Point and Interval Estimation Using a Structural Equation Modeling Approach.Structural Equation Modeling. 2002;9:195-212.
21. Raymond M. R.,Harik P.,Clauser B. E.The impact of statistically adjusting for rater effects on conditional standard errors of performance ratings.Applied Psychological Measurement. 2011;35:235-246.
22. Satorra A.Innovations in multivariate statistical analysis. A festschrift for Heinz Neudecker. Heijmans R. D. H.Pollock D. S. G.Satorra A., ed. London, England: Kluwer Academic; 2000:233-247.
23. Schuster C.A mixture model to indexing rater agreement.British Journal of Mathematical and Statistical Psychology. 2002;55:289-303.
24. Schuster C.,Smith D. A.Indexing systematic rater agreement with a latent-class model.Psychological Methods. 2002;7:384-395.
25. Schuster C.,Smith D. A.Dispersion-weighted kappa: An integrative framework for metric and nominal scale agreement coefficients.Psychometrika. 2005;70:135-146.
26. Schuster C.,von Eye A.Models for ordinal agreement data.Biometrical Journal. 2001;43:795-808.
27. Shavelson R. J.,Webb N. M.Generalizability theory. Thousand Oaks, CA: Sage; 1991:.
28. Shoukri M. M.Measures of interobserver agreement and reliability. Boca Raton, FL: Chapman & Hall; 2011:.
29. Shrout P. E.,Fleiss J.Intraclass correlations: Uses in assessing reliability.Psychological Bulletin. 1979;86:420-428.
30. Shrout P. E.,Lane S.APA handbook of research methods in psychology. Cooper H., ed. Washington, DC: American Psychological Association.
31. Skrondal A.,Rabe-Hesketh S.Generalized latent linear and mixed models. Boca Raton, FL: Chapman & Hall; 2004:.
32. Tanner M. A.,Young M. A.Modeling agreement among raters.Journal of the American Statistical Association. 1985;80:175-180.
33. von Eye A. A.,Mun E. Y.Analyzing rater agreement. Mahwah, NJ: Erlbaum; 2005:.
34. Wolfe E. W. A report on the reliability of a large-scale portfolio assessment for language arts, mathematics, and science. Paper presented at the annual meeting of the National Council for Measurement in Education; 1996New York, NY; 1996.
35. Uebersax J. S.Statistical modeling of expert ratings on medical treatment appropriateness.Journal of the American Statistical Association. 1993;88:421-427.