Examining the reliability of interval level data using root mean square differences and concordance correlation coefficients

Psychological Methods

Volumn 17, Issue 2, 2012, Pages 294-308

  Barchard, Kimberly A ^a  

^a UNIVERSITY OF NEVADA   (United States)

Author keywords

Concordance correlation coefficient; Consistency; Intraclass correlation; Reliability; Root mean square difference

Indexed keywords

ARTICLE; CHILD; EDUCATION; FEMALE; HUMAN; MALE; METHODOLOGY; PREDICTIVE VALUE; PRESCHOOL CHILD; PSYCHOLOGY; PSYCHOMETRY; REPRODUCIBILITY; STATISTICAL ANALYSIS; STATISTICS;

CHILD; CHILD, PRESCHOOL; DATA INTERPRETATION, STATISTICAL; EDUCATIONAL MEASUREMENT; FEMALE; HUMANS; MALE; PREDICTIVE VALUE OF TESTS; PSYCHOLOGY; PSYCHOMETRICS; REPRODUCIBILITY OF RESULTS; STATISTICS AS TOPIC;

EID: 84866445535     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0023351     Document Type: Article

References (67)

1. Allen, M. J., & Yen, W. M. (1979). Introduction to measurement theory. Prospect Heights, IL: Waveland Press.
2. Alwin, D. F., & Jackson, D. J. (1980). Measurement models for response errors in surveys: Issues and applications. In K. F. Schuessler (Ed.), Sociological methodology (pp. 68-119). San Francisco, CA: JosseyBass.
3. Barchard, K. A. (2009, May). Evaluating score consistency through score change probabilities. Poster presented at the meeting of the Association for Psychological Science in San Francisco, CA.
4. Barnhart, H. X., Haber, M., & Song, J. L. (2002). Overall concordance correlation coefficient for evaluating agreement among multiple observers. Biometrics, 58, 1020-1027. doi:10.1111/j.0006-341X.2002.01020.x
5. Barnhart, H. X., Lokhnygina, Y., Kosinski, A. S., & Haber, M. (2007). Comparison of concordance correlation coefficient and coefficient of individual agreement in assessing agreement. Journal of Biopharmaceutical Statistics, 17, 721-738. doi:10.1080/10543400701329497
6. Barnhart, H. X., Song, J., & Haber, M. (2005). Assessing intra, inter and total agreement with replicated readings. Statistics in Medicine, 24, 1371-1384. doi:10.1002/sim.2006
7. Barnhart, H. X., & Williamson, J. M. (2001). Modeling concordance correlation via GEE to evaluate reproducibility. Biometrics, 57, 931-940. doi:10.1111/j.0006-341X.2001.00931.x
8. Bentler, P. M., & Weeks, D. G. (1980). Linear structural equations with latent variables. Psychometrika, 45, 289-308. doi:10.1007/BF02293905
9. Box, G. E. P., Jenkins, G. M., & Reinsel, G. C. (2008). Time series analysis: Forecasting and control (4th ed.). Hoboken, NJ: Wiley.
10. Brennan, R. L. (2001). Generalizability theory. New York, NY: Springer.
11. Byrne, B. M. (2006). Structural equation modeling with EQS: Basic concepts, applications, and programming (2nd ed.). Mahwah, NJ: Erlbaum.
12. Carrasco, J. L., & Jover, L. (2003). Estimating the generalized concordance correlation coefficient through variance components. Biometrics, 59, 849-858. doi:10.1111/j.0006-341X.2003.00099.x
13. Carrasco, J. L., Jover, L., King, T. S., & Chinchilli, V. M. (2007). Comparison of concordance correlation coefficient estimating approaches with skewed data. Journal of Biopharmaceutical Statistics, 17, 673-684. doi:10.1080/10543400701329463
14. Chen, C.-C., & Barnhart, H. X. (2008). Comparison of ICC and CCC for assessing agreement for data without and with replications. Computational Statistics and Data Analysis, 53, 554-564. doi:10.1016/j.csda.2008.09.026
15. Clitheroe, H. C. Jr., Stokols, D., & Zmuidzinas, M. (1998). Conceptualizing the context of environment and behavior. Journal of Environmental Psychology, 18, 103-112. doi:10.1006/jevp.1998.0091
16. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
17. Cox, T. F., & Cox, M. A. A. (2001). Multidimensional scaling (2nd ed.). London, England: Chapman & Hall.
18. Crawford, S. B., Kosinski, A. S., Lin, H.-M., Williamson, J. M., & Barnhart, H. X. (2007). Computer programs for the concordance correlation coefficient. Computer Methods and Programs in Biomedicine, 88, 62-74. doi:10.1016/j.cmpb.2007.07.003
19. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Orlando, FL: Harcourt Brace Jovanovich.
20. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334. doi:10.1007/BF02310555
21. Cronbach, L. J., & Gleser, G. (1953). Assessing the similarity between profiles. Psychological Bulletin, 50, 456-473. doi:10.1037/h0057173
22. Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory for generalizability of scores and profiles. New York, NY: Wiley.
23. Deyo, R. A., Diehr, P., & Patrick, D. L. (1991). Reproducibility and responsiveness of health status measures. Controlled Clinical Trials, 12, S142-S158. doi:10.1016/S0197-2456(05)80019-4
24. Educational Testing Service, Office of Research and Development (1998). Score change when retaking the SAT I: Reasoning Test (Research Note RN-05). Retrieved from http://professionals.collegeboard.com/profdownload/pdf/rn05_10756.pdf
25. Edwards, J. R. (1994). The study of congruence in organizational behavior research: Critique and a proposed alternative. Organizational Behavior and Human Decision Processes, 58, 51-100. doi:10.1006/obhd.1994.1029
26. Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Mahwah, NJ: Erlbaum.
27. Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4, 272-299. doi:10.1037/1082989X.4.3.272
28. Fagot, R. F. (1993). A generalized family of coefficients of relationship agreement for numerical scales. Psychometrika, 58, 357-370. doi: 10.1007/BF02294581
29. Fisher, R. A. (1925). Statistical methods for research workers. Edinburgh, Scotland: Oliver & Boyd.
30. Furr, R. M. (2009). Profile analysis in person-situation integration. Journal of Research in Personality, 43, 196-207. doi:10.1016/j.jrp.2008.08.002
31. Greenhouse, S. W., & Geisser, S. (1959). On methods in the analysis of profile data. Psychometrika, 24, 95-112. doi:10.1007/BF02289823
32. Griffo, R., & Colvin, C. R. (2009). A brief look at interactionism: Past and present. Journal of Research in Personality, 43, 243-244. doi:10.1016/ j.jrp.2008.12.038
33. Hájek, A. (2010). Interpretations of probability. In E. N. Zalta (Ed.), The Stanford encyclopedia of philosophy. Retrieved from http://plato.stanford.edu/archives/spr2010/entries/probability-interpret/
34. Harmon, H. H. (1976). Modern factor analysis (3rd ed., rev.). Chicago, IL: University of Chicago Press.
35. Johnson, J. A. (2009). Wrong and right questions about persons and situations. Journal of Research in Personality, 43, 251-252. doi: 10.1016/j.jrp.2008.12.022
36. Jöreskog, K. G. (1971). Statistical analysis of sets of congeneric tests. Psychometrika, 36, 109-133. doi:10.1007/BF02291393
37. Kano, Y., & Azuma, Y. (2003). Use of SEM programs to precisely measure scale reliability. In H. Yanai, A. Akada, K. Shigemasu, Y. Kano, & J. J. Meulman (Eds.), New developments in psychometrics: Proceedings of the International Meeting of the Psychometric Society 2001 (pp. 141-148). Tokyo, Japan: Springer.
38. King, T. S., & Chinchilli, V. M. (2001). A generalized concordance correlation coefficient for continuous and categorical data. Statistics in Medicine, 20, 2131-2147. doi:10.1002/sim.845
39. Kingston, N., & Turner, N. (1984). Analysis of score change patterns of examinees repeating the Graduate Record Examinations General Test (Research Report No. 34-22). Princeton, NJ: Educational Testing Service.
40. Koopman, R. (2005, May 26). Re: Pearson for reliability studies? [Online forum comment]. Retrieved from http://groups.google.ca/group/sci.stat.consult/msg/f777e4699c74b017
41. Lin, L. I. (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics, 45, 255-268. doi:10.2307/2532051
42. Lin, L. I. (1992). Assay validation using the concordance correlation coefficient. Biometrics, 48, 599-604. doi:10.2307/2532314
43. Lin, L. I. (2000). Total deviation index for measuring individual agreement: With application in lab performance and bioequivalence. Statistics in Medicine, 19, 255-270. doi:10.1002/(SICI)1097-0258(20000130)19:2<255:: AID-SIM293>223.0.CO;2-8
44. Lord, F. M. (1963). Elementary models for measuring change. In W. Harris (Ed.), Problems in measuring change (pp. 21-38). Madison: University of Wisconsin Press.
45. Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Erlbaum.
46. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
47. Ma, Y., Tang, W., Yu, W., & Tu, X. M. (2010). Modeling concordance correlation coefficient for longitudinal study data. Psychometrika, 75, 99-119. doi:10.1007/s11336-009-9142-z
48. McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30-46. doi:10.1037/1082-989X.1.1.30
49. Van Mechelen, I. (2009). A royal road to understanding the mechanisms underlying person in context behavior. Journal of Research in Personality, 43, 179-186. doi:10.1016/j.jrp.2008.12.012
50. Murrie, D. E., Boccaccini, M. T., Turner, D. B., Meeks, M., Woods, C., & Tussey, C. (2009). Rater (dis)agreement on risk assessment measures in sexually violent predator proceedings. Psychology, Public Policy, and Law, 15, 19-53. doi:10.1037/a0014897
51. Nickerson, C. A. (1997). Comment on "A concordance correlation coefficient to evaluate reproducibility." Biometrika, 53, 1503-1507.
52. Orom, H., & Cervone, D. (2009). Personality dynamics, meaning, and idiosyncrasy: Identifying cross-situational coherence by assessing personality architecture. Journal of Research in Personality, 43, 228-240. doi:10.1016/j.jrp.2009.01.015
53. Raykov, T., & Shrout, P. E. (2002). Reliability of scales with general structure: Point and interval estimation using a structural equation modeling approach. Structural Equation Modeling, 9, 195-212. doi:10.1207/ S15328007SEM0902_3
54. Reis, H. T. (2009). Relationships are situations, and situations involve relationships. Journal of Research in Personality, 43, 266. doi:10.1016/ j.jrp.2008.12.036
55. Robieson, W. Z. (1999). On weighted kappa and concordance correlation coefficient (Unpublished doctoral dissertation). University of Illinois at Chicago, Chicago.
56. Romesburg, H. C. (1984). Cluster analysis for researchers. Belmont, CA: Wadsworth.
57. Schall, R., & Luus, H. G. (1993). On population and individual bioequivalence. Statistics in Medicine, 12, 1109-1124.
58. Schuck, P. (2004). Assessing reproducibility for interval data in health-related quality of life questionnaires: Which coefficient should be used? Quality of Life Research, 13, 571-585. doi:10.1023/B:QURE.0000021318.92272.2a
59. Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer. Newbury Park, CA: Sage.
60. Shrout, P. E., & Fleiss, J. L. (1979). The intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420-428. doi: 10.1037/0033-2909.86.2.420
61. Sokal, R. R., & Sneath, P. H. A. (1963). Principles of numerical taxonomy. San Francisco, CA: Freeman.
62. Stevens, S. S. (1946, June 7). On the theory of scales of measurement. Science, 103, 677-680. doi:10.1126/science.103.2684.677
63. Tabachnick, B. G., & Fidell, L. S. (2007). Using multivariate statistics (5th ed.). Boston, MA: Pearson.
64. Wechsler, D. (1997). WAIS-III administration and scoring manual. San Antonio, TX. Psychological Corporation.
65. Weiss, P., & Cotsonis, G. (2005). ICCs Macro (SAS Version 9.1). [Computer software]. Retrieved from http://www.sph.emory.edu/observeragreement/software.htm
66. Willet, J. B. (1997). Measuring change: What individual growth modeling buys you. In E. Amsel & K. Renninger (Eds.), Change and development: Issues of theory, method and application (pp. 213-241). Mahwah, NJ: Erlbaum.
67. Williamson, J. M., Crawford, S. B., & Lin, H. M. (2007). Resampling dependent concordance correlation coefficients. Journal of Biopharmaceutical Statistics, 17, 685-696. doi:10.1080/10543400701329471