Estimating ordinal reliability for likert-type and ordinal item response data: A conceptual, empirical, and practical guide

Practical Assessment, Research and Evaluation

Volumn 17, Issue 3, 2012, Pages 1-13

  Gadermann, Anne M ^a   Guhn, Martin ^b   Zumbo, Bruno D ^b  

^a HARVARD MEDICAL SCHOOL   (United States)

^b UNIVERSITY OF BRITISH COLUMBIA   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84865541067     PISSN: None     EISSN: 15317714     Source Type: Journal    
DOI: None     Document Type: Article

References (50)

1. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association.
2. Bentler, P. (2009). Alpha, dimension-free, and model-based internal consistency reliability. Psychometrika, 74, 137-143. doi: 10.1007/s11336-008-9100-1
3. Bernaards, C. A., & Jennrich, R. I. (2005) Gradient projection algorithms and software for arbitrary rotation criteria in factor analysis. Educational and Psychological Measurement, 65, 676-696. doi: 10.1177/0013164404272507
4. Brennan, R. L. (2001). Generalizability Theory. New York: Springer-Verlag.
5. Carroll, J. B. (1961). The nature of data, or how to choose a correlation coefficient. Psychometrika, 26, 347-372. doi: 10.1007/BF02289768
6. Cizek, G. J., Rosenberg, S., & Koons, H. (2008). Sources of validity evidence for educational and psychological tests. Educational and Psychological Measurement, 68, 397-412. doi: 10.1177/0013164407310130
7. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.
8. Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Maywah, NJ: Erlbaum.
9. Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78, 98-104. doi: 10.1037/0021-9010.78.1.98
10. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334. doi: 10.1007/BF02310555
11. Cronbach, L. J. (2004). My current thoughts on coefficient alpha and successor procedures. Educational and Psychological Measurement, 64, 391-418. doi: 10.1177/0013164404266386
12. Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements. New York: Wiley.
13. Elosua Oliden, P., & Zumbo, B. D. (2008). Coeficientes de fiabilidad para escalas de respuesta categórica ordenada. Psicothema, 20, 896-901.
14. Flora, D. B., & Curran, P. J. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9, 466-491. doi: 10.1037/1082-989X.9.4.466
15. Ford, J. K., MacCallum, R. C., & Tait, M. (1986). The application of exploratory factor-analysis in applied psychology-a critical review and analysis. Personnel Psychology, 39, 291-314.doi: 10.1111/j.1744-6570.1986.tb00583.x
16. Fox, J. (2005). The R Commander: A basic-statistics graphical user interface to R. Journal of Statistical Software, 14, 1-42. Retrieved from http://www.jstatsoft.org
17. Fox, J. (2006). Getting started with the R Commander. Retrieved from http://socserv.mcmaster.ca/jfox/Misc/Rcmdr/Getting-Started-with-the-Rcmdr.pdf
18. Fox, J. (2011). R Commander. Reference manual. Retrieved from http://cran.rproject.org/web/packages/Rcmdr/Rcmdr.pdf
19. Gelin, M. N., Beasley, T. M., & Zumbo, B. D. (2003, April). What is the impact on scale reliability and exploratory factor analysis of a Pearson correlation matrix when some respondents are not able to follow the rating scale? Paper presented at the Annual Meeting of the American Educational Research Association (AERA), Chicago, Il.
20. Green, S. B., & Yang, Y. (2009a). Reliability of summed item scores using structural equation modeling: An alternative to coefficient alpha. Psychometrika, 74, 155-167. doi: 10.1007/s11336-008-9099-3
21. Green, S. B., & Yang, Y. (2009b). Commentary on coefficient alpha: A cautionary tale. Psychometrika, 74, 121-135. doi: 10.1007/s11336-008-9098-4
22. Guhn, M., Janus, M. & Hertzman, C. (Eds.) (2007). The Early Development Instrument [Special issue]. Early Education and Development, 18(3).
23. Guhn, M., Zumbo, B. D., Janus, M., & Hertzman, C. (Eds.) (2011). Validation theory and research for a populationlevel measure of children's development, wellbeing, and school readiness [Special issue]. Social Indicators Research: An International Interdisciplinary Journal for Quality of Life Measurement, 103(2).
24. Janus, M., & Offord, D. (2007). Development and psychometric properties of the Early Development Instrument (EDI): A measure of children's school readiness. Canadian Journal of Behavioral Science, 39, 1-22. doi: 10.1037/cjbs2007001
25. Kane, M. (2006). Validation. In R. Brennan (Ed.), Educational measurement (4th ed., pp. 17-64). Washington, DC: American Council on Education and National Council on Measurement in Education.
26. Lewis, C. (2007). Classical test theory. In C. R. Rao and S. Sinharay (Eds.), Handbook of Statistics, Vol. 26: Psychometrics, (pp. 29-43). Amsterdam, The Netherlands: Elsevier Science B.V.
27. Lissitz, R. W. (Ed.) (2009). The concept of validity: Revisions, new directions and applications. Charlotte, NC: Information Age Publishing.
28. Liu, Y., Wu, A. D., & Zumbo, B. D. (2010). The impact of outliers on Cronbach's coefficient alpha estimate of reliability: Ordinal/rating scale item responses. Educational & Psychological Measurement, 70, 5-21. doi: 10.1177/0013164409344548
29. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Welsley Publishing Company.
30. Maydeu-Olivares, A., Coffman, D. L., & Hartmann, W. M. (2007). Asymptotically distribution free (ADF) interval estimation of coefficient alpha. Psychological Methods, 12, 157-176. doi: 10.1037/1082-989X.12.2.157
31. McDonald, R. P. (1985). Factor analysis and related methods. Hillsdale NJ: Erlbaum.
32. Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York, NY: McGraw-Hill.
33. Olsson, U. (1979). Maximum likelihood estimation of the polychoric correlation coefficient. Psychometrika, 44, 443-460. doi: 10.1007/BF02296207
34. Osburn, H. G. (2000). Coefficient alpha and related internal consistency reliability coefficients. Psychological Methods, 5, 343-355. doi: 10.1037/1082-989X.5.3.343
35. R Development Core Team (2011). R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org
36. Revelle, W. (2009a). Appendix A. R: Getting started. Retrieved from http://personalityproject. org/r/book/AppendixA.pdf
37. Revelle, W. (2009b). An introduction to R. Retrieved from http://personalityproject. org/r/book/R_short_course.pdf
38. Revelle, W. (2011). An overview of the psych package. Retrieved from http://www.personalitytheory.org/r/book/overview.pdf
39. Revelle, W., & Zinbarg, R. E. (2009). Coefficients alpha, beta, omega, and the glb: Comments on Sijtsma. Psychometrika, 74, 145-154. doi: 10.1007/s11336-008-9102-z
40. Rigdon, E. (1997). Not positive definite matrices-causes and cures. Retrieved from http://www2.gsu.edu/~mkteer/npdmatri.html
41. Schmitt, N. (1996). Uses and abuses of coefficient alpha. Psychological Assessment, 8, 350-353. doi: 10.1037/1040-3590.8.4.350
42. Shavelson, R. J., & Webb, N. M. (1991). Generalizability Theory: A Primer. Newbury Park, CA: Sage Publications.
43. Shavelson, R. J., & Webb, N. M., & Rowley, G. L. (1989). Generalizability theory. American Psychologist, 44, 922-932. doi: 10.1037/0003-066X.44.6.922
44. Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach's alpha. Psychometrika, 74, 107-120. doi: 10.1007/s11336-008-9101-0
45. Uebersax, J. S. (2006). The tetrachoric and polychoric correlation coefficients. Statistical Methods for Rater Agreement web site. Retrieved from http://john-uebersax.com/stat/tetra.htm
46. Wothke, W. (1993). Nonpositive definite matrices in structural modeling. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 256-93). Newbury Park, CA: Sage
47. Zinbarg, R.E., Revelle, W., Yovel, I., & Li. W. (2005). Cronbach's Alpha, Revelle's Beta, McDonald's Omega: Their relations with each other and two alternative conceptualizations of reliability. Psychometrika. 70, 123-133. doi: 10.1007/s11336-003-0974-7
48. Zumbo, B. D. (2007). Validity: Foundational issues and statistical methodology. In C. R. Rao and S. Sinharay (Eds.) Handbook of Statistics, Vol. 26: Psychometrics, (pp. 45-79). Amsterdam, The Netherlands: Elsevier Science B.V.
49. Zumbo, B. D., Gadermann, A. M., & Zeisser, C. (2007). Ordinal versions of coefficients alpha and theta for Likert rating scales. Journal of Modern Applied Statistical Methods, 6, 21-29.
50. Zumbo, B. D., & Zimmerman, D. W. (1993). Is the selection of statistical methods governed by level of measurement? Canadian Psychology, 34, 390-400. doi: 10.1037/h0078865