Understanding Measurement: Reliability

Understanding Measurement: Reliability

Volumn , Issue , 2010, Pages 1-160

  Meyer, Patrick ^a  

^a NONE

Author keywords

Classical test theory methods; Critiquing analyses; Decision consistency indices; Generalizability theory coefficients; Infer; Test scores

Indexed keywords

EID: 84869197774     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1093/acprof:oso/9780195380361.001.0001     Document Type: Book

References (112)

1. The No Child Left Behind Act of 2001, Pub. L. No. 107-110 C.F.R. (2002).
2. Allen, M. J., & Yen, W. M. (1979). Introduction to Measurement Theory. Monterey, CA:Brooks/Cole.
3. Allen, N. L., Carlson, J. E., & Zelenak, C. A. (1999). The NAEP 1996 Technical Report. Washington D.C.: National Center for Education Statistics.
4. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, D.C.: American Educational Research Association.
5. Angoff, W. H. (1953). Test reliability and effective test length. Psychometrika, 18, 1-14.
6. Becker, D. F., & Pomplun, M. R. (2006). Technical reporting and documentation. In S. M. Downing & T. M. Haladyna (Eds.), Handbook of test development (pp. 711-723). Mahwah, NJ: Lawrence Erlbaum Associates.
7. Bejar, I. I. (1991). A generative approach to psychological and educational measurement. Princeton, NJ: Educational Testing Service.
8. Bernardo, J. M. (1996). The concept of exchangeability and its applications. Far East J. Math. Sci. (Special Volume, Part II), 111-121.
9. Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee's ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
10. Borko, H., Stecher, B., & Kuffner, K. (2007). Using artifacts to characterize reformoriented instruction: The scoop notebook and rating guide. Los Angeles: National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
11. Brennan, R. L. (2001a). An essay on the history and future of reliability from the perspective of replications. Journal of Educational Measurement, 38(4), 295-317.
12. Brennan, R. L. (2001b). Generalizability Theory. New York: Springer-Verlag.
13. Brennan, R. L., & Kane, M. T. (1977). An index of dependability for mastery tests. Journal of Educational Measurement, 14, 277-289.
14. Brennan, R. L., & Wan, L. (2004). A bootstrap procedure for estimating decision consistency for single-administration complex assessments. Iowa City: Center for Advanced Studies in Measurement and Assessment.
15. Brown, W. (1910). Some experimental results in the correlation of mental abilities. British Journal of Psychology, 3, 296-322.
16. Callender, J. C., & Osburn, H. G. (1979). An empirical comparison of coefficient alpha, Guttman's lambda-2, and MSPLIT maximized split-half reliability estimates. Journal of Educational Measurement, 16, 89-99.
17. Cizek, G. J., & Bunch, M. B. (2007). Standard setting: A guide to establishing and evaluating performance standards for tests. Thousand Oaks CA: Sage.
18. Clopper, C. J., & Pearson, E. S. (1934). The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26, 404-413.
19. Cohen, J. (1960). A Coefficient of Agreement for Nominal Scales.0020 Educational and Psychological Measurement, 20(1), 37-46.
20. Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78(1), 98-104.
21. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. New York: Holt, Rinehart, and Winston.
22. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.
23. Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York:John Wiley and Sons.
24. Cronbach, L. J., Rajaratnum, N., & Gleser, G. C. (1963). Theory of generalizability: A liberalization of reliability theory. British Journal of Statistical Psychology, 16, 137-163.
25. Cronbach, L. J., Schönemann, P., & McKie, D. (1965). Alpha coefficients for Stratified-Parallel Tests. Educational and Psychological Measurement, 25(2), 291-312.
26. Cronbach, L. J., & Shavelson, R. J. (2004). My current thoughts on coefficient alpha and successor procedures. Educational and Psychological Measurement, 64, 391-418.
27. Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. Boca Raton, FL:Chapman & Hall/CRC.
28. Elliot, A. J., & McGregor, H. A. (2001). A 2 × 2 achievement goal framework. Journal of Personality and Social Psychology, 80(3), 501-519.
29. Fan, X., & Thompson, B. (2001). Confidence intervals about score reliability coefficients, please: An EPM guidelines editorial. Educational and Psychological Measurement, 6(4), 517-531.
30. Feldt, L. S. (1965). The approximate sampling distribution of Kuder-Richardson reliability coefficient 20. Psychometrika, 30, 357-370.
31. Feldt, L. S. (1975). Estimation of the reliability of a test divided into two parts of unequal length. Psychometrika, 40, 557-561.
32. Feldt, L. S. (2002). Estimating the internal consistency reliability of tests composed of testlets varying in length. Applied Measurement in Education, 15(1), 33-48.
33. Feldt, L. S., & Brennan, R. L. (1989). Reliability. In R. L. Linn (Ed.), Educational Measurement (3rd ed., pp. 105-146). New York: American Council on Education and MacMillan.
34. Feldt, L. S., & Qualls, A. L. (1996). Bias in coefficient alpha arising from heterogeneity. Applied Measurement in Education, 9(3), 277.
35. Feldt, L. S., Woodruff, D. J., & Salih, F. A. (1987). Statistical inference for coefficient alpha. Applied Psychological Measurement, 11, 93-103.
36. Finch, H., & Habing, B. (2007). Performance of DIMTEST-and NOHARM-Based Statistics for Testing Unidimensionality. Applied Psychological Measurement, 31(4), 292-307.
37. Gagné, P., & Hancock, G. R. (2006). Measurement Model Quality, Sample Size, and Solution Propriety in Confirmatory Factor Models. Multivariate Behavioral Research, 41(1), 65-83.
38. Gao, X., & Brennan, R. (2001). Variability of estimated variance components and related statistics in a performance assessment. Applied Measurement in Education, 14, 191-203.
39. Gilmer, J. S., & Feldt, L. S. (1983). Reliability estimation for a test with parts of unknown lengths. Psychometrika, 48, 99-111.
40. Glaser, R. (1963). Instructional technology and the measurement of learning outcomes. American Psychologist, 18, 519-521.
41. Green, S. B., & Hershberger, S. L. (2000). Correlated errors in true score models and their effect on coefficient alpha. Structural Equation Modeling, 7(2), 251-270.
42. Green, S. B., Lissitz, R. B., & Mulaik, S. A. (1977). Limitations of coefficient alpha as an index of test unidimensionality. Educational and Psychological Measurement, 37, 827-838.
43. Gulliksen, H. (1950). Theory of mental tests. Hillsdale, New Jersey: Lawrence Erlbaum.
44. Guttman, L. (1945). A basis for analyzing test-retest reliability. Psychometrika, 10(4), 255-282.
45. Haertel, E. H. (2006). Reliability. In R. L. Brennan (Ed.), Educational Measurement (4th ed., pp. 65-110). Westport, CT: American Council on Education and Praeger Publishers.
46. Hambleton, R. K., & Novick, M. R. (1973). Toward an integration of theory and method for criterion-referenced tests. Journal of Educational Measurement, 10(3), 159-170.
47. Hanson, B. A., & Brennan, R. L. (1990). An investigation of classification consistency indexes estimated under alternative strong true score models. Journal of Educational Measurement, 27(4), 345-359.
48. Hogg, R. V., & Tanis, E. A. (2001). Probability and statistical inference (6th ed.). Upper Saddle River, NJ: Prentice Hall.
49. Huynh, H. (1976a). On the reliability of decisions in domain-referenced testing. Journal of Educational Measurement, 13(4), 253-264.
50. Huynh, H. (1976b). Statistical consideration of mastery scores. Psychometrika, 41(1), 65-78.
51. Huynh, H., Meyer, J. P., & Barton, K. (2000). Technical documentation for the 1999 Palmetto Achievement Challenge Tests of English language arts and mathematics, grade three through eight. Columbia, SC: South Carolina Department of Education.
52. Johnson, R. L., Penny, J. A., & Gordon, B. (2009). Assessing performance: Designing, scoring, and validating performance tasks. New York: Guilford Press.
53. Jöreskog, K. G. (1971). Statistical analysis of sets of congeneric tests. Psychometrika, 36(2), 109-133.
54. Kane, M. T. (2002). Inferences about variance components and reliability-generalizability coefficients in the absence of random sampling. Journal of Educational Measurement, 39, 165-181.
55. Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational Measurement (4th ed., pp. 17-64). Westport, CT: American Council on Education and Praeger Publishers.
56. Keats, J. A. (1957). Estimation of error variance of test scores. Psychometrika, 22(1), 29-41.
57. Keats, J. A., & Lord, F. M. (1962). A theoretical distribution for mental test scores. Psychometrika, 27(1), 59-72.
58. Kelley, T. L. (1947). Fundamentals of statistics. Cambridge, MA: Harvard University Press.
59. Keppel, G., & Wickens, T. D. (2004). Design and analysis: A researcher's handbook. Upper Saddle River, NJ: Pearson Prentice Hall.
60. Kline, R. B. (1998). Principles and practice of structural equation modeling. New York:Guilford.
61. Kolen, M., J. (2006). Scaling and norming. In R. L. Brennan (Ed.), Educational Measurement (4th ed., pp. 155-186). Westport, CT: American Council on Education and Praeger Publishers.
62. Komaroff, E. (1997). Effect of simultaneous violations of essential τ-equivalence and uncorrelated error on coefficient α. Applied Psychological Measurement, 21, 337-348.
63. Kopriva, R. J., & Shaw, D. G. (1991). Power estimates: The effect of dependent variable reliability on the power of one-factor ANOVAs. Educational and Psychological Measurement, 51, 585-595.
64. Kristof, W. (1974). Estimation of reliability and true score variance from a split of a test into three arbitrary parts. Psychometrika, 39, 491-499.
65. Kuder, G. F., & Richardson, M. W. (1937). Theory and estimation of test reliability. Psychometrika, 2(3), 151-160.
66. Lee, W.-C. (2007). Multinomial and compound multinomial error models for tests with complex item scoring. Applied Psychological Measurement, 31, 255-274.
67. Linn, R. L., & Gronlund, N. E. (2000). Measurement and assessment in teaching (8th ed.). Upper Saddle River, NJ: Prentice-Hall.
68. Livingston, S. A. (1972). Criterion-referenced applications of classical test theory. Journal of Educational Measurement, 9(1), 13-26.
69. Lord, F. M. (1955a). Estimating test reliability. Educational and Psychological Measurement, 15, 325-336.
70. Lord, F. M. (1955b). Sampling fluctuations resulting from the sampling of test items. Psychometrika, 20(1), 1-22.
71. Lord, F. M. (1965). A strong true-score theory, with applications. Psychometrika, 30(3), 239-270.
72. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
73. Lucke, J. F. (2005). "Rassling the Hog": The influence of correlated error on internal consistency, classical reliability, and congeneric reliability. Applied Psychological Measurement, 29, 106-125.
74. Osburn, H. G. (2000). Coefficient alpha and related internal consistency reliability coefficients. Psychological Methods, 5(3), 343-355.
75. Pianta, R. C., La Paro, K., & Hamre, B. (2007). Classroom assessment scoring system [CLASS] Manual, K-3. Baltimore, MD: Paul H. Brookes Publishing Company.
76. Popham, W. J., & Husek, T. R. (1969). Implications of criterion-referenced measurement. Journal of Educational Measurement, 6(1), 1-9.
77. Qualls, A. L. (1995). Estimating the reliability of a test containing multiple item formats. Applied Measurement in Education, 8(2), 111-120.
78. Qualls, A. L., & Moss, A. D. (1996). The degree of congruence between test standards and test documentation within journal publications. Educational and Psychological Measurement, 56, 209-214.
79. Rajaratnam, N., Cronbach, L. J., & Gleser, G. C. (1965). Generalizability of stratifiedparallel tests. Psychometrika, 30, 39-56.
80. Raju, N. S. (1970). New formula for estimating total test reliability from parts of unequal length. Proceedings of the 78th Annual Convention of the American Psychological Association, 5, 143-144.
81. Raju, N. S. (1977). A generalization of coefficient alpha. Psychometrika, 42(4), 549-565.
82. Raykov, T. (1997). Estimation of composite reliability for congeneric measures. Applied Psychological Measurement, 21, 173-184.
83. Raykov, T. (1998). Coefficient Alpha and Composite Reliability With Interrelated Nonhomogeneous Items. Applied Psychological Measurement, 22(4), 375-385.
84. Raykov, T. (2001). Bias of coefficient α for fixed congeneric measures with correlated errors. Applied Psychological Measurement, 25, 69-76.
85. Ree, M. J., & Carretta, T. R. (2006). The role of measurement error in familiar statistics. Organizational Research Methods, 9, 99-112.
86. Reuterberg, S.-E., & Gustafsson, J.-E. (1992). Confirmatory factor analysis and reliability:Testing measurement model assumptions. Educational and Psychological Measurement, 52, 795-811.
87. Rost, J. (1990). Rasch models in latent classes: An integration of two approaches to item analysis. Applied Psychological Measurement, 14, 271-282.
88. Rulon, P. J. (1939). A simplified procedure for determining the reliability of a test by splithalves. Harvard Educational Review, 9, 99-103.
89. Schmitt, N. (1996). Uses and abuses of coefficient alpha. Psychological Assessment, 8, 350-353.
90. Shavelson, R. J., & Webb, N. (1991). Generalizability Theory: A primer. Thousand Oaks, CA: Sage.
91. South Carolina Department of Education. (1998a). Mathematics: South Carolina curriculum standards. Columbia, SC: South Carolina Department of Education.
92. South Carolina Department of Education. (1998b). PACT mathematics assessment: A blueprint for success. Columbia, SC: South Carolina Department of Education.
93. South Carolina Department of Education. (2002). English language arts curriculum standards 2002. Retrieved January 30, 2007, from
94. South Carolina Department of Education. (2005). Performance-Level descriptors for the Palmetto Achievement Challenge Tests. Columbia, SC: South Carolina Department of Education.
95. Spearman, C. (1904). The proof and measurement of association between two things. American Journal of Psychology, 15, 72-101.
96. Spearman, C. (1910). Correlation calculated from faulty data. British Journal of Psychology, 3, 271-295.
97. Stevens, J. (1996). Applied multivariate statistics for the social sciences (3rd. ed.). Mahwah, NJ: Lawrence Erlbaum Associates.
98. Stout, W. (1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52, 589-617.
99. Subkoviak, M. J. (1976). Estimating reliability from a single administration of a criterionreferenced test. Journal of Educational Measurement, 13(4), 265-276.
100. Subkoviak, M. J. (1978). Empirical investigation of procedures for estimating reliability for mastery tests. Journal of Educational Measurement, 15(2), 111-116.
101. Swaminathan, H., Hambleton, R. K., & Algina, J. J. (1975). A Bayesian decision-theoretic procedure for use with criterion-referenced tests. Journal of Educational Measurement, 12(2), 87-98.
102. Thompson, B., & Vacha-Haase, T. (2000). Psychometrics is datametrics: The test is not reliable. Educational and Psychological Measurement, 60, 174-195.
103. Traub, R. E. (1994). Reliability for the social sciences: Theory and applications. Thousand Oaks, CA: Sage.
104. Traub, R. E., & Rowley, G. L. (1991). An NCME instructional module on Understanding Reliability. Educational Measurement: Issues and Practice, 10(1), 171-179.
105. Vacha-Haase, T., Kogan, L. R., & Thompson, B. (2000). Sample compositions and variabilities in published studies versus those in test manuals: validity of score reliability inductions. Educational and Psychological Measurement, 60, 509-522.
106. van der Linden, W., & Glass, C. A. W. (Eds.). (2000). Computerized adaptive testing:Theory and practice. Dordrecht, The Netherlands: Kluwer Academic Publishers.
107. Wainer, H., Bradlow, E. T., & Wang, X. (Eds.). (2007). Testlet Response Theory and Its Applications. New York: Cambridge University Press.
108. Wainer, H., & Kiely, G. L. (1987). Item clusters and computerized adaptive testing: A case for testlets. Journal of Educational Measurement, 24, 185-201.
109. Waller, N. G. (2008). Comingled samples: A neglected source of bias in reliability analysis. Applied Psychological Measurement, 32, 211-223.
110. Whittington, D. (1998). How well do researchers report their measures?: An evaluation of measurement in published educational research. Educational and Psychological Measurement, 58, 21-37.
111. Zimmerman, D. W., & Williams, r. H. (1980). Is classical test theory 'robust' under violation of the assumption of uncorrelated errors? Canadian journal of Psychology, 34, 227-237.
112. Zimmerman, D. W., Zumbo, B. D., & LaLonde, C. (1993). Coefficient alpha as an estimate of test reliability under violation of two assumptions. Educational and Psychological Measurement, 53, 33-49.