4 Reliability Coefficients and Generalizability Theory

Handbook of Statistics

Volumn 26, Issue , 2006, Pages 81-124

  Webb, Noreen M ^a   Shavelson, Richard J ^a   Haertel, Edward H ^a  

^a NONE

Author keywords

[No Author keywords available]

Indexed keywords

EID: 67649371508     PISSN: 01697161     EISSN: None     Source Type: Book Series    
DOI: 10.1016/S0169-7161(06)26004-8     Document Type: Review

References (82)

1. Bachman L.F., Lynch B.K., and Mason M. Investigating variability in tasks and rater judgements in a performance test of foreign language speaking. Language Testing 12 (1995) 238-257
2. Bock R.D., Brennan R.L., and Muraki E. The information in multiple ratings. Applied Psychological Measurement 26 (2002) 364-375
3. Box G.E.P., and Tiao G.C. Bayesian Inference in Statistical Analysis (1973), Addison-Wesley, Reading, MA
4. Brennan R.L. Variance components in generalizability theory. In: Reynolds C.R. (Ed). Cognitive Assessment: A Multidisciplinary Perspective (1994), Plenum Press, New York 175-207
5. Brennan R.L. The conventional wisdom about group mean scores. Journal of Educational Measurement 32 (1995) 385-396
6. Brennan R.L. (1996). Conditional standard errors of measurement in generalizability theory. Iowa Testing Programs Occasional Paper No. 40. Iowa Testing Programs, University of Iowa, Iowa City, IA
7. Brennan R.L. A perspective on the history of generalizability theory. Educational Measurement: Issues and Practice 16 (1997) 14-20
8. Brennan R.L. Raw-score conditional standard errors of measurement in generalizability theory. Applied Psychological Measurement 22 (1998) 307-331
9. Brennan R.L. (Mis)conceptions about generalizability theory. Educational Measurement: Issues and Practice 19 (2000) 5-10
10. Brennan R.L. Generalizability Theory (2001), Springer-Verlag, New York
11. Briggs, D.C. Wilson, M. (2004). Generalizability in item response modeling. Paper presented at the 2004 Annual Meeting of the Psychometric Society, Pacific Grove, CA
12. Burdick R.K., and Graybill F.A. Confidence Intervals on Variance Components (1992), Dekker, New York
13. Cardinet J., Tourneur Y., and Allal L. The symmetry of generalizability theory: Applications to educational measurement. Journal of Educational Measurement 13 (1976) 119-135
14. Cardinet J., Tourneur Y., and Allal L. Extension of generalizability theory and its applications in educational measurement. Journal of Educational Measurement 18 (1981) 183-204
15. Chiu C.W.T., and Wolfe E.W. A method for analyzing sparse data matrices in the generalizability theory framework. Applied Psychological Measurement 26 (2002) 321-338
16. Chiu C.W.T. Generalizability Theory: Scoring Large-Scale Assessments based on Judgments (2001), Kluwer Academic, Boston, MA
17. Cornfield J., and Tukey J.W. Average values of mean squares in factorials. Annals of Mathematical Statistics 27 (1956) 907-949
18. Crick, J.E., Brennan, R.L. (1983). GENOVA: A generalized analysis of variance system [Computer software and manual]. University of Iowa, Iowa City, IA. Available from http://www.education.uiowa.edu/casma/
19. Cronbach L.J. Coefficient alpha and the internal structure of tests. Psychometrika 16 (1951) 297-334
20. Cronbach L.J. My current thoughts on coefficient alpha and successor procedures. Educational and Psychological Measurement 64 (2004) 391-418
21. Cronbach L.J., Rajaratnman N., and Gleser G.C. Theory of generalizability: A liberalization of reliability theory. British Journal of Statistical Psychology 16 (1963) 137-163
22. Cronbach L.J., Gleser G.C., Nanda H., and Rajaratnam N. The Dependability of Behavioral Measurements (1972), Wiley, New York
23. Cronbach L.J., Linn R.L., Brennan R.L., and Haertel E.H. Generalizability analysis for performance assessments of student achievement or school effectiveness. Educational and Psychological Measurement 57 (1997) 373-399
24. de Finetti B.D. Foresight: Its logical laws, its subjective sources. Annales de l'Institut Henri Poincare 7 (1937) 95-158
25. de Finetti B.D. Foresight: Its logical laws, its subjective sources. In: Kyburg H.E., and Smokler G.E. (Eds). Studies in Subjective Probability (1964), Wiley, New York 95-158
26. Efron B. The Jackknife, the Bootstrap, and Other Resampling Plans (1982), SIAM, Philadelphia, PA
27. Efron B., and Tibshirani R. Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Statistical Science 1 (1986) 54-77
28. Embertson S.E., and Reise S.P. Item Response Theory as Model-Based Measurement. Item Response Theory for Psychologists (2000), Erlbaum, Mahwah, NJ
29. Erlich O., and Shavelson R.J. The search for correlations between measures of teacher behavior and student achievement: Measurement problem, conceptualization problem, or both?. Journal of Educational Measurement 15 (1978) 77-89
30. Feldt L.S., and Brennan R.L. Reliability. In: Linn R.L. (Ed). Educational Measurement. 3rd ed (1989), The American Council on Education, Macmillan 105-146
31. Feller W. An Introduction to Probability Theory and its Applications (1966), Wiley, New York
32. Fyans Jr., L.J. (1977). A new multiple level approach to cross-cultural psychological research. Unpublished doctoral dissertation. University of Illinois at Urbana-Champaign
33. Goodenough F.L. A critical note on the use of the term 'reliability' in mental measurement. Journal of Educational Psychology 27 (1936) 173-178
34. Haertel, E.H. (in press). Reliability. In: Brennan, R.L. (Ed.), Educational Measurement. 4th ed. Greenwood, Westport, CT
35. Hambleton R.K., Swaminathan H., and Rogers H.J. Fundamentals of Item Response Theory (1991), Sage Publications, Newbury Park, CA
36. Henderson C.R. Estimation of variance and covariance components. Biometrics 9 (1953) 227-252
37. Joe G.W., and Woodward J.A. Some developments in multivariate generalizability. Psychometrika 41 (1976) 205-217
38. Kane M.T., and Brennan R.L. The generalizability of class means. Review of Educational Research 47 (1977) 267-292
39. Kane M.T., Gillmore G.M., and Crooks T.J. Student evaluations of teaching: The generalizability of class means. Journal of Educational Measurement 13 (1976) 171-183
40. Kolen, M.J., Harris, D.J. (1987). A multivariate test theory model based on item response theory and generalizability theory. Paper presented at the Annual Meeting of the American Educational Research Association, Washington, DC
41. Linacre J.M., and Wright B.D. A User's Guide to FACETS: Rasch Model Computer Program, Version 2.4 for PC-Compatible Computers (1993), MESA Press, Chicago
42. Lord F.M. Estimating test reliability. Educational and Psychological Measurement 16 (1955) 325-336
43. Lord F.M. Applications of Item Response Theory to Practical Testing Problems (1980), Erlbaum, Hillsdale, NJ
44. Lynch B.K., and McNamara T.F. Using G-theory and many-facet Rasch measurement in the development of performance assessments of the ESL speaking skills of immigrants. Language Testing 15 (1998) 158-180
45. Marcoulides G.A., and Drezner Z. A procedure for detecting pattern clustering in measurement designs. In: Wilson M., and Engelhard G. (Eds). Objective Measurement: Theory into Practice vol. 5 (2000), Ablex, Stamford, CT 287-303
46. Marcoulides G.A., and Goldstein Z. Selecting the number of observations in multivariate measurement studies under budget constraints. Educational and Psychological Measurement 51 (1991) 573-584
47. Marcoulides G.A., and Goldstein Z. The optimization of multivariate generalizability studies with budget constraints. Educational and Psychological Measurement 52 (1992) 301-309
48. Marcoulides, G.A. (1987). An alternative method for variance component estimation: Applications to generalizability theory. Unpublished doctoral dissertation. University of California, Los Angeles, CA
49. Marcoulides G.A. An alternate method for estimating variance components in generalizability theory. Psychological Reports 66 (1990) 379-386
50. Marcoulides G.A. Maximizing power in generalizability studies under budget constraints. Journal of Educational Statistics 18 (1993) 197-206
51. Marcoulides G.A. Selecting weighting schemes in multivariate generalizability studies. Educational and Psychological Measurement 54 (1994) 3-7
52. Marcoulides G.A. Designing measurement studies under budget constraints: Controlling error of measurement and power. Educational and Psychological Measurement 55 (1995) 423-428
53. Marcoulides G.A. Generalizability theory: Picking up where the Rasch IRT model leaves off?. In: Embretson S.E., and Hershberger S.L. (Eds). The New Rules of Measurement: What Every Psychologist Should Know (1999), Erlbaum, Mahwah, NJ 129-152
54. MathSoft Inc. S-Plus 4.5 Standard Edition (1997), Author, Cambridge, MA
55. Sanders P.F. Alternative solutions for optimizing problems in generalizability theory. Psychometrika 57 (1992) 351-356
56. Sanders P.F., Theunissen T.J.J.M., and Baas S.M. Maximizing the coefficient of generalizability under the constraint of limited resources. Psychometrika 56 (1991) 87-96
57. SAS Institute, Inc. The SAS System for Windows Release 6.12 (1996), Author, Cary, NC
58. Satterthwaite F.E. Synthesis of variance. Psychometrika 6 (1941) 309-316
59. Satterthwaite F.E. An approximate distribution of estimates of variance components. Biometrics 2 (1946) 110-114
60. Searle S.R., Casella G., and McCulloch C.E. Variance Components (1992), Wiley, New York
61. Searle S.R. Linear Models (1971), Wiley, New York
62. Shao J., and Tu D. The Jackknife and the Bootstrap (1995), Springer-Verlag, New York
63. Shavelson R.J. Editor's preface to Lee J. Cronbach's "My current thoughts on coefficient alpha and successor procedures". Educational and Psychological Measurement 64 (2004) 389-390
64. Shavelson R.J., and Webb N.M. Generalizability theory: 1973-1980. British Journal of Mathematical and Statistical Psychology 34 (1981) 133-165
65. Shavelson R.J., and Webb N.M. Generalizability Theory: A Primer (1991), Sage Publications, Newbury Park, CA
66. Shavelson R.J., Baxter G.P., and Gao X. Sampling variability of performance assessments. Journal of Educational Measurement 30 (1993) 215-232
67. Shavelson R.J., Ruiz-Primo M.A., and Wiley E.W. Note on sources of sampling variability in science performance assessments. Journal of Educational Measurement 36 (1999) 61-71
68. Short, L.M., Shavelson, R.J., Webb, M.N. (1986). Issues in multivariate generalizability: Weighting schemes and dimensionality. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, CA
69. Smith E.V., and Kulikowich J.M. An application of generalizability theory and many-facet Rasch measurement using a complex problem-solving skills assessment. Educational and Psychological Measurement 64 (2004) 617-639
70. Smith P.L. Sampling errors of variance components in small sample multifacet generalizability studies. Journal of Educational Statistics 3 (1978) 319-346
71. Smith P.L. Gaining accuracy in generalizability theory: Using multiple designs. Journal of Educational Measurement 18 (1981) 147-154
72. SPSS, Inc. SPlanet. Space Sci. for Windows Release 8.0.0 (1997), Author, Chicago
73. Stahl, J.A., Lunz, M.E. (1992). A comparison of generalizability theory and multi-faceted Rasch measurement. Paper presented at the Midwest Objective Measurement Seminar, University of Chicago
74. Thorndike, R.L. (1947). Research Problems and Techniques. Report No. 3. AAF Aviation Psychology Program Research Reports. U.S. Government Printing Office, Washington, DC
75. Ting N., Burdick R.K., Graybill F.A., Jeyaratnam S., and Lu T.C. Confidence intervals on linear combinations of variance components that are unrestricted in sign. Journal of Statistical Computational Simulation 35 (1990) 135-143
76. Tukey J.W. Bias and confidence in not quite large samples. Annals of Mathematical Statistics 29 (1958) 614
77. Webb N.M., Schlackman J., and Sugrue B. The dependability and interchangeability of assessment methods in science. Applied Measurement in Education 13 (2000) 277-301
78. Webb N.M., Shavelson R.J., Kim K.S., and Chen Z. Reliability (generalizability) of job performance measurements: Navy machinists mates. Military Psychology 1 (1989) 91-110
79. Wiley, E. (2000). Bootstrap strategies for variance component estimation: Theoretical and empirical results. Unpublished doctoral dissertation. Stanford University
80. Wood R. Trait measurement and item banks. In: DeGruijter D.N.M., and van der Kamp L.J.T. (Eds). Advances in Psychological and Educational Measurement (1976), Wiley, New York 247-263
81. Woodward J.A., and Joe G.W. Maximizing the coefficient of generalizability in multi-facet decision studies. Psychometrika 38 (1973) 173-181
82. Yen W.M. The technical quality of performance assessments: Standard errors or percents of students reaching standards. Educational Measurement: Issues and Practice 16 (1997) 5-15