The Effect of Ignoring Classroom-Level Variance in Estimating the Generalizability of School Mean Scores

Educational Measurement: Issues and Practice

Volumn 30, Issue 1, 2011, Pages 13-22

  Wei, Xin ^a   Haertel, Edward ^b  

^a SRI INTERNATIONAL   (United States)

^b STANFORD UNIVERSITY   (United States)

Author keywords

Generalizability theory; Reliability; School level scores

Indexed keywords

EID: 79952760119     PISSN: 07311745     EISSN: 17453992     Source Type: Journal    
DOI: 10.1111/j.1745-3992.2010.00196.x     Document Type: Article

References (37)

1. Beaumont, C. (1991). Comparison of Henderson's Method I and restricted maximum likelihood estimation of genetic parameters of reproductive traits. Poultry Science, 70(7), 1462-1470.
2. Bowen, J., & Huang, M. (1990). A comparison of maximum likelihood with method of moment procedures for separating individual and group effects. Journal of Personality and Social Psychology, 58(1), 90-94.
3. Brennan, R. L. (1975). The calculation of reliability from a split-plot factorial design. Educational and Psychological Measurement, 35, 779-788.
4. Brennan, R. L. (1983). Elements of generalizability theory. Iowa City, IA American College Testing Program.
5. Brennan, R. L. (1995). The conventional wisdom about group mean scores. Journal of Educational Measurement, 32, 385-396.
6. Brennan, R. L. (2001a). Generalizability theory. New York Springer-Verlag.
7. Brennan, R. L. (2001b). urGENOVA (Version 2.1) [Computer software and manual]. Iowa City, IA University of Iowa. Retrieved March 8, 2008, from
8. Coleman, J. S., Campbell, E. Q., Hobson, C. J., McPartland, J., Mood, A. M., Weinfeld, F.D., et al. (1966). Equality of educational opportunity. Washington, DC US Government Printing Office.
9. Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements. New York Wiley.
10. Cronbach, L. J., Linn, R. L., Brennan, R. L, & Haertel, E. H. (1997). Generalizability analysis for performance assessments of student achievement or school effectiveness. Educational and Psychological Measurement, 57, 373-399.
11. Feldt, L. S., & Brennan, R. L. (1989). Reliability. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp.105-146). New York American Council on Education and Macmillan.
12. Gulliford, M. C., Ukoumunne, O. C., & Chinn, S. (1999). Components of variance and intraclass correlations for the design of community-based surveys and intervention studies. Data from the Health Survey for England 1994. American Journal of Epidemiology, 149, 876-883.
13. Haertel, E. H. (2006). Reliability. In R. L. Brennan (Ed.). Educational measurement (4th ed., pp. 65-110). Westport, CT American Council on Education/Praeger.
14. Hedges, L. V., & Hedberg, E. C. (2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29(1), 60-87.
15. Henderson, C. R. (1953). Estimation of variance and covariance components. Biometrics, 9, 227-252.
16. Ho, A. D. (2008). The problem with "Proficiency": Limitations of statistics and policy under No Child Left Behind. Educational Researcher, 37(6), 351-360.
17. Holland, P. (2002). Two measures of change in the gaps between the CDFs of test-score distributions. Journal of Educational and Behavioral Statistics, 27, 3-17.
18. Ingelhart, R. (1985). Aggregate stability and individual-level flux in mass belief systems: The level of analysis paradox. American Political Science Review, 79, 97-116.
19. Jones, B. S., & Norrander, B. (1996). The reliability of aggregated public opinion measures. American Journal of Political Science, 40(1), 295-309.
20. Kane, M. T., & Brennan, R. L.(1977). The generalizability of class means. Review of Educational Research, 47(2), 267-292.
21. Kane, M. T., Gillmore, G. M., & Crooks, T. J. (1976). Student evaluations of teaching: The generalizability of class means. Journal of Educational Measurement, 13(3), 171-183.
22. Kane, T. J., & Staiger, D. O. (2002). The promise and pitfalls of using imprecise school accountability measures. The Journal of Economic Perspectives, 16(4), 91-114.
23. Kim, J.-S., & Frees, E.W. (2006). Omitted variables in multilevel models. Psychometrika, 71(4), 659-690.
24. Murray, D. M., & Blitstein, J. L. (2003). Methods to reduce the impact of intraclass correlation in group randomized trials. Evaluation Review, 27, 79-103.
25. Murray, D. M., Varnell, S. P., & Blitstein, J. L. (2004). Design and analysis of group-randomized trials: Review of recent methodological developments. American Journal of Public Health, 94, 423-432.
26. No Child Left Behind Act of 2001, Pub. L. No. 197-110, 115 Stat. 1425 (2002). Retrieved March 13, 2008, from
27. O'Brien, R. M. (1990). Estimating the reliability of aggregate-level variables based on individual-level characteristics. Sociological Methods Research, 18, 473-504.
28. Opdenakker, M., & Van Damme, J. (2000). The importance of identifying levels in multilevel analysis: An illustration of the effects of ignoring the top and intermediate levels of school effectiveness research. School Effectiveness and School Improvement, 11, 103-130.
29. Rogosa, D. R. (2005). Statistical misunderstandings of the properties of school scores and school accountability. In J. L. Herman & E. H. Haertel (Eds.), Uses and misuses of data for educational accountability and improvement (The 104th yearbook of the National Society for the Study of Education, Part 2, pp. 147-174). Malden, MA Blackwell.
30. Schechtman, E., & Yitzhaki, S. (2009). Ranking groups' abilities: Is it always reliable? International Journal of Testing, 9(3), 195-214.
31. Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer. Newbury Park, CA Sage Publications.
32. Shrout, P. E., & Fleiss, J. L (1979), Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420-428.
33. U.S. Department of Education. (2009). Standards and assessments peer review guidance: Information and examples for meeting requirements of the No Child Left Behind Act of 2001. Washington, DC Office of Elementary and Secondary Education. Retrieved June 18, 2010, from
34. U.S. Department of Education (2010). Race to the Top application for phase 2 funding (CFDA Number: 84.395A). Washington, DC Office of Elementary and Secondary Education. Retrieved June 18, 2010, from
35. Verma, V., & Lee, T. (1996). An analysis of sampling errors for demographic and health surveys. International Statistical Review, 64, 265-294.
36. Yen, W. M. (1997). The technical quality of performance assessments: Standard errors of percents of pupils reaching standards. Educational Measurement: Issues and Practice, 16(3), 5-15.
37. Yen, W. M., & Ferrara, S. (1997). The Maryland School Performance Assessment Program: Performance assessment with psychometric quality suitable for high-stakes usage. Educational and Psychological Measurement, 57(1), 60-84.