Cross-Level Measurement Invariance in School and Classroom Environment Surveys: Implications for Policy and Practice

Educational Evaluation and Policy Analysis

Volumn 36, Issue 3, 2014, Pages 259-280

  Schweig, Jonathan ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

educational policy; factor analysis; learning environments; research methodology; school teacher effectiveness

Indexed keywords

EID: 84908572320     PISSN: 01623737     EISSN: 19351062     Source Type: Journal    
DOI: 10.3102/0162373713509880     Document Type: Article

References (63)

1. Alker H. R.Quantitative ecological analysis in the social sciences. Dogan M.Stein R., ed. Cambridge, MA: MIT Press ; 1969:69-86.
2. Alwin D. F. The use of factor analysis in the construction of linear composites in social research.Sociological Methods and Research. 1973;2: 191-214
3. BalchR. T. (2012). The validation of a student survey on teacher practice (Unpublished doctoral dissertation). Graduate School of Vanderbilt University, Nashville, TN.
4. Learning about teaching: Initial findings from the measures of effective teaching project. Seattle, WA: Author ; 2010:
5. Bliese P. D.Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions. Klein K. J.Kozlowski S. W. J., ed. San Francisco, CA: Jossey-Bass ; 2000:349-381.
6. Bollen K. A.Structural equations with latent variables. New York, NY: Wiley ; 1989:
7. Bollen K. A.,Bauldry S. Three Cs in measurement models: Causal indicators, composite indicators, and covariates.Psychological Methods. 2011;16: 265-284
8. Bollen K. A.,Lennox R. Conventional wisdom on measurement: A structural equation perspective.Psychological Bulletin. 1991;110: 305-314
9. Briggs N. E.,MacCallum R. C. Recovery of weak common factors by maximum likelihood and ordinary least squares estimation.Multivariate Behavioral Research. 2003;38: 25-56
10. Browne M. W.,MacCallum R. C.,Kim C. T.,Andersen B. L.,Glaser R. When fit indices and residuals are incompatible.Psychological Methods. 2002;7: 403-421
11. ButrymowiczS. A. (2012, May14). Student surveys for children as young as 5 years old may help rate teachers. The Washington Post. Retrieved from http://www.washingtonpost.com/local/education/student-surveys-may-help-rate-teachers/2012/05/11/gIQAN78uMU_story.html
12. Chan D. Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models.Journal of Applied Psychology. 1998;83: 234-246
13. Crawford C. B.,Koopman P. A note on Horn’s test for the number of factors in factor analysis.Multivariate Behavioral Research. 1973;8: 117-125
14. Cronbach L. J.. Occasional Paper of the Stanford Evaluation Consortium; 1976Stanford University, Stanford, CA; 1976.
15. Cronbach L. J.,Webb N. Between-class and within-class effects in a reported aptitude × treatment interaction: Reanalysis of a study by G. L. Anderson.Journal of Educational Psychology. 1975;67: 717-724
16. Croon M. A.,van Veldhoven M. J. Predicting group-level outcome variables from variables measured at the individual level: A latent variable multilevel model.Psychological Methods. 2007;12: 45-57
17. D’haenens E.,Van Damme J.,Onghena P. Multilevel exploratory factor analysis: Illustrating its surplus value in educational effectiveness research.School Effectiveness and School Improvement. 2010;21: 209-235
18. DinnoA. (2012). paran: Horn’s Test of Principal Components/Factors [R package version 1.5.1]. Retrieved from http://CRAN.R-project.org/package=paran
19. DiStefano C.,Monrad D. M.,May R. J.,Smith J.,Gay J.,Mindrila D.,Rawls A.. Paper presented at the annual meeting of the American Educational Research Association; 2008New York, NY; 2008.
20. Fabrigar L. R.,Wegener D. T.,MacCallum R. C.,Strahan E. J. Evaluating the use of exploratory factor analysis in psychological research.Psychological Methods. 1999;4: 272-299
21. Ferguson R.Student perceptions of teaching effectiveness. Cambridge, MA: Harvard University ; 2010:
22. Floyd F. J.,Widaman K. F. Factor analysis in the development and refinement of clinical assessment instruments.Psychological Assessment. 1995;7: 286-299
23. Ford J. C.,MacCallum R. C.,Tait M. The application of exploratory factor analysis in applied psychology: A critical review and analysis.Personnel Psychology. 1986;39: 291-314
24. Harnqvist K. Primary mental abilities at collective and individual levels.Journal of Educational Psychology. 1978;70: 706-716
25. Harris A.Effective leadership for school improvement. Harris A.Day C.Hopkins D.Hadfield M.Hargreaves A.Chapman C., ed. London, England: Routledge ; 2003:72-83.
26. Harris A. Distributed leadership and school improvement leading or misleading?.Educational Management Administration & Leadership. 2004;32: 11-24
27. Hayton J. C.,Allen D. G.,Scarpello V. Factor retention decisions in exploratory factor analysis: A tutorial on parallel analysis.Organizational Research Methods. 2004;7: 191-205
28. Holfve-Sabel M.,Gustafsson J. Attitudes towards school, teacher, and classmates at classroom and individual levels: An application of two-level confirmatory factor analysis.Scandinavian Journal of Educational Research. 2005;49: 187-202
29. Horn J. L. A rationale and test for the number of factors in factor analysis.Psychometrika. 1965;30: 179-185
30. Hoy W. K.,Clover S. I. R. Elementary school climate: A revision of the OCDQ.Educational Administration Quarterly. 1986;22: 93-110
31. Humphreys L. G.,Montanelli R. G. An investigation of the parallel analysis criterion for determining the number of common factors.Multivariate Behavioral Research. 1975;10: 193-205
32. Ingersoll R. Teacher turnover and teacher shortages: An organizational analysis.American Educational Research Journal. 2001;38: 499-534
33. JöreskogK. G. (1999). How large can a standardized coefficient be? Retrieved from http://www.ssicentral.com/lisrel/techdocs/HowLargeCanaStandardizedCoefficientbe.pdf
34. Julian M. W. The consequences of ignoring multilevel data structures in nonhierarchical covariance modeling.Structural Equation Modeling. 2001;8: 325-352
35. Ladd H. Teachers’ perceptions of their working conditions: How predictive of planned and actual teacher movement?.Educational Evaluation and Policy Analysis. 2011;33: 235-261
36. Lee S. Y. Multilevel analysis of structural equation models.Biometrika. 1990;77: 763-772
37. Loeb S.,Darling-Hammond L.,Luczak J. How teaching conditions predict teacher turnover in California schools.Peabody Journal of Education. 2005;80: 44-70
38. Longford N. T.,Muthén B. O. Factor analysis for clustered observations.Psychometrika. 1992;57: 581-597
39. Lüdtke O.,Marsh H. W.,Robitzsch A.,Trautwein U. A 2 × 2 taxonomy of multilevel latent contextual models: Accuracy and bias trade-offs in full and partial error-correction models.Psychological Methods. 2011;16: 444-467
40. Marsh H. W.,Lüdtke O.,Nagengast B.,Trautwein U.,Morin A. J.,Abduljabbar A. S.,Köller O. Classroom climate and contextual effects: Conceptual and methodological issues in the evaluation of group-level effects.Educational Psychologist. 2012;47: 106-124
41. Marsh H. W.,Lüdtke O.,Robitzsch A.,Trautwein U.,Asparouhov T.,Muthén B. Doubly-latent models of school contextual effects: Integrating multilevel and structural equation approaches to control measurement and sampling error.Multivariate Behavioral Research. 2009;44: 764-802
42. McDonald R. P.,Goldstein H. Balanced versus unbalanced designs for linear structural relations in two-level data.British Journal of Mathematical and Statistical Psychology. 1989;42: 215-232
43. Meredith W. Measurement invariance, factor analysis and factorial invariance.Psychometrika. 1993;58: 525-543
44. The MetLife Survey of the American Teacher: Collaborating for Student Success. (2009). Retrieved from https://www.metlife.com/assets/cao/contributions/foundation/american-teacher/MetLife_Teacher_Survey_2009_Part_1.pdf
45. Mihaly K.,McCaffrey D. F.,Staiger D. O.,Lockwood J. R.A composite estimator of effective teaching. Seattle, WA: Bill & Melinda Gates Foundation ; 2013:
46. Muijs D.,Harris A. Teacher leadership—Improvement through empowerment? An overview of the literature.Educational Management Administration & Leadership. 2003;31: 437-448
47. Muthén B. O. Multilevel factor analysis of class and student achievement components.Journal of Educational Measurement. 1991;28: 338-354
48. Muthén B. O. Multilevel covariance structure analysis.Sociological Methods & Research. 1994;22: 376-398
49. Muthén B. O.,Asparouhov T.Handbook of advanced multilevel analysis. Hox J.Roberts J. K., ed. New York, NY: Taylor & Francis ; 2011:15-40.
50. Muthén B. O.,Muthén L. K.Los Angeles, CA: Author ; 2010:
51. Santa Cruz, CA: Author ; 2009:
52. New York City School Survey. (n.d.). Retrieved from http://schools.nyc.gov/Accountability/tools/survey/default.htm
53. Preacher K. J.,Zyphur M. J.,Zhang Z. A general multilevel SEM framework for assessing multilevel mediation.Psychological Methods. 2010;15: 209-233
54. Rabe-Hesketh S.,Skrondal A.,Zheng X.Handbook of latent variable and related models. Lee S.-Y., ed. Amsterdam, Netherlands: Elsevier ; 2007:209-227.
55. Raudenbush S. W.,Bryk A. S.Hierarchical linear models. 2002. Hierarchical linear models (2nd ed.). Newbury Park, CA: SAGE. Newbury Park, CA: SAGE ; 2002:
56. Raudenbush S. W.,Sadoff S. Statistical inference when classroom quality is measured with error.Journal of Research on Educational Effectiveness. 2008;1: 138-154
57. Reise R. P.,Ventura J.,Nuechterlein K. H.,Kim K. H. An illustration of multilevel factor analysis.Journal of Personality Assessment. 2005;84: 126-136
58. Ryan A. M.,Patrick H. The classroom social environment and changes in adolescents’ motivation and engagement during middle school.American Educational Research Journal. 2001;28: 437-460
59. Schmitt T. A. Current methodological considerations in exploratory and confirmatory factor analysis.Journal of Psychoeducational Assessment. 2011;29: 304-321
60. Skrondal A.,Laake P. Regression among factor scores.Psychometrika. 2001;66: 563-575
61. Toland M. D.,De Ayala R. J. A multilevel factor analysis of students’ evaluations of teaching.Educational and Psychological Measurement. 2005;65: 272-296
62. Van de Vijver F. J.,Poortinga Y. H. Structural equivalence in multilevel research.Journal of Cross-Cultural Psychology. 2002;33: 141-156
63. Zyphur M.,Kaplan S.,Christian M. Assumptions of cross-level measurement and structural invariance in the analysis of multilevel data: problems and solutions.Group Dynamics: Theory, Research, and Practice. 2008;12: 127-140