Uncovering Multivariate Structure in Classroom Observations in the Presence of Rater Errors

Educational Measurement: Issues and Practice

Volumn 34, Issue 2, 2015, Pages 34-46

  McCaffrey, Daniel F ^a   Yuan, Kun ^b   Savitsky, Terrance D ^c   Lockwood, J R ^d   Edelen, Maria O ^b  

^a EDUCATIONAL TESTING SERVICE   (United States)

^b RAND CORPORATION   (United States)

^c BUREAU OF LABOR STATISTICS   (United States)

^d Educational Testing Service   (United States)

Author keywords

CLASS S; Classroom assessment scoring system; Classroom observation protocols; Factor analysis; Multilevel models

Indexed keywords

EID: 84931072184     PISSN: 07311745     EISSN: 17453992     Source Type: Journal    
DOI: 10.1111/emip.12061     Document Type: Article

References (36)

1. Allen, J., Gregory, A., Mikami, A., Lun, J., Hamre, B., & Pianta, R. (2013). Observations of effective teacher-student interactions in secondary school classrooms: Predicting student achievement with the CLASS-S. School Psychology Review, 42, 76-97.
2. Allen, J. P., Pianta, R. C., Gregory, A., Mikami, A. Y., & Lun, J. (2011). An interaction-based approach to enhancing secondary school instruction and student achievement. Science, 333, 1034-1037.
3. Bejar, I. I. (2012). Rater cognition: Implications for validity. Journal of Educational Measurement, 31, 2-9.
4. Bell, C. A., Gitomer, D.H., McCaffrey, D.F., Hamre, B. K., Pianta, R. C., & Qi, Y. (2012). An argument approach to observation protocol validity. Educational Assessment, 17, 62-87.
5. Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238-246.
6. Bill and Melinda Gates Foundation. (2012). Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains. Seattle, WA: Bill and Melinda Gates Foundation. Retrieved November 5, 2014, from http://www.metproject.org/downloads/MET_Gathering_Feedback_Research_Paper.pdf
7. Casabianca, J. M., Lockwood, J. R., & McCaffrey, D. F. (2014). Trends in classroom observation scores. Educational and Psychological Measurement. Advance online publication. doi:10.1177/0013164414539163
8. Casabianca, J. M., McCaffrey, D. F., Gitomer, D., Bell, C., Hamre, B. K., & Pianta, R. C. (2013). Effect of observation mode on measures of secondary mathematics teaching. Educational and Psychological Measurement, 73, 757-783.
9. Danielson, C. (2007). Enhancing professional practice: A framework for teaching (2nd ed.). Alexandria, VA: Association for Supervision and Curriculum Development.
10. Early Childhood Learning and Knowledge Center. (2014). Use of Classroom Assessment Scoring System (CLASS™) in Head Start. Retrieved November 5, 2014, from http://eclkc.ohs.acf.hhs.gov/hslc/hs/sr/class
11. Gitomer, D. H., Bell, C. A., Qi, Y., McCaffrey, D. F., Hamre, B. K., & Pianta, R. C. (2014). The instructional challenge in improving teaching quality: Lessons from a classroom observation protocol. Teachers College Record, 116(6). Retrieved June 1, 2014, from http://www.tcrecord.org/Content.asp?ContentId=17460
12. Goldstein, H. (1995). Multilevel statistical models (2nd ed.). London, UK: Arnold.
13. Grossman, P., Cohen, J., Ronfeldt, M., & Brown, L. (2014). The test matters: The relationship between classroom observation scores and teacher value added. Educational Researcher, 43, 293-303.
14. Hafen, C. A., Hamre, B. K., Allen, J. P., Bell, C. A., Gitomer, D. H., & Pianta, R. C. (2014). Teaching through interactions in secondary school classrooms: Revisiting the factor structure and practical application of the Classroom Assessment Scoring System-Secondary. Journal of Early Adolescence. Advance online publication. doi:10.1177/0272431614537117.
15. Hamre, B. K., Hatfield, B., Pianta, R. C., & Jamil, F. (2014). Evidence for general and domain-specific elements of teacher-child interactions: Associations with preschool children's development. Child Development, 85, 1257-1274.
16. Hamre, B. K., Pianta, R. C., Downer, J. T., Hakigami, A., Mashburn, A. J., Jones, S., & Brackett, M. (2011). Teaching through interactions: Testing a developmental framework of teacher effectiveness in over 4, 000 classrooms. The Elementary School Journal, 113, 461-487. doi:10.1086/669616.
17. Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indices in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.
18. Kaiser, H. F. (1958). The varimax criterion for analytic rotation in factor analysis. Psychometrika, 23, 187-200. doi:10.1007/BF02289233.
19. Lockwood, J. R., Savitsky, T. D., & McCaffrey, D. F. (2014). Inferring constructs of effective teaching from classroom observations: An application of Bayesian exploratory factor analysis without restrictions. Working paper, Educational Testing Service.
20. Malmberg, L., Hagger, H., Burn, K., Mutton, T., & Colls, H. (2010). Observed classroom quality during teacher education and two years of professional practice. Journal of Educational Psychology, 102, 916-932.
21. Mashburn, A. J., Downer, J. T., Rivers, S. E., Brackett, M. A., & Martinez, A. (2014). Improving the power of an experimental study of a social and emotional learning program: Application of generalizability theory to the measurement of classroom-level outcomes. Prevention Science, 15, 146-155. doi:10.1007/s11121--012--0357--3
22. Mashburn, A. J., Pianta, R., Hamre, B. K., Downer, J. T., Barbarin, O., Bryant, D., Burchinal, M., Clifford, R., Early, D., & Howes, C. (2008). Measures of classroom quality in pre-kindergarten and children's development of academic, language and social skills. Child Development, 79, 732-749.
23. Muthén, L. K., & Muthén, B. O. (1998-2012). Mplus user's guide (7th ed.). Los Angeles, CA: Muthén & Muthén.
24. Muthén, B. O. (1994). Multilevel covariance structure analysis. Sociological Methods and Research, 22, 376-398.
25. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory. New York, NY: McGraw Hill.
26. Pianta, R.C., Hamre, B. K., Haynes, N. J., Mintz, S. L., & La Paro, K. M. (2009). Classroom Assessment Scoring System (CLASS): Secondary manual. Charlottesville, VA: University of Virginia Center for Advanced Study of Teaching and Learning.
27. Pianta, R. C., La Paro, K. M., & Hamre, B. K. (2006). CLASS: Classroom Assessment Scoring System (Manual for Middle/Secondary School, Version June 2006). Unpublished manuscript, University of Virginia.
28. Pianta, R. C., La Paro, K. M., & Hamre, B. K. (2008). The Classroom Assessment Scoring System. Manual, pre-K. Baltimore, MD: Brookes.
29. Savitsky, T. D., & McCaffrey, D. F. (2014). Bayesian hierarchical multivariate formulation with factor analysis for nested ordinal data. Psychometrika, 79, 275-302.
30. Schreiber, J. B., Nora, A., Stage, F. K., Barlow, E. A., & King, J. (2006). Reporting structural equation modeling and confirmatory factor analysis results: A review. The Journal of Educational Research, 99, 323-338.
31. Sinharay, S., & Haberman, S. J. (2008). Reporting subscores: A survey (ETS Research Memorandum No. RM-08-18). Princeton, NJ: Educational Testing Service.
32. Stapleton, L. M., & Yang, J. S. (2014, April). A framework for structural aspects of construct validation in multilevel settings. Paper presented at the meeting of the American Education Research Association, Philadelphia, PA.
33. Steiger, J. H. (1989). EzPATH: Causal modeling. Evanston, IL: SYSTAT.
34. Thorndike, E. L. (1920). A constant error in psychological ratings. Journal of Applied Psychology, 4, 25-29.
35. Tucker, L. R., & Lewis, C. (1973). The reliability coefficient for maximum likelihood factor analysis. Psychometrica, 38, 1-10.
36. Wasik, B. A., & Hindman, A. H. (2011). Improving vocabulary and pre-literacy skills of at-risk preschoolers through teacher professional development. Journal of Educational Psychology, 103, 455-469.