Student and Teacher Ratings of Instructional Quality: Consistency of Ratings Over Time, Agreement, and Predictive Power

Journal of Educational Psychology

Volumn 108, Issue 5, 2016, Pages 705-721

  Wagner, Wolfgang ^a   Göllner, Richard ^a   Werth, Sarah ^a   Voss, Thamar ^a   Schmitz, Bernhard ^b   Trautwein, Ulrich ^a  

^a UNIVERSITY OF TÜBINGEN   (Germany)

^b DARMSTADT UNIVERSITY OF TECHNOLOGY   (Germany)

Author keywords

Instructional variability; Repeated measurements; Standardized math achievement and math self concept; Teacher student ratings of instructional quality

Indexed keywords

EID: 84942018612     PISSN: 00220663     EISSN: 19392176     Source Type: Journal    
DOI: 10.1037/edu0000075     Document Type: Article

References (71)

1. Ainsworth, M. D., Blehar, M. C., Waters, E., & Wall, D. (1978). Patterns of attachment: A psychological study of the strange situation. Hillsdale, NJ: Erlbaum.
2. Allen, J. P., Pianta, R. C., Gregory, A., Mikami, A. Y., & Lun, J. (2011). An interaction-based approach to enhancing secondary school instruction and student achievement. Science, 333, 1034-1037. http://dx.doi.org/10.1126/science.1207998
3. Ames, C. (1992). Classrooms: Goals, structures, and student motivation. Journal of Educational Psychology, 84, 261-271. http://dx.doi.org/10.1037/0022-0663.84.3.261
4. Anderson, J., & Gerbing, D. (1984). The effect of sampling error on convergence, improper solutions, and goodness-of-fit indices for maximum likelihood confirmatory factor analysis. Psychometrika, 49, 155- 173. http://dx.doi.org/10.1007/BF02294170
5. Baumert, J., Blum, W., Brunner, M., Dubberke, T. V., Jordan, A., Klusmann, U., Tsai, Y. M. (2009). Professionswissen von Lehrkräften, kognitiv aktivierender Mathematikunterricht und die Entwicklung von mathematischer Kompetenz (COACTIV): Dokumentation der Erhebungsinstrumente [Professional competence of teachers, cognitively activating instruction, and the development of students' mathematical literacy (COACTIV): Documentation of the instruments]. Berlin, Germany: Max-Planck-Institut für Bildungsforschung.
6. Baumert, J., Blum, W., Brunner, M., Krauss, S., Kunter, M., & Neubrand, M. (2004). Mathematikunterricht aus Sicht der PISA-Schülerinnen und -Schüler und ihrer Lehrkräfte. [Mathematics teaching from the perspective of the PISA students and their teachers] In M. Prenzel, J. Baumert, W. Blum, R. Lerhmann, D. Leutner, M. Neubrand, U. Schiefele (Eds.), PISA 2003. Der Bildungsstand der Jugendlichen in Deutschland- Ergebnisse des zweiten internationalen Vergleichs (pp. 314-354). Münster, Germany: Waxmann-Verlag.
7. Bos, W., Lankes, E.-M., Prenzel, M., Schwippert, K., Valtin, R., & Walther, G. (Eds.). (2005). IGLU: Vertiefende Analysen zu Leseverständnis, Rahmenbedingungen und Zusatzstudien [PIRLS Germany: In depth analysis of reading ability, determining factors and further supplemental studies]. Münster, Germany: Waxmann.
8. Bransford, J. D., Brown, A. L., & Cocking, R. R. (2000). How people learn: Brain, mind, experience, and school: Expanded ed. Washington, DC: National Academy Press.
9. Brophy, J. (1979). Teacher behavior and its effects. Journal of Educational Psychology, 71, 733-750. http://dx.doi.org/10.1037/0022-0663.71.6.733
10. Cameron, C. E., Connor, C. M., & Morrison, F. J. (2005). Effects of variation in teacher organization on classroom functioning. Journal of School Psychology, 43, 61-85. http://dx.doi.org/10.1016/j.jsp.2004.12.002
11. Carroll, J. B. (1963). A model of school learning. Teachers College Record, 64, 723-733.
12. Chen, F., Bollen, K. A., Paxton, P., Curran, P. J., & Kirby, J. B. (2001). Improper solutions in structural equation models: Causes, consequences, and strategies. Sociological Methods & Research, 29, 468-508. http:// dx.doi.org/10.1177/0049124101029004003
13. Clausen, M. (2002). Unterrichtsqualität: Eine Frage der Perspektive? Empirische Analysen zur Ü bereinstimmung, Konstukt-und Kriteriumsvalidität. [Quality of instruction: A matter of perspective?]. Münster, Germany: Waxmann.
14. Curby, T. W., Stuhlman, M., Grimm, K., Mashburn, A., Chomat-Mooney, L., Downer, J., Pianta, R. C. (2011). Within-day variability in the quality of classroom interactions during third and fifth grade: Implications for children's experiences and conducting classroom observations. The Elementary School Journal, 112, 16-37. http://dx.doi.org/10.1086/ 660682
15. De Jong, R., & Westerhof, K. J. (2001). The quality of student ratings of teacher behaviour. Learning Environments Research, 4, 51-85. http:// dx.doi.org/10.1023/A:1011402608575
16. Desimone, L., Smith, T., & Frisvold, D. (2009). Survey measures of classroom Instruction: Comparing student and teacher reports. Educational Policy, 24, 267-329. http://dx.doi.org/10.1177/0895904808330173
17. Ellis, M. W., Malloy, C. M., Meece, J. L., & Sylvester, P. (2007). Convergence of observer ratings and student perceptions of reform practices in sixth-grade mathematics classrooms. Learning Environments Research, 10, 1-15. http://dx.doi.org/10.1007/s10984-007-9022-3
18. Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Mahwah, NJ: Erlbaum.
19. Enders, C. K., & Bandolos, D. L. (2001). The relative performance to full information maximum likelihood estimation for missing data in structural equation models. Structural Equation Modeling, 8, 430-457. http://dx.doi.org/10.1207/S15328007SEM0803_5
20. Fauth, B., Decristan, J., Rieser, S., Klieme, E., & Büttner, G. (2014). Grundschulunterricht aus Schüler-, Lehrer und Beobachterperspektive: Zusammenhänge und Vorhersage von Lernerfolg. [Instructional quality in elementary school from the perspective of students, teachers, and observers: Associations and prediction of learning outcomes.]. Zeitschrift fur Padagogische Psychologie/German Journal of Educational Psychology, 28, 127-137. http://dx.doi.org/10.1024/1010-0652/ a000129
21. Fisher, D. L., & Fraser, B. J. (1981). Validity and use of My Class Inventory. Science Education, 65, 145-156. http://dx.doi.org/10.1002/ sce.3730650206
22. Fraser, B. J., & O'Brien, P. (1985). Student and teacher perceptions of the environment of elementary school classrooms. The Elementary School Journal, 85, 567-580. http://dx.doi.org/10.1086/461422
23. Garet, M. S., Wayne, A. J., Stancavage, F., Taylor, J., Eaton, M., Walters, K., Hurlburt, S. (2010). Middle school mathematics professional development impact study: Findings after the first year of implementation. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences. Retrieved from http://ies.ed.gov/ncee/pubs/20104009/pdf/20104009.pdf
24. Griffin, B. W. (2004). Grading leniency, grade discrepancy, and student ratings of instruction. Contemporary Educational Psychology, 29, 410- 425. http://dx.doi.org/10.1016/j.cedpsych.2003.11.001
25. Hamre, B. K., & Pianta, R. (2010). Classroom environments and developmental processes: Conceptualization, measurement, & improvement. In J. L. Meece & J. S. Eccles (Eds.), Handbook of research on schools, schooling and human development (pp. 25-41). New York, NY: Routledge.
26. Hamre, B. K., Pianta, R. C., & Chomat-Mooney, L. (2009). Conducting classroom observations in school-based research. In L. M. Dinella (Ed.), Conducting science-based psychology research in schools (pp. 79-105). Washington, DC: American Psychological Association. http://dx.doi.org/10.1037/11881-004
27. Hattie, J. (2008). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. New York, NY: Routledge.
28. Hox, J. J. (2010). Multilevel analysis. Techniques and applications (2nd ed.). New York, NY: Routledge.
29. Jonkmann, K., Rose, N., & Trautwein, U. (Eds.). (2013). Tradition und Innovation: Entwicklungsverläufe an Haupt- und Realschulen in Baden- Württemberg und Mittelschulen in Sachsen. Abschlussbericht für die Länder Baden-Württemberg und Sachsen [Tradition and innovation: Academic and psychosocial development in vocational track schools in the states of Baden-Württemberg and Sachsen], Unpublished project report, University of Tübingen, Tübingen, Germany.
30. Kane, T. J., & Staiger, D. O. (2012). Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains (Policy and practice brief prepared for the Bill and Melinda Gates Foundation). Retrieved from http://www.metproject.org/downloads/MET_Gathering_Feedback_Research_Paper.pdf
31. Klieme, E., Pauli, C., & Reusser, K. (2009). The Pythagoras Study: Investigating effects of teaching and learning in Swiss and German mathematics classrooms. In T. Janik & T. Seidel (Eds.), The power of video studies in investigating teaching and learning in the classroom (pp. 137-160). Münster, Germany: Waxmann.
32. Kunter, M., & Baumert, J. (2006). Who is the expert? Construct and criteria validity of student and teacher ratings of instruction. Learning Environments Research, 9, 231-251. http://dx.doi.org/10.1007/s10984- 006-9015-7
33. Kunter, M., Klusmann, U., Baumert, J., Richter, D., Voss, T., & Hachfeld, A. (2013). Professional competence of teachers: Effects on instructional quality and student development. Journal of Educational Psychology, 105, 805-820. http://dx.doi.org/10.1037/a0032583
34. Lanahan, L., McGrath, D. J., McLaughlin, M., Burian-Fitzgerald, M. A., & Salganik, L. (2005). Fundamental problems in the measurement of instructional processes: Estimating reasonable effect sizes and conceptualizing what is important to measure. Washington, DC: American Institutes for Research.
35. Larsen, R. (2011). Missing data imputation versus full information maximum likelihood with second-level dependencies. Structural Equation Modeling, 18, 649 - 662. http://dx.doi.org/10.1080/10705511.2011.607721
36. Lipowsky, F., Rakoczy, K., Pauli, C., Drollinger-Vetter, B., Klieme, E., & Reusser, K. (2009). Quality of geometry instruction and its short-term impact on students' understanding of the Pythagorean Theorem. Learning and Instruction, 19, 527-537. http://dx.doi.org/10.1016/j.learninstruc.2008.11.001
37. Lüdtke, O., Marsh, H. W., Robitzsch, A., Trautwein, U., Asparouhov, T., & Muthén, B. (2008). The multilevel latent covariate model: A new, more reliable approach to group-level effects in contextual studies. Psychological Methods, 13, 203-229. http://dx.doi.org/10.1037/ a0012869
38. Lüdtke, O., Trautwein, U., Kunter, M., & Baumert, J. (2006). Reliability and agreement of student ratings of the classroom environment: A reanalysis of TIMSS data. Learning Environments Research, 9, 215- 230. http://dx.doi.org/10.1007/s10984-006-9014-8
39. Malmberg, L.-E., Hagger, H., Burn, K., Mutton, T., & Colls, H. (2010). Observed classroom quality during teacher education and two years of professional practice. Journal of Educational Psychology, 102, 916- 932. http://dx.doi.org/10.1037/a0020920
40. Marsh, H. W., Lüdtke, O., Robitzsch, A., Trautwein, U., Asparouhov, T., Muthén, B., & Nagengast, B. (2009). Doubly-latent models of school contextual effects: Integrating multilevel and structural equation approaches to control measurement and sampling error. Multivariate Behavioral Research, 44, 764 - 802. http://dx.doi.org/10.1080/ 00273170903333665
41. Marsh, H. W., & Roche, L. A. (2000). Effects of grading leniency and low workload on students evaluations of teaching: Popular myth, bias, validity, or innocent bystanders? Journal of Educational Psychology, 92, 202-228. http://dx.doi.org/10.1037/0022-0663.92.1.202
42. Mullis, I. V. S., Martin, M. O., Beaton, A. E., Gonzales, E. J., Kelly, D. L., & Smith, T. A. (1998). Mathematics and science achievement in the final year of secondary school: IEA's Third International Mathematics and Science Study (TIMSS). Chestnut Hill, MA: Boston College Center for Study of Testing, Evaluation, and Educational Policy.
43. Muthén, L. K., & Muthén, B. O. (1998 -2012). Mplus user's guide (7th ed.). Los Angeles, CA: Muthén & Muthén.
44. Organization for Economic Co-operation and Development [OECD]. (1999). Measuring student knowledge and skills. A new framework for assessment. Paris, France: Author.
45. Pekrun, R., Götz, T., Jullien, S., Zirngibl, A., Hofe, R. v., & Blum, W. (2002). Skalenhandbuch PALMA: 1. Messzeitpunkt (5. Klassenstufe) [Scale handbook PALMA, First point of measurement (Grade 5)]. München, Germany: Universität München, Institut Pädagogische Psychologie.
46. Pianta, R. C. (2006). Schools, schooling, and developmental psychopathology. In D. Cicchetti & D. J. Cohen (Eds.), Developmental psychopathology: Vol. 1. Theory and method (pp. 494-529). Hoboken, NJ: Wiley.
47. Pianta, R. C., La Paro, K. M., & Hamre, B. K. (Eds.). (2008). Classroom Assessment Scoring System (CLASS). Baltimore, MD: Brookes.
48. Praetorius, A.-K., Pauli, C., Reusser, K., Rakoczy, K., & Klieme, E. (2014). One lesson is all you need? Stability of instructional quality across lessons. Learning and Instruction, 31, 2-12. http://dx.doi.org/10.1016/j.learninstruc.2013.12.002
49. Rakoczy, K. (2008). Motivationsunterstützung im Mathematikunterricht: Unterricht aus der Perspektive von Lernenden und Beobachtern [Motivational support in mathematics lessons: Instruction from the perspectives of learners and observers]. Münster, Germany: Waxmann.
50. Rakoczy, K., Buff, A., & Lipowsky, F. (2005). Befragungsinstrumente. [Questionnaires] In E. Klieme, C. Pauli, & K. Reusser (Eds.), Dokumentation der Erhebungs- und Auswertungsinstrumente zur schweizerisch-deutschen Videostudie "Unterrichtsqualität, Lernverhalten und mathematisches Verständnis" (Teil 1, p. 274). Frankfurt, Germany: M. GFPF/DIPF.
51. Ramm, G., Prenzel, M., Baumert, J., Blum, W., Lehmann, R., Leutner, D., Schiefele, U. (Eds.). (2006). PISA 2003. Dokumentation der Erhebungsinstrumente [Questionnaires]. Münster, Germany: Waxmann.
52. Rosenshine, B. (1970). Evaluation of classroom instruction. Review of Educational Research, 40, 279-300.
53. Scheerens, J., & Bosker, R. J. (1997). The foundations of educational effectiveness. Oxford, UK: Pergamon Press.
54. Seidel, T., & Prenzel, M. (2006). Stability of teaching patterns in physics instruction: Findings from a video study. Learning and Instruction, 16, 228-240. http://dx.doi.org/10.1016/j.learninstruc.2006.03.002
55. Seidel, T., Rimmele, R., & Prenzel, M. (2005). Clarity and coherence of lesson goals as a scaffold for student learning. Learning and Instruction, 15, 539-556. http://dx.doi.org/10.1016/j.learninstruc.2005.08.004
56. Seidel, T., & Shavelson, R. J. (2007). Teaching effectiveness research in the past decade: The role of theory and research design in disentangling meta-analysis results. Review of Educational Research, 77, 454-499. http://dx.doi.org/10.3102/0034654307310317
57. Skinner, E. A., Zimmer-Gembeck, M. J., & Connell, J. P. (1998). Individual differences and the development of perceived control. Monographs of the Society for Research in Child Development, 63(2-3, Serial No. 254). 1-220. http://dx.doi.org/10.2307/1166220
58. Snijders, T. A. B., & Bosker, R. J. (2012). Multilevel analysis: An introduction to basic and advanced multilevel modeling (2nd ed.). London, UK: Sage.
59. Steyer, R., & Schmitt, M. (1990). The effects of aggregation across and within occasions on consistency, specificity, and reliability. Methodika, 4, 58-94.
60. Steyer, R., Schmitt, M., & Eid, M. (1999). Latent state-trait theory and research in personality and individual differences. European Journal of Personality, 13, 389 - 408. http://dx.doi.org/10.1002/(SICI)1099- 0984(199909/10)13:5389:AID-PER3613.0.CO;2-A
61. Tsai, Y.-M., Kunter, M., Lüdtke, O., Trautwein, U., & Ryan, R. M. (2008). What makes lessons interesting? The role of situational and individual factors in three school subjects. Journal of Educational Psychology, 100, 460-472. http://dx.doi.org/10.1037/0022-0663.100.2.460
62. Urdan, T. (2004). Using multiple methods to assess student's perceptions of classroom goal structures. European Psychologist, 9, 222-231. http:// dx.doi.org/10.1027/1016-9040.9.4.222
63. Vygotsky, L. S. (1991). Genesis of the higher mental functions. In P. Light, S. Sheldon, & M. Woodhead (Eds.), Learning to think (pp. 32-41). Florence, KY: Routledge.
64. Wagner, W., Göllner, R., Helmke, A., Trautwein, U., & Lüdtke, O. (2013). Construct validity of student perceptions of instructional quality is high, but not perfect: Dimensionality and generalizability of domainindependent assessments. Learning and Instruction, 28, 1-11. http://dx.doi.org/10.1016/j.learninstruc.2013.03.003
65. Wagner, W., Werth, S., Trautwein, U., & Lüdtke, O. (2015). Opening the black box: Are effects of a teacher training to foster students' selfregulation on students' math competencies mediated by trained or untrained aspects of teaching practice? Manuscript in preparation.
66. Wang, M. C., Haertel, G. D., & Walberg, H. J. (1993). Toward a knowledge base for school learning. Review of Educational Research, 63, 249-294. http://dx.doi.org/10.3102/00346543063003249
67. Weinert, F. E., Schrader, F. W., & Helmke, A. (1989). Quality of instruction and achievement outcomes. International Journal of Educational Research, 13, 895-914. http://dx.doi.org/10.1016/0883-0355(89)90072-4
68. Weinstein, R. S. (1985). Student mediation of classroom expectancy effects. In J. B. Dusek (Ed.), Teacher expectancies (pp. 329-350). Hillsdale, NJ: Erlbaum.
69. Wentzel, K. R. (1999). Social-motivational processes and interpersonal relationships: Implications for understanding motivation at school. Journal of Educational Psychology, 91, 76-97. http://dx.doi.org/10.1037/ 0022-0663.91.1.76
70. Werth, S., Wagner, W., Ogrin, S., Trautwein, U., Friedrich, A., Keller, S., Schmitz, B. (2012). Förderung des selbstregulierten Lernens durch die Lehrkräftefortbildung Lernen mit Plan: Effekte auf fokale Trainingsinhalte und die allgemeine Unterrichtsqualität [Teaching teachers how to teach self-regulated learning: Effects of a training program on the promotion of self-regulation and instructional quality]. Zeitschrift fur Padagogische Psychologie/Germany Journal of Educational Psychology, 26, 291-305. http://dx.doi.org/10.1024/1010-0652/a000080
71. Yu, C. Y. (2002). Evaluating cutoff criteria of model fit indices for latent variable models with binary and continuous outcomes. Unpublished doctoral dissertation, University of California, Los Angeles.