One lesson is all you need? Stability of instructional quality across lessons

Learning and Instruction

Volumn 31, Issue , 2014, Pages 2-12

  Praetorius, Anna Katharina ^a   Pauli, Christine ^b   Reusser, Kurt ^b   Rakoczy, Katrin ^c   Klieme, Eckhard ^c  

^a UNIVERSITY OF AUGSBURG   (Germany)

^b UNIVERSITY OF ZURICH   (Switzerland)

^c GERMAN INSTITUTE FOR INTERNATIONAL EDUCATIONAL RESEARCH   (Germany)

Author keywords

Cognitive activation; Generalizability theory; Instructional quality; Number of lessons; Observer ratings

Indexed keywords

EID: 84892461116     PISSN: 09594752     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.learninstruc.2013.12.002     Document Type: Article

References (85)

1. Anderson L.W., Burns R.B. Research in classrooms: The study of teachers, teaching and instruction 1989, Pergamon Press, Oxford.
2. Baumert J., Kunter M., Blum W., Brunner M., Voss T., Jordan A., et al. Teachers' mathematical knowledge, cognitive activation in the classroom, and student progress. American Educational Research Journal 2010, 47:133-180. 10.3102/0002831209345157.
3. Baumert J., Lehmann R., Lehrke M., Schmitz B., Clausen M., Hosenfeld I., et al. TIMSS - Mathematisch-naturwissenschaftlicher Unterricht im internationalen Vergleich: Deskriptive Befunde 1997, Leske & Budrich, Opladen, Germany.
4. Berliner D.C. The near impossibility of testing for teacher quality. Journal of Teacher Education 2005, 56:205-213. 10.1177/0022487105275904.
5. Brennan R.L. Generalizability theory 2001, Springer, New York, NY.
6. Brennan R.L. Manual for urGenova 2001, (Version 2.1), Iowa Testing Programs, University of Iowa, Iowa City, IA.
7. Brophy J. Teaching 2000, International Academy of Education, Brussels, Belgium.
8. Brophy J. Observational research on generic aspects of classroom teaching. Handbook of educational psychology 2006, 755-780. Erlbaum, Mahwah, NJ. P.A. Alexander, P.H. Winne (Eds.).
9. Bürgermeister A., Klimczak M., Klieme E., Rakoczy K., Blum W., Leiß D., et al. Leistungsbeurteilung im Mathematikunterricht: Eine Darstellung des Projekts "Nutzung und Auswirkungen der Kompetenzmessung in mathematischen Lehr-Lernprozessen". Schulpädagogik - Heute 2011, 2:1-18.
10. Calkins D., Borich G.D., Pascone M., Kluge S., Marston P.T. Generalizability of teacher behaviors across classroom observation systems. Journal of Classroom Interaction 1997, 13:9-22.
11. Chiu C.W.T., Wolfe E.W. Amethod for analyzing sparse data matrices in the generalizability theory framework. Applied Psychological Measurement 2002, 26:321-338. 10.1177/0146621602026003006.
12. Clare L., Valdés R., Pascal J., Steinberg J. Teachers' assignments as indicators of instructional quality in elementary schools 2001, (CSE technical report no. 545), Center for Research on Evaluation, Standards, and Student Testing, University of California, Los Angeles, CA.
13. Clarke D., Keitel C., Shimizu Y. Mathematics classrooms in twelve countries - The insider's perspective 2006, Sense Publishers, Rotterdam, Netherlands.
14. Clarke D., Mesiti C., O'Keefe C., Xu L.H., Jablonka E., Mok I.A.C., et al. Addressing the challenge of legitimate international comparisons of classroom practice. International Journal of Educational Research 2007, 46:280-293. 10.1016/j.ijer.2007.10.009.
15. Clausen M. Qualität von Unterricht - Eine Frage der Perspektive? 2002, Waxmann, Münster, Germany.
16. Cohen D.K. Teaching for understanding: Challenges for policy and practice 1993, Jossey-Bass, San Francisco.
17. Creemers B.P.M., Kyriakides L. The dynamics of educational effectiveness: A contribution to policy, practice, and theory in contemporary schools 2008, Routledge, London.
18. Crick J.E., Brennan R.L. Manual for GENOVA: A generalized analysis of variance system 1983, (American College Testing technical bulletin no. 43), ACT, Inc, Iowa City, IA.
19. Dalehefte I.M., Rimmele R., Prenzel M., Seidel T., Labudde P., Herweg C. Observing instruction "next-door": a video study about science teaching and learning in Germany and Switzerland. The power of video studies in investigating teaching and learning in the classroom 2009, 83-99. Waxmann, Münster, Germany. T. Janik, T. Seidel (Eds.).
20. Deci E.L., Ryan R.M. Intrinsic motivation and self-determination in human behavior 1985, Plenum, New York.
21. Doyle W. Paradigms for research on teacher effectiveness. Review of Research in Education 1977, 5:163-198. 10.3102/0091732X005001163.
22. Erlich O., Shavelson R.J. The search for correlations between measures of teacher behavior and student achievement: measurement problem, conceptualization problem, or both?. Journal of Educational Measurement 1978, 15:77-89. 10.1111/j.1745-3984.1978.tb00059.x.
23. Helmke A. Unterrichtsqualität und Lehrerprofessionalität: Diagnose, evaluation und Verbesserung des Unterrichts 2009, Klett-Kallmeyer, Seelze, Germany.
24. Helmke A., Helmke T., Heyne N., Hosenfeld A., Hosenfeld I., Schrader F.-W., et al. Zeitnutzung im Grundschulunterricht. Ergebnisse der Unterrichtsstudie "VERA - Gute Unterrichtspraxis". Zeitschrift für Grundschulforschung 2008, 1:23-36.
25. Helmke T., Helmke A., Schrader F.-W., Wagner W., Nold G., Schröder K. Die Videostudie des Englischunterrichts. Unterricht und Kompetenzerwerb in Deutsch und Englisch. Ergebnisse der DESI-Studie 2008, 345-363. Beltz, Weinheim. E. Klieme (Ed.).
26. Hiebert J., Gallimore R., Garnier H., Givvin K.B., Hollingsworth H., Jacobs J., et al. Teaching mathematics in seven countries. Results from the TIMSS 1999 video study 2003, U.S. Department of Education, National Center for Education Studies, Washington, DC.
27. Hiebert J., Grouws D.A. The effects of classroom mathematics teaching on students' learning. Second handbook of research on mathematics teaching and learning 2007, 374-404. Information Age, Charlotte, NC. F.K. Lester (Ed.).
28. Hill H.C., Charalambous C.Y., Kraft M.A. When rater reliability is not enough: teacher observation systems and a case for the generalizability study. Educational Researcher 2012, 41:56-64. 10.3102/0013189X12437203.
29. Hoyt W.T., Melby J.N. Dependability of measurement in counseling psychology: an introduction to generalizability theory. Counseling Psychologist 1999, 27:325-352.
30. Janík T., Miková M., Najvar P., Najvarová V. Unterrichtsformen und -phasen im tschechischen Physikunterricht: design und Ergebnisse der CPV Videostudie Physik. Zeitschrift für Didaktik der Naturwissenschaften 2006, 12:219-238.
31. The power of video studies in investigating teaching and learning in the classroom 2009, Waxmann, Münster, Germany. T. Janík, T. Seidel, P. Najvar (Eds.).
32. Kane T.J., McCaffrey D.F., Miller T., Staiger D.O. Have we identified effective teachers? Validating measures of effective teaching using random assignment 2013, (Research paper prepared for the Bill and Melinda Gates Foundation). Retrieved from MET project website. http://metproject.org/downloads/MET_Validating_Using_Random_Assignment_Research_Paper.pdf.
33. Kane T.J., Staiger D.O. Gathering feedback for teachers: Combining high-quality observations with student surveys and achievement gains 2012, (Research paper prepared for the Bill and Melinda Gates Foundation). Retrieved from MET project website. http://metproject.org/downloads/MET_Gathering_Feedback_Research_Paper.pdf.
34. Kennedy M.M. Attribution error and the quest for teacher quality. Educational Researcher 2010, 39:591-598. 10.3102/0013189X10390804.
35. Klieme E. Empirische Unterrichtsforschung: Aktuelle Entwicklungen, theoretische Grundlagen und fachspezifische Befunde. Zeitschrift für Pädagogik 2006, 52:765-773.
36. Klieme E., Pauli C., Reusser K. The Pythagoras study. The power of video studies in investigating teaching and learning in the classroom 2009, 137-160. Waxmann, Münster, Germany. T. Janik, T. Seidel (Eds.).
37. Klieme E., Schümer G., Knoll S. Mathematikunterricht in der Sekundarstufe I: Aufgabenkultur und Unterrichtsgestaltung. TIMSS - Impulse für Schule und Unterricht. Forschungsbefunde, Reforminitiativen, Praxisberichte und Video-Dokumente 2001, 43-57. Medienhaus Biering, Munich, Germany. Bundesministerium für Bildung und Forschung (BMBF) (Ed.).
38. Kobarg M., Dalehefte I.M., Menk M. Der Einsatz systematischer Videoanalysen zur Untersuchung der Wirksamkeit des Unterrichtsentwicklungsprogramms SINUS an Grundschulen. Maßnahmen zur Lehrerprofessionalisierung wissenschaftlich begleiten - verschiedene Strategien nutzen 2012, Waxmann, Münster, Germany. M. Kobarg, C. Fischer, I.M. Dalehefte, F. Trepke, M. Menk (Eds.).
39. Kounin J.S. Discipline and group management in classrooms 1970, Holt, Rinehart & Winston, New York.
40. Kunter M. Multiple Ziele im Mathematikunterricht 2005, Waxmann, Münster, Germany.
41. Kunter M., Baumert J. Who is the expert? Construct and criteria validity of student and teacher ratings of instruction. Learning Environments Research 2006, 9:231-251. 10.1007/s10984-006-9015-7.
42. Kunter M., Baumert J., Köller O. Effective classroom management and the development of subject-related interest. Learning and Instruction 2007, 17:494-509. 10.1016/j.learninstruc.2007.09.002.
43. Kunter M., Klusmann U., Baumert J., Richter D., Voss T., Hachfeld A. Professional competence of teachers: effects on quality and student development. Journal of Educational Psychology 2013, 105:805-820. 10.1037/a0032583.
44. Lakes K.D., Hoyt W.T. What sources contribute to variance in observer ratings? Using generalizability theory to assess construct validity of psychological measures. Infant and Child Development 2008, 17:269-284. 10.1002/icd.551.
45. Lipowsky F., Rakoczy K., Pauli C., Drollinger-Vetter B., Klieme E., Reusser K. Quality of geometry instruction and its short-term impact on students' understanding of the Pythagorean theorem. Learning and Instruction 2009, 19:527-537. 10.1016/j.learninstruc.2008.11.001.
46. Dokumentation der Erhebungsinstrumente des Projekts "Persönlichkeits- und Lernentwicklung von Grundschulkindern" (PERLE) - Teil 3. Technischer Bericht zu den PERLE-Videoanalysen. Materialien zur Bildungsforschung 2013, Band 23/3. GFPF, Frankfurt am Main, Germany. M. Lotz, F. Lipowsky, G. Faust (Eds.).
47. Lüdtke O., Robitzsch A., Trautwein U., Kunter M. Assessing the impact of learning environments: how to use student ratings of classroom or school characteristics in multilevel modeling. Contemporary Educational Psychology 2009, 34:120-131. 10.1016/j.cedpsych.2008.12.001.
48. Malmberg L.-E., Hagger H., Burn K., Mutton T., Colls H. Observed classroom quality during teacher education and two years of professional practice. Journal of Educational Psychology 2010, 102:916-932. 10.1037/a0020920.
49. Mayer D.P. Measuring instructional practice: can policymakers trust survey data?. Educational Evaluation and Policy Analysis 1999, 21:29-45. 10.3102/01623737021001029.
50. Mayer R.E. Should there be a three-strikes rule against pure discovery learning? The case for guided methods of instruction. American Psychologist 2004, 59:14-19. 10.1037/0003-066X.59.1.14.
51. Medley D.M., Mitzel H. Measuring classroom behavior by systematic observation. Handbook of research on teaching 1963, 247-328. Rand-McNally, Chicago. N.L. Gage (Ed.).
52. Meyer L., Seidel T., Prenzel M. Wenn Lernsituationen zu Leistungssituationen werden: Untersuchung zur Fehlerkultur in einer Videostudie. Schweizerische Zeitschrift für Bildungswissenschaften 2006, 28:21-41.
53. Neumann K., Fischer H.E., Labudde P., Viiri J. Postersymposium Physikunterricht im Vergleich: Unterrichtsqualität in Deutschland, Finnland und Schweiz. Chemie- und Physikdidaktik für die Lehramtsausbildung 2009, 357-359. LIT, Berlin, Germany. D. Höttecke (Ed.).
54. Newton X.A. Developing indicators of classroom practice to evaluate the impact of district mathematics reform initiative: a generalizability analysis. Studies in Educational Evaluation 2010, 36:1-13. 10.1016/j.stueduc.2010.10.002.
55. Pauli C., Reusser K. Expertise in Swiss mathematics instruction. Expertise in mathematics instruction: An international perspective 2011, 85-107. Springer, New York, NY. Y. Li, G. Kaiser (Eds.).
56. Pauli C., Reusser K., Grob U. Teaching for understanding and/or self-regulated learning? A video-based analysis of reform-oriented mathematics instruction in Switzerland. International Journal of Educational Research 2007, 46:294-305. 10.1016/j.ijer.2007.10.004.
57. Petko D., Waldis M., Pauli C., Reusser K. Methodologische Überlegungen zur videogestützten Forschung in der Mathematikdidaktik: Ansätze der TIMSS 1999 Video Studie und ihrer schweizerischen Erweiterung. Zentralblatt für Didaktik der Mathematik 2003, 35:265-280.
58. Pianta R.C., Hamre B.K. Conceptualization, measurement, and improvement of classroom processes: standardized observation can leverage capacity. Educational Researcher 2009, 38:109-119. 10.3102/0013189X09332374.
59. Pietsch M. Evaluation von Unterrichtsstandards. Zeitschrift für Erziehungswissenschaft 2010, 13:121-148. 10.1007/s11618-010-0113-z.
60. Porter A.C. Measuring the content of instruction: Uses in research and practice. Educational Researcher 2002, 31:3-14.
61. Praetorius A.-K., Lenske G., Helmke A. Observer ratings of instructional quality: do they fulfill what they promise?. Learning and Instruction 2012, 22:387-400. 10.1016/j.learninstruc.2012.03.002.
62. Rakoczy K. Motivationsunterstützung im Mathematikunterricht: Unterricht aus der Perspektive von Lernenden und Beobachtern 2008, Waxmann, Münster, Germany.
63. Rakoczy K., Pauli C. Hoch inferentes Rating: Beurteilung der Qualität unterrichtlicher Prozesse. Videoanalysen. Dokumentation der Erhebungs- und Auswertungsinstrumente zur schweizerisch-deutschen Videostudie "Unterrichtsqualität, Lernverhalten und mathematisches Verständnis". Materialien zur Bildungsforschung 2006, Vol. 15:206-233. GFPF, Frankfurt am Main, Germany.
64. Renkl A., Helmke A. Prinzip, Nutzen und Grenzen der Generalisierungstheorie. Empirische Pädagogik 1993, 7:63-85.
65. Reusser K. Konstruktivismus: Vom epistemologischen Leitbegriff zur Erneuerung der didaktischen Kultur. Didaktik auf psychologischer Grundlage. Von Hans Aeblis kognitionspsychologischer Didaktik zur modernen Lehr- und Lernforschung 2006, 151-168. h.e.p. verlag, Bern. M. Baer, M. Fuchs, P. Füglister, K. Reusser, H. Wyss (Eds.).
66. Reyes M.R., Brackett M.A., Rivers S.E., White M., Salovey P. Classroom emotional climate, student engagement, and academic achievement. Journal of Educational Psychology 2012, 104:700-712. 10.1037/a0027268.
67. Rosenshine B., Furst N. Research on teacher performance criteria. Research in teacher education: A symposium 1973, Prentice Hall, Englewood Cliffs, NJ. B.O. Smith (Ed.).
68. Roth K.J., Druker S.L., Garnier H., Lemmens M., Chen C., Kawanaka T., et al. Teaching science in five countries: Results from the TIMSS 1999 video study 2006, National Center for Education Statistics, Washington, DC.
69. Seidel T., Prenzel M. Stability of teaching patterns in physics instruction: findings from a video study. Learning and Instruction 2006, 16:228-240. 10.1016/j.learninstruc.2006.03.002.
70. Seidel T., Prenzel M., Duit R., Euler M., Geiser H., Hoffmann L., et al. "Jetzt bitte alle nach vorne schauen!" Lehr-Lernskripts im Physikunterricht und damit verbundene Bedingungen für individuelle Lernprozesse. Unterrichtswissenschaft 2002, 30:52-77.
71. Seidel T., Prenzel M., Rimmele R., Dalehefte I.M., Herweg C., Kobarg M., et al. Blicke auf den Physikunterricht: Ergebnisse der IPN Videostudie. Zeitschrift für Pädagogik 2006, 52:799-821.
72. Seidel T., Prenzel M., Schwindt K., Rimmele R., Kobarg M., Dalehefte I.M. The link between teaching and learning - investigating effects of physics teaching on student learning in the context of the IPN video study. The power of video studies in investigating teaching and learning in the classroom 2009, 161-180. Waxmann, Münster, Germany. T. Janík, T. Seidel, P. Najvar (Eds.).
73. Seidel T., Shavelson R. Teaching effectiveness research in the past decade: the role of theory and research design in disentangling meta-analysis results. Review of Educational Research 2007, 77:454-499. 10.3102/003465430731031.
74. Shavelson R., Dempsey-Atwood N. Generalizability of measures of teaching behavior. Review of Educational Research 1976, 46:553-611. 10.3102/00346543046004553.
75. Shavelson R., Webb N. Generalizability theory: A primer 1991, Sage, Thousand Oaks, CA.
76. Shavelson R.J., Webb N.M., Burstein L. Measurement of teaching. Handbook of research on teaching 1986, 50-91. Macmillan, New York, NY. M.C. Wittrock (Ed.).
77. Shumate S., Surles J., Johnson R., Penny J. The effect of the number of scale and non-normality on the generalizability coefficient: a Monte Carlo study. Applied Measurement in Education 2007, 20:1-20. 10.1080/08957340701429645.
78. Smith P.L. Sampling errors of variance components in small sample generalizability studies. Journal of Educational Statistics 1978, 3:319-346.
79. Staub F.C. Mathematics classroom cultures: methodological and theoretical issues. International Journal of Educational Research 2007, 46:319-326. 10.1016/j.ijer.2007.10.007.
80. Stigler J.W., Gallimore R., Hiebert J. Using video surveys to compare classrooms and teaching across cultures: examples and lessons from the TIMSS video studies. Educational Psychologist 2000, 35:87-100. 10.1207/S15326985EP3502_3.
81. Stigler J., Gonzales P., Kawanaka T., Knoll S., Serrano A. The TIMSS-videotape classroom study 1999, (Technical report). Los Angeles, CA.
82. Tschannen-Moran M., Woolfolk Hoy A. Teacher efficacy: capturing an elusive construct. Teaching and Teacher Education 2001, 17:783-805. 10.1016/S0742-051X(01)00036-1.
83. Vieluf S., Klieme E. Cross-nationally comparative results on teachers' qualification, beliefs, and practices. Expertise in mathematics instruction 2011, 295-325. Springer, New York, NY. Y. Li, G. Kaiser (Eds.).
84. Waldis M., Grob U., Pauli C., Reusser K. Der schweizerische Mathematikunterricht aus der Sicht von Schülerinnen und Schülern und in der Perspektive hochinferenter Beobachterurteile. Unterrichtsgestaltung und Unterrichtsqualität. Ergebnisse einer internationalen und schweizerischen Videostudie zum Mathematikunterricht 2010, 171-208. Waxmann, Münster, Germany. K. Reusser, C. Pauli, M. Waldis (Eds.).
85. Williams E.J. The comparison of regression variables. Journal of the Royal Statistical Society, Series B 1959, 21:396-399.