Qualities of classroom observation systems

School Effectiveness and School Improvement

Volumn 30, Issue 1, 2019, Pages 3-29

  Bell, Courtney A ^a   Dobbelaer, Marjoleine J ^b   Klette, Kirsti ^c   Visscher, Adrie ^b  

^a EDUCATIONAL TESTING SERVICE   (United States)

^b UNIVERSITY OF TWENTE   (Netherlands)

^c UNIVERSITY OF OSLO   (Norway)

Author keywords

classroom observation; observation systems; Teacher evaluation; teaching evaluation; teaching quality

Indexed keywords

EID: 85057311086     PISSN: 09243453     EISSN: 17445124     Source Type: Journal    
DOI: 10.1080/09243453.2018.1539014     Document Type: Article

References (84)

1. Abadzi, H., (2009). Instructional time loss in developing countries: Concepts, measurement, and implications. The World Bank Research Observer, 24(2), 267–290. doi:10.1093/wbro/lkp008
2. Abry, T., Rimm-Kaufman, S. E., Larsen, R. A., & Brewer, A. J., (2013). The influence of fidelity of implementation on teacher–student interaction quality in the context of a randomized controlled trial of the Responsive Classroom approach. Journal of School Psychology, 51(4), 437–453. doi:10.1016/j.jsp.2013.03.001
3. Archer, J., Cantrell, S., Holtzman, S. L., Joe, J. N., Tocci, C. M., & Wood, J., (2017). Better feedback for better teaching: A practical guide to improving classroom observations. Retrieved from http://k12education.gatesfoundation.org/resource/better-feedback-for-better-teaching-a-practical-guide-to-improving-classroom-observations/
4. Baumert, J., Kunter, M., Blum, W., Brunner, M., Voss, T., Jordan, A., … Tsai, Y.-M., (2010). Teachers’ mathematical knowledge, cognitive activation in the classroom, and student progress. American Educational Research Journal, 47(1), 133–180. doi:10.3102/0002831209345157
5. Bell, C. A., Gitomer, D. H., McCaffrey, D. F., Hamre, B. K., Pianta, R. C., Qi, Y., (2012). An argument approach to observation protocol validity. Educational Assessment, 17(2–3), 62–87. doi:10.1080/10627197.2012.715014
6. Bill & Melinda Gates Foundation. (2018). Measures of effective teaching project: Frequently asked questions. Retrieved from http://k12education.gatesfoundation.org/blog/measures-of-effective-teaching-project-faqs/
7. Black, P., & Wiliam, D., (1998). Assessment and classroom learning. Assessment in Education: Principles, Policy & Practice, 5(1), 7–74. doi:10.1080/0969595980050102
8. Black, P., & Wiliam, D., (2010). Inside the black box: Raising standards through classroom assessment. Phi Delta Kappan, 92(1), 81–90. doi:10.1177/003172171009200119
9. Boekaerts, M., Pintrich, P. R., & Zeidner, M., (Eds.). (2000). Handbook of self-regulation. San Diego, CA: Academic Press.
10. Brophy, J. E., & Good, T. L., (1986). Teacher behavior and student achievement. In M. C., Wittrock (Ed.), The handbook of research on teaching (3rd ed., pp. 328–375). New York, NY: Macmillan.
11. Carnine, D. W., Dixon, R. C., & Silbert, J., (1998). Effective strategies for teaching mathematics. In E. J., Kame’enui & D. W., Carnine (Eds.), Effective teaching strategies that accommodate diverse learners (pp. 93–112). Columbus, OH: Merrill.
12. Charalambous, C. Y., & Praetorius, A.-K., (2018). Studying mathematics instruction through different lenses: Setting the ground for understanding instructional quality more comprehensively. ZDM Mathematics Education, 50(3), 355–366. doi:10.1007/s11858-018-0914-8
13. Clarke, D., Emanuelsson, J., Jablonka, E., & Mok, I. A. C., (Eds.). (2006). Making connections: Comparing mathematics classrooms around the world. Rotterdam: Sense.
14. Cohen, J., (2015). Challenges in identifying high leverage practices. Teachers College Record, 117(7), 1–41.
15. Cohen, J., & Grossman, P., (2016). Respecting complexity and measures of teaching: Keeping students and schools in focus. Teaching and Teacher Education, 55, 308–317. doi:10.1016/j.tate.2016.01.017
16. Dalland, C. P., Klette, K., & Svenkerud, S., (2018). Video studies and the challenge of selecting time scales. Manuscript submitted for publication.
17. Danielson, C., (2013). The Framework for Teaching Evaluation Instrument. Princeton, NJ: Danielson Group.
18. Decristan, J., Klieme E., Kunter M., Hochweber, J., Büttner, G., Fauth, B., … Hardy, I., (2015). Embedded formative assessment and classroom process quality: How do they interact in promoting science understanding? American Educational Research Journal, 52(6), 1133–1159. doi:10.3102/0002831215596412
19. Dobbelaer, M. J., & Visscher, A.J., (2018). The quality of classroom observation systems for measuring teaching quality in primary education–A systematic review. Manuscript submitted for publication.
20. Flanders, N. A., (1970). Analyzing teaching behavior. Boston, MA: Addison Wesley.
21. Givvin, K. B., Hiebert, J., Jacobs, J. K., Hollingsworth, H., & Gallimore, R., (2005). Are there national patterns of teaching? Evidence from the TIMSS 1999 video study. Comparative Education Review, 49(3), 311–343. doi:10.1086/430260
22. Grossman, P. L., (2018). The Protocol for Language Arts Teaching Observation (PLATO). Retrieved from http://platorubric.stanford.edu/index.html
23. Grossman, P., Cohen, J., & Brown, L., (2014). Understanding instructional quality in English Language Arts: Variations in PLATO scores by content and context. In T. J., Kane, K. A., Kerr, & R. C., Pianta (Eds.), Designing teacher evaluation systems: New guidance from the Measures of Effective Teaching Project (pp. 303–331). San Francisco, CA: Jossey-Bass.
24. Grossman, P., Cohen, J., Ronfeldt, M., & Brown, L., (2014). The test matters: The relationship between classroom observation scores and teacher value added on multiple types of assessment. Educational Researcher, 43(6), 293–303. doi:10.3102/0013189X14544542
25. Grossman, P., Greenberg, S., Hammerness, K., Cohen, J. Alston, C., & Brown, M., (2009, April). Development of the Protocol for Language Arts Teaching Observation (PLATO). Paper presented at the Annual Meeting of the American Educational Research Association, San Diego, CA.
26. Grossman, P., Loeb, S., Cohen, J., & Wyckoff, J., (2013). Measure for measure: The relationship between measures of instructional practice in middle school English language arts and teachers’ value-added scores. American Journal of Education 119(3), 445–470. doi:10.1086/669901
27. Grossman, P., & McDonald, M., (2008). Back to the future: Directions for research in teaching and teacher education. American Educational Research Journal, 45(1), 184–205. doi:10.3102/0002831207312906
28. Henry, A. E., (2010). Advantages to and challenges of using ratings of observed teacher-child interactions (Unpublished doctoral dissertation). University of Virginia, Charlottesville, VA.
29. Hiebert, J., & Grouws, D. A., (2007). The effects of classroom mathematics teaching on students’ learning. In F. K., Lester (Ed.), Second handbook of research on mathematics teaching and learning (pp. 371–404). Charlotte, NC: Information Age.
30. Hill, H. C., (2018). Mathematical Quality of Instruction (MQI) domains. Retrieved from https://cepr.harvard.edu/mqi-domains
31. Hill, H. C., Blunk, M. L., Charalambous, C. Y., Lewis, J. M., Phelps, G. C., Sleep, L., & Ball, D. L., (2008). Mathematical knowledge for teaching and the mathematical quality of instruction: An exploratory study. Cognition and Instruction, 26(4), 430–511. doi:10.1080/07370000802177235
32. Hill, H. C., Charalambous, C. Y., & Kraft, M. A., (2012). When rater reliability is not enough: Teacher observation systems and a case for the generalizability study. Educational Researcher, 41(2), 56–64. doi:10.3102/0013189X12437203
33. Hill, H. C., & Grossman, P., (2013). Learning from teacher observations: Challenges and opportunities posed by new teacher evaluation systems. Harvard Educational Review, 83(2), 371–384.
34. Hugener, I., Pauli, C., Reusser, K., Lipowsky, F., Rakoczy, K., & Klieme, E., (2009). Teaching patterns and learning quality in Swiss and German mathematics lessons. Learning and Instruction, 19(1), 66–78. doi:10.1016/j.learninstruc.2008.02.001
35. International Association for the Evaluation of Educational Achievement. (2018). The TIMSS Video study. Retrieved from http://www.timssvideo.com/the-study/
36. Jacobs, J., Garnier, H., Gallimore, R., Hollingsworth, H., Givvin, K. B., Rust, K., … Stigler, J. W., (2003). Third International Mathematics and Science Study 1999 Video Study Technical Report: Volume 1: Mathematics (NCES 2003012). Washington, DC: National Center for Education Statistics.
37. Joe, J. N., McClellan, C. A., & Holtzman, S. L., (2014). Scoring design decisions: Reliability and the length and focus of classroom observations. In T. J., Kane, K. A., Kerr, & R. C., Pianta (Eds.), Designing teacher evaluation systems: New guidance from the Measures of Effective Teaching Project (pp. 415–443). San Francisco, CA: Jossey-Bass.
38. Kane, M., (2006). Validation. In R. L., Brennan (Ed.), Educational measurement (4th ed., pp. 17–64). Westport, CT: American Council on Education and Praeger.
39. Kane, T. J., Kerr, K. A., & Pianta, R. C., (Eds.). (2014). Designing teacher evaluation systems: New guidance from the Measures of Effective Teaching Project. San Francisco, CA: Jossey-Bass.
40. Kane, T. J., & Staiger, D. O., (2012). Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains. Retrieved from https://files.eric.ed.gov/fulltext/ED540962.pdf
41. Keuning, T., Van Geel, M., Frèrejean, J., Van Merriënboer, J., Dolmans, D., & Visscher, A. J., (2017). Differentiëren bij rekenen: Een cognitieve taakanalyse van het denken en handelen van basisschoolleerkrachten [Differentiated instruction for mathematics: A cognitive task analysis of primary school teachers’ reasoning and acting], Pedagogische Studiën, 94(3), 160–181.
42. Klette, K., & Blikstad-Balas, M., (2018). Observation manuals as lenses to classroom teaching: Pitfalls and possibilities. European Educational Research Journal, 17(1), 129–146. doi:10.1177/1474904117703228
43. Klette, K., Blikstad-Balas, M., & Roe, A., (2017). Linking instruction and student achievement: Research design for a new generation of classroom studies. Acta Didactica Norge, 11(3): Art. 10. doi:10.5617/adno.4729
44. Kloser, M., (2014). Identifying a core set of science teaching practices: A Delphi expert panel approach. Journal of Research in Science Teaching, 51(9), 1185–1217. doi:10.1002/tea.21171
45. Kor, K., (2011, April). The measurement properties of the PLATO rubric. Paper presented at the Annual Meeting of the American Educational Research Association, New Orleans, LA.
46. Kraft, M. A, & Blazar, D., (2017). Individualized coaching to improve teacher practice across grades and subjects: New experimental evidence. Educational Policy, 31(7), 1033–1068. doi:10.1177/0895904816631099
47. Kunter, M., & Baumert, J., (2006). Linking TIMSS to research on learning and instruction: A re-analysis of the German TIMSS and TIMSS video data. In S. J., Howie & T., Plomp (Eds.), Contexts of learning mathematics and science: Lessons learned from TIMSS (pp. 335–351). London: Routledge.
48. Leung, F. K. S., (2005). Some characteristics of East Asian mathematics classrooms based on data from the TIMSS 1999 video study. Educational Studies in Mathematics, 60(2), 199–215. doi:10.1007/s10649-005-3835-8
49. Littell, J. H., Corcoran, J., & Pillai, V., (2008). Systematic reviews and meta-analysis. New York, NY: Oxford University Press.
50. Liu, S., Bell, C. A., & Jones, N. D., (2017, March). The validity of classroom observation systems in research and applied contexts. Paper presented at the annual spring meeting of the Society for Research on Educational Effectiveness (SREE), Washington, DC.
51. Marzano, R. J., Marzano, J. S., & Pickering, D. J., (2003). Classroom management that works: Research-based strategies for every teacher. Alexandria, VA: ASCD.
52. Maulana, R., Helms-Lorenz, M., & Van de Grift, W., (2017). Validating a model of effective teaching behaviour of pre-service teachers. Teachers and Teaching: Theory and Practice, 23(4), 471–493. doi:10.1080/13540602.2016.1211102
53. Muijs, D., Kyriakides, L., Van der Werf, G., Creemers, B., Timperley, H., & Earl, L., (2014). State of the art–Teacher effectiveness and professional learning. School Effectiveness and School Improvement, 25(2), 231–256. doi:10.1080/09243453.2014.885451
54. Osborne, J., Borko, H., Busch, K. C., Fishman, E., Millon, S., & Tseng, A., (2015, August). Assessing the quality of classroom discourse in science classrooms. Paper presented at the biennial conference of the European Association for Research on Learning and Teaching, Limmasol, Cyprus.
55. Oser, F. K., & Baeriswyl, F. J., (2001). Choreographies of teaching: Bridging instruction to learning. In V., Richardson (Ed.), Handbook on research on teaching (4th ed., pp. 1031–1065). Washington, DC: American Educational Research Association.
56. Park, Y. S., Chen, J., & Holtzman, S. L., (2014). Evaluating efforts to minimize rater bias in scoring classroom observations. In T. J., Kane, K. A., Kerr, & R. C., Pianta (Eds.), Designing teacher evaluation systems: New guidance from the Measures of Effective Teaching Project (pp. 383–414). San Francisco, CA: Jossey-Bass.
57. Pianta, R. C., Hamre, B. K., & Mintz, S., (2012). Classroom Assessment Scoring System (CLASS) manual upper elementary. Baltimore, MD: Paul H. Brookes.
58. Pianta, R. C., La Paro, K. M., & Hamre, B. K., (2008). Classroom Assessment Scoring System: Manual K-3. Baltimore, MD: Paul H. Brookes.
59. Praetorius, A.-K., & Charalambous, C. Y., (2018). Classroom observation frameworks for studying instructional quality: Looking back and looking forward. ZDM Mathematics Education, 50(3), 535–553. doi:10.1007/s11858-018-0946-0
60. Praetorius, A.-K., Klieme, E., Herbert, B., & Pinger, P, (2018). Generic dimensions of teaching quality: The German framework of Three Basic Dimensions. ZDM Mathematics Education, 50(3), 407–426. doi:10.1007/s11858-018-0918-4
61. Praetorius, A.-K., Pauli, C., Reusser, K., Rakoczy, K., & Klieme, E., (2014). One lesson is all you need? Stability of instructional quality across lessons. Learning and Instruction, 31, 2–12. 10.1016/j.learninstruc.2013.12.002
62. Rosenshine, B., (1980). How time is spent in elementary classrooms. In C., Denham & A., Lieberman (Eds.), Time to learn (pp. 107–126). Washington, DC: National Institute of Education.
63. Saginor, N., (2008). Diagnostic classroom observation: Moving beyond best practice. Thousand Oaks, CA: Corwin Press.
64. Sandilos, L. E., Shervey, S. W., DiPerna, J. C., Lei, P., & Cheng, W., (2016). Structural validity of CLASS K-3 in primary grades: Testing alternative models. School Psychology Quarterly, 32(2), 226–239. doi:10.1037/spq0000155
65. Sawada, D., Piburn, M. D., Judson, E., Turley, J., Falconer, K., Benford, R., & Bloom, I., (2002). Measuring reform practices in science and mathematics classrooms: The Reformed Teaching Observation Protocol. School Science and Mathematics, 102(6), 245–253. doi:10.1111/j.1949-8594.2002.tb17883.x
66. Schacter, J., & Thum, Y. M., (2004). Paying for high-and low-quality teaching. Economics of Education Review, 23(4), 411–430. doi:10.1016/j.econedurev.2003.08.002
67. Scheerens, J., (2014). School, teaching, and system effectiveness: Some comments on three state-of-the-art reviews. School Effectiveness and School Improvement, 25(2), 282–290. doi:10.1080/09243453.2014.885453
68. Seidel, T., Prenzel, M., & Kobarg, M., (Eds.). (2005). How to run a video study: Technical report of the IPN video study. Münster: Waxmann.
69. Seidel, T., & Shavelson, R. J., (2007). Teaching effectiveness research in the past decade: The role of theory and research design in disentangling meta-analysis results. Review of Educational Research, 77(4), 454–499. doi:10.3102/0034654307310317
70. Slavin, R. E., (1996). Education for all. Lisse: Swets & Zeitlinger.
71. Stallings, J. A., (1973). Follow through program classroom observation evaluation 1971–72. (Report No. SRI-URU-7370). Menlo Park, CA: Stanford Research Institute.
72. Stigler, J. W., Gallimore, R., & Hiebert, J., (2000). Using video surveys to compare classrooms and teaching across cultures: Examples and lessons from the TIMSS video studies. Educational Psychologist, 35(2), 87–100. doi:10.1207/S15326985EP3502_3
73. Teachstone. (2015). Why class? Exploring the promise of the Classroom Assessment Scoring System (CLASS). Retrieved from http://cdn2.hubspot.net/hubfs/336169/What_Is_CLASS_ebook_Final.pdf?t=1446
74. TIMSS Video Mathematics Research Group. (2003). Understanding and improving mathematics teaching: Highlights from the TIMSS 1999 Video Study. Phi Delta Kappan, 84(10), 768–775. doi:10.1177/003172170308401011
75. Tomlinson, C. A., (2004). The Möbius effect: Addressing learner variance in schools. Journal of Learning Disabilities, 37(6), 516–524. doi:10.1177/00222194040370060601
76. Tomlinson, C. A., Brimijoin, K., & Narvaez, L., (2008). The differentiated school: Making revolutionary changes in teaching and learning. Alexandria, VA: ASCD.
77. Van de Grift, W., (2007). Quality of teaching in four European countries: A review of the literature and application of an assessment instrument. Educational Research, 49(2), 127–152. doi:10.1080/00131880701369651
78. Van de Grift, W., Van der Wal, M., & Torenbeek, M., (2011). Ontwikkeling in de pedagogisch didactische vaardigheid van leraren in het basisonderwijs [The development of primary school teachers’ pedagogical and didactical skill]. Pedagogische Studiën, 88(6), 416–432.
79. Van der Lans, R. M., Van de Grift, W. J. C. M., & Van Veen, K., (2018). Developing an instrument for teacher feedback: Using the Rasch model to explore teachers’ development of effective teaching strategies and behaviors. The Journal of Experimental Education, 86(2), 247–264. doi:10.1080/00220973.2016.1268086
80. Van der Lans, R. M., Van de Grift, W. J. C. M., Van Veen, K., & Fokkens-Bruinsma, M., (2016). Once is not enough: Establishing reliability criteria for feedback and evaluation decisions based on classroom observations. Studies in Educational Evaluation, 50, 88–95. doi:10.1016/j.stueduc.2016.08.001
81. Van Veen, K., Zwart, R., & Meirink, J., (2012). What makes teacher professional development effective? A literature review. In M., Kooy & K., van Veen (Eds.), Teacher learning that matters: International perspectives (pp. 3–21). Abingdon: Routledge.
82. Wang, M. C., Haertel, G. D., & Walberg, H. J., (1993). Toward a knowledge base for school learning. Review of Educational Research, 63(3), 249–294. doi:10.3102/00346543063003249
83. World Bank. (2010). Inside Indonesia’s mathematics classrooms: A TIMSS video study of teaching practices and student achievement (Report No. 54936–ID). Jakarta: Author.
84. Zimmerman, B. J., (1990). Self-regulated learning and academic achievement: An overview. Educational Psychologist, 25(1), 3–17. doi:10.1207/s15326985ep2501_2