Once is not enough: Establishing reliability criteria for feedback and evaluation decisions based on classroom observations

Studies in Educational Evaluation

Volumn 50, Issue , 2016, Pages 88-95

  van der Lans, Rikkert M ^a   van de Grift, Wim J C M ^a   van Veen, Klaas ^a   Fokkens Bruinsma, Marjon ^a  

^a UNIVERSITY OF GRONINGEN   (Netherlands)

Author keywords

Classroom observation; Generalizability Theory; Item Response Theory; Reliability; Teacher evaluation; Teacher feedback

Indexed keywords

EID: 84989873997     PISSN: 0191491X     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.stueduc.2016.08.001     Document Type: Article

References (34)

1. Bates, D., Maechler, M., Bolker, B., Walker, S., Lme4: Linear mixed-effects models using S4 classes. R Package Version 1, 2014, 1–7 http://CRAN.R-project.org/package=lme4.
2. Brennan, R.L., Generalizability theory: statistics for social science and public policy. 2001, Springer-Verlag, New York.
3. Briggs, D.C., Wislon, M., Generalizability in item response theory. Journal of Educational Measurement 44 (2007), 131–155.
4. Centra, J.A., Colleagues as raters of classroom instruction. The Journal of Higher Education 46:3 (1975), 327–337, 10.2307/1980806.
5. Choi, J., Advances in combining generalizability theory and item response theory. Doctoral dissertation, 2012, University of California, Berkeley.
6. Darling-Hammond, L., Amrein-Beardsley, A., Heartel, E., Rothstein, J., Evaluation teacher evaluation. Phi Delta Kappan 93 (2012), 8–15.
7. De Boeck, P., Bakker, M., Zwitser, R., Nivard, M., Abe, H., Tuerlinckx, F., et al. The estimation of item response models with the lmer function from the lme4 package in R. Journal of Statistical Software 39 (2011), 1–25.
8. DfEE (Department for Education), (2012). Teacher appraisal and capability. A model policy for schools. Retrieved June 30, 2015, from: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/282598/Teacher_appraisal_and_capability.pdf.
9. Firestone, W.A., Teacher evaluation policy and conflicting theories of motivation. Educational Researcher 43 (2014), 100–107.
10. French-Lazovik, G., Documentary evidence in the evaluation of teaching. Millman, J., (eds.) Handbook of teacher evaluation, 1981, Sage Publications, Beverly Hills, CA.
11. Hill, H., Charalambous, C.Y., Kraft, M.A., When interrater-reliability is not enough: Teacher observation systems and a case for the generalizability theory. Educational Researcher 41 (2012), 56–64, 10.3102/0013189x12437203.
12. Ho, A.D., Kane, T.J., The reliability of classroom observations by school personnel. 2013, Bill & Melinda Gates Foundation, Seattle, WA.
13. Kane, T.J., Staiger, D.O., McCaffrey, D., Cantrell, S., Archer, J., Buhayar, S., et al. Gathering feedback for teaching: combining high-quality observations with student surveys and achievement gains. 2012, Bill & Melinda Gates Foundation, Seattle, WA.
14. Kane, M.T., Validation. Brennan, R.L., (eds.) Educational measurement, 4th ed., 2006, The National Council on Measurement in Education & the American Council on Education, Washington, DC, 17–64.
15. Marzano, R.J., The two purposes of teacher evaluation. Educational Leadership 70 (2012), 14–19.
16. Mourshed, M., Chijioke, C., Barber, M., How the world's most improved school systems keep getting better. 2010, McKinsey Company, London.
17. Muijs, D., Measuring teacher effectiveness: Some methodological reflections. Educational Research and Evaluation: An International Journal on Theory and Practice 12 (2006), 53–74.
18. Nunnally, J.C., Psychometric theory. 1978, McGraw-Hill, New York.
19. OCW [Dutch Ministry of Education, Culture, Science], Peer review in de praktijk [Peer review in practice]. 2013, VOION, Rotterdam.
20. Peterson, K.D., Chenoweth, T., School teachers’ control and involvement in their own evaluation. Journal of Personnel Evaluation in Education 6:2 (1992), 177–189.
21. Peterson, K.D., Teacher evaluation: a comprehensive guide to new directions and practice. 2000, Corwin press, Thousand Oaks, CA.
22. Popham, The dysfunctional marriage of formative and summative teacher evaluation. Journal of Personnel Evaluation in Education 1 (1988), 269–273.
23. Raju, N.S., Price, L.R., Oshima, T.C., Nering, M.L., Standardized conditional SEM: a case for conditional reliability. Applied Psychological Measurement 30 (2006), 1–12, 10.1177/0146621606291569.
24. Rasch, G., Probabilistic models for some intelligence and attainment tests. 1960, Nielsen & Lydiche, Copenhagen.
25. Scriven, M., Summative teacher evaluation. Millman, J., (eds.) Handbook of teacher evaluation, 1981, Sage Publications, Beverly Hills, CA.
26. Shavelson, R., Dempsey-Atwood, N., Generalizability of teaching behavior. Review of Educational Research 46 (1976), 553–611.
27. State of the States, Connect the dots: Using evaluations of teaching effectiveness to inform policy and practice. 2013, National Council on Teacher Quality, Washington, DC.
28. Strong, M., The highly qualified teacher: what is teacher quality and how do we measure it?. 2011, Teachers College Press, NY, New York.
29. Toch, T., Rothman, R., Rush to judgment: teacher evaluation in public schools. 2008, Education Sector, Washington DC.
30. van de Grift, W.J.C.M., Measuring teaching quality in several European countries. School effectiveness and school improvement, 25(3), 2013, 10.1080/09243453.2013.794845.
31. van de Grift, W.J.C.M., Helms-Lorenz, M., Maulana, R., Teaching skills of student teachers: Calibration of an evaluation instrument and its value in predicting student academic engagement. Studies in Educational Evaluation 43 (2014), 150–159, 10.1016/j.stueduc.2014.09.003.
32. van der Lans, van de Grift, & van Veen (2016). Developing an instrument for teacher feedback: Using the Rasch model to explore teachers’ development of effective teaching strategies and behaviors. Manuscript submitted for publication.
33. Weisberg, D., Sexton, S., Mulhern, J., Keeling, D., Schunck, J., Palcisco, A., et al. The widget effect: Our national failure to acknowledge and act on differences in teacher effectiveness. New Teacher Project, 2009.
34. Winters, M.A., Cowen, J.M., Who would stay, who would be dismissed? An empirical consideration of value-added teacher retention policies. Educational Researcher 42 (2014), 330–337, 10.3102/0013189x13496145.