Observer ratings of instructional quality: Do they fulfill what they promise?

Learning and Instruction

Volumn 22, Issue 6, 2012, Pages 387-400

  Praetorius, Anna Katharina ^a   Lenske, Gerlinde ^a   Helmke, Andreas ^b  

^a University of Koblenz Landau   (Germany)

^b University of Koblenz   (Germany)

Author keywords

Generalizability theory; Instructional quality; Observer ratings; Reliability; Validity

Indexed keywords

EID: 84865576473     PISSN: 09594752     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.learninstruc.2012.03.002     Document Type: Article

References (75)

1. Ambady N., Rosenthal R. Thin slices of expressive behavior as predictors of interpersonal consequences: a meta-analysis. Psychological Bulletin 1992, 111:256-274. 10.1037/0033-2909.111.2.256.
2. Barnes L.L.B., Barnes M.W. Academic discipline and generalizability of student evaluations of instruction. Research in Higher Education 1993, 34(2):135-149. 10.1007/BF00992160.
3. Borich G.D. Effective teaching methods: Research-based practice 2007, Pearson, Upper Saddle River. 6th ed.
4. Brennan R.L. Generalizability theory 2001, Springer, New York, NY.
5. Brophy J.E. Teaching 2000, International Academy of Education/International Bureau of Education (IAE), Brussels, Belgium.
6. Centra J.A. Colleagues as raters of classroom instruction. The Journal of Higher Education 1975, 46:327-337.
7. Clare L., Valdés R., Pascal J., Steinberg J. Teachers' assignments as indicators of instructional quality in elementary schools 2001, (CSE technical report no. 545), National Center for Research on Evaluation, Standards, and Student Testing, Los Angeles, Retrieved from. http://www.cse.ucla.edu/products/Reports/TR545.pdf.
8. Clausen M. Qualität von Unterricht - Eine Frage der Perspektive? 2002, Waxmann, Münster, Germany.
9. Clausen M., Reusser K., Klieme E. Unterrichtsqualität auf der Basis hoch-inferenter Unterrichtsbeurteilungen: Ein Vergleich zwischen Deutschland und der deutschsprachigen Schweiz. Unterrichtswissenschaft 2003, 31:122-141.
10. Crick J.E., Brennan R.L. Manual for GENOVA: A generalized analysis of variance system 1983, (ACT Technical Bulletin No. 43), ACT, Inc, Iowa City, IA.
11. Cronbach L.J., Gleser G.C., Nanda H., Rajaratnam N. The dependability of behavioral measurements: Theory of generalizability for scores and profiles 1972, Wiley, New York.
12. Crooks T.J., Kane M.T. The generalizability of student ratings of instructors: item specificity and section effects. Research in Higher Education 1981, 15:305-313. 10.1007/BF00973511.
13. DeMaio T.J., Rothgeb J.M. Cognitive interviewing techniques in the lab and in the field. Answering questions. Methodology for determining cognitive and communicative processes in survey research 1996, 177-195. Jossey-Bass, San Francisco. N. Schwarz, S. Sudman (Eds.).
14. Desimone L.M., Le Floch K.C. Are we asking the right questions? Using cognitive interviews to improve surveys in education research. Educational Evaluation and Policy Analysis 2004, 26:1-22. 10.3102/01623737026001001.
15. Dweck C.S. Self-theories: Their role in motivation, personality, and development 1999, Psychology Press, Philadelphia, PA.
16. Farkas R.D. Effects of traditional versus learning-styles instructional methods on middle school students. The Journal of Educational Research 2003, 97:42-51.
17. Fiske S.T. Words! Words! Words! Confronting the problem of observer and self reports. Personality research, methods, and theory. A festschrift honoring 1995, 221-240. Erlbaum Associates, Hillsdale, N.J. D.W. Fiske, P.E. Shrout, S.T. Fiske, D.W. Fiske (Eds.).
18. Foddy W. An empirical evaluation of in-depth probes used to pretest survey questions. Sociological Methods & Research 1998, 27:103-133. 10.1177/0049124198027001003.
19. Gaugler B.B., Thornton G.C. Number of assessment center dimensions as a determinant of assessor accuracy. Journal of Applied Psychology 1989, 74:611-618.
20. Gillmore G.M., Kane M.T., Smith P.L. The dependability of student evaluations of teaching effectiveness: matching the conclusions to the design. Educational and Psychological Measurement 1983, 43:1015-1018. 10.1177/001316448304300409.
21. Greguras G.J., Robie C. A new look at within-source interrater reliability of 360-degree feedback ratings. Journal of Applied Psychology 1998, 83:960-968.
22. Guilford J.P. Psychometric methods 1954, McGraw-Hill, New York, NY. 2nd ed.
23. Hattie J. Visible learning: A synthesis of meta-analyses relating to achievement 2009, Routledge, London.
24. Helmke A. Unterrichtsqualität und Lehrerprofessionalität. Diagnose, Evaluation und Verbesserung des Unterrichts 2010, Klett-Kallmeyer, Seelze, Germany.
25. Helmke A., Helmke T., Heyne N., Hosenfeld A., Hosenfeld I., Schrader F.-W., et al. Zeitnutzung im Grundschulunterricht. Ergebnisse der Unterrichtsstudie " VERA - Gute Unterrichtspraxis" . Zeitschrift für Grundschulforschung 2008, 1:23-36.
26. Helmke A., Helmke T., Lenske L., Pham G., Praetorius A.-K., Schrader F.-W., et al. Studienbrief Unterrichtsdiagnostik 2011, University of Koblenz-Landau, Landau, Germany.
27. Hill H.C., Kapitula L., Umland K. A validity argument approach to evaluating teacher value-added scores. American Educational Research Journal 2011, 48:794-831. 10.3102/0002831210387916.
28. Hoyt W.T. Rater bias in psychological research: when is it a problem and what can we do about it?. Psychological Methods 2000, 5(1):64-86. 10.1037/1082-989X.5.1.64.
29. Hoyt W.T., Kerns M.-D. Magnitude and moderators of bias in observer ratings: a meta-analysis. Psychological Methods 1999, 4(4):403-424. 10.1037/1082-989X.4.4.403.
30. Hoyt W.T., Melby J.N. Dependability of measurement in counseling psychology: an introduction to generalizability theory. The Counseling Psychologist 1999, 27:325-352. 10.1177/0011000099273003.
31. Hugener I., Pauli C., Reusser K., Lipowsky F., Rakoczy K., Klieme E. Teaching patterns and learning quality in Swiss and German mathematics lessons. Learning and Instruction 2009, 19:66-78.
32. Jones E.E., Nisbett R.E. The actor and the observer: Divergent perceptions of the causes of behavior 1971, General Learning Press, Morristown, NJ.
33. Klieme E. Empirische Unterrichtsforschung: Aktuelle Entwicklungen, theoretische Grundlagen und fachspezifische Befunde [Empirical research on classroom teaching: Current developments, theoretical basis and subject-specific results]. Zeitschrift für Pädagogik 2006, 52:765-773.
34. Klieme E., Lipowsky F., Rakoczy K., Ratzka N. Qualitätsdimensionen und Wirksamkeit von Mathematikunterricht. Theoretische Grundlagen und ausgewählte Ergebnisse des Projekts " Pythagoras" . Untersuchungen zur Bildungsqualität von Schule. Abschlussbericht des DFG-Schwerpunktprogramms 2006, 127-146. Waxmann, Münster, Germany. M. Prenzel, L. Allolio-Näcke (Eds.).
35. Klieme E., Schümer G., Knoll S. Mathematikunterricht in der Sekundarstufe I: Aufgabenkultur und Unterrichtsgestaltung. TIMSS - Impulse für Schule und Unterricht. Forschungsbefunde, Reforminitiativen, Praxisberichte und Video-Dokumente 2001, 43-57. Medienhaus Biering, Munich, Germany. Bundesministerium für Bildung und Forschung (BMBF) (Ed.).
36. Kluger A.N., DeNisi A. Effects of feedback intervention on performance: a historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin 1996, 119:254-284. 10.1037/0033-2909.119.2.254.
37. Kobarg M., Seidel T. Coding manual - process-oriented teaching. How to run a video study. Technical report of the IPN video study 2005, 108-144. Waxmann, Münster, Germany. T. Seidel, M. Prenzel, M. Kobarg (Eds.).
38. Kultusministerkonferenz Standards für die Lehrerbildung: Bildungswissenschaften Retrieved from. http://www.kmk.org/fileadmin/veroeffentlichungen_beschluesse/2004/2004_12_16-Standards-Lehrerbildung.pdf.
39. Kunter M. Multiple Ziele im Mathematikunterricht 2005, Waxmann, Münster, Germany.
40. Lanahan L., McGrath D., McLaughlin M., Burian-Fitzgerald M., Slaganik L. Fundamental problems in the measurement of instructional processes: Estimating reasonable effect sizes and conceptualizing what is important to measure 2005, American Institutes of Research, Washington, D.C.
41. Lievens F. Factors which improve the construct validity of assessment centers: a review. International Journal of Selection and Assessment 1998, 6:141-152. 10.1111/1468-2389.00085.
42. Lüdtke O., Robitzsch A., Trautwein U., Kunter M. Assessing the impact of learning environments: how to use student ratings of classroom or school characteristics in multilevel modeling. Contemporary Educational Psychology 2009, 34:120-131.
43. Lumley T., McNamara T. Rater characteristics and rater bias: implications for training. Language Testing 1995, 12:54-71. 10.1177/026553229501200104.
44. Matsumura L.C., Garnier H.E., Pascal J., Valdés R. Measuring instructional quality in accountability systems: Classroom assignments and student achievement 2002, (No. 3), National Center for Research on Evaluation, Standards, and Student Testing, Los Angeles, CA, Retrieved from. http://www.cse.ucla.edu/products/Reports/TR582.pdf.
45. Mayring P. Qualitative content analysis. A companion to qualitative research 2004, 266-269. SAGE, London. U. Flick (Ed.).
46. Myford C.M., Wolfe E.W. Detecting and measuring rater effects using many-facet Rasch measurement: part I. Journal of Applied Measurement 2003, 4:386-422.
47. Newton X.A. Developing indicators of classroom practice to evaluate the impact of district mathematics reform initiative: a generalizability analysis. Studies in Educational Evaluation 2010, 36:1-13. 10.1016/j.stueduc.2010.10.002.
48. Nunnally J.C. Psychometric theory 1978, McGraw-Hill, New York, NY. 2nd ed.
49. Oelkers J., Reusser K. Developing quality - Safeguarding standards - Handling differentiation. Education research 2008, Vol. 27. Federal Ministry of Education and Research (BMBF), Bonn.
50. Petko D., Waldis M., Pauli C., Reusser K. Methodologische überlegungen zur videogestützten Forschung in der Matemhamtikdidaktik. Ansätze der TIMSS 1999 Video Studie und ihrer schweizerischen Erweiterung. Zentralblatt für Didaktik der Mathematik 2003, 35:265-280.
51. Pianta R.C., Hamre B.K. Conceptualization, measurement, and improvement of classroom processes: standardized observation can leverage capacity. Educational Researcher 2009, 38(2):109-119. 10.3102/0013189X09332374.
52. Pietsch M., Tosana S. Beurteilereffekte bei der Messung von Unterrichtsqualität. Das Multifacetten-Rasch-Modell und die Generalisierbarkeitstheorie als Methoden der Qualitätssicherung in der externen Evaluation von Schulen. Zeitschrift für Erziehungswissenschaft 2008, 11:430-452. 10.1007/s11618-008-0021-7.
53. Rakoczy K., Pauli C. Hoch inferentes Rating: Beurteilung der Qualität unterrichtlicher Prozesse. Materialien zur Bildungsforschung 2006, Vol. 15:206-233. GFPF, Frankfurt am Main, Germany. I. Hugener, C. Pauli, K. Reusser (Eds.).
54. Reusser K. Konstruktivismus - vom epistemologischen Leitbegriff zur Erneuerung der didaktischen Kultur. Didaktik auf psychologischer Grundlage. Von Hans Aeblis kognitionspsychologischer Didaktik zur modernen Lehr- und Lernforschung 2006, 151-168. h.e.p., Bern, Swiss. M. Baer, M. Fuchs, P. Füglister, K. Reusser, H. Wyss (Eds.).
55. Schacter J., Thum Y.M. Paying for high- and low-quality teaching. Economics of Education Review 2004, 23:411-430. 10.1016/j.econedurev.2003.08.002.
56. Scheele B. Struktur-Lege-Verfahren als Dialog-Konsens-Methodik. Ein Zwischenfazit zur Forschungsentwicklung bei der rekonstruktiven Erhebung Subjektiver Theorien 1992, Aschendorff, Münster, Germany.
57. Seidel T. Video analysis strategies of the IPN video study - a methodological overview. How to run a video study. Technical report of the IPN video study 2005, 70-78. Waxmann, Münster, Germany. T. Seidel, M. Prenzel, M. Kobarg (Eds.).
58. Shavelson R.J., Dempsey-Atwood N. Generalizability of measures of teaching behavior. Review of Educational Research 1976, 46:553-611. 10.3102/00346543046004553.
59. Shavelson R.J., Webb N.M. Generalizability theory: 1973-1980. British Journal of Mathematical and Statistical Psychology 1981, 34:133-166. 10.1111/j.2044-8317.1981.tb00625.x.
60. Shavelson R.J., Webb N.M. Generalizability theory: A primer 1991, Sage, Newbury Park.
61. Shuell T.J. Learning theories and educational paradigms. International encyclopedia of the social and behavioral sciences 2001, 8613-8620. Elsevier, Oxford. P.B. Baltes (Ed.).
62. Snow R.E. Aptitude-treatment interaction as a framework for research on individual differences in learning. Learning and individual differences: Advances in theory and research 1989, 13-59. Freeman, New York, NY. P.L. Ackerman, R.J. Sternberg, R. Glaser (Eds.).
63. Spillane J.P., Zeuli J.S. Reform and teaching: exploring patterns of practice in the context of national and state mathematics reforms. Educational Evaluation and Policy Analysis 1999, 21:1-27. 10.3102/01623737021001001.
64. Stigler J., Gonzales P., Kawanaka T., Knoll S., Serrano A. The TIMSS videotape classroom study: Methods and findings from an exploratory research project on eighth-grade mathematics instruction in Germany, Japan, and the United States 1999, National Center for Education Statistics, Washington, D.C.
65. Stigler J.W., Hiebert J. The teaching gap: Best ideas from the world's teachers for improving education in the classroom 1999, Free Press, New York, NY.
66. Storms M.D. Videotape and the attribution process: reversing actors' and observers' points of view. Journal of Personality and Social Psychology 1973, 27:165-175.
67. Strong M., Gargani J., Hacifazlioglu ö Do we know successful teachers when we see one? Experiments in the identification of effective teachers. Journal of Teacher Education 2011, 62:367-382. 10.1177/0022487110390221.
68. Tourangeau R. Cognitive aspects of survey measurement and mismeasurement. International Journal of Public Opinion Research 2003, 15:3-7.
69. Uggerslev K.L., Sulsky L.M. Using frame-of-reference training to understand the implications of rater idiosyncrasy for rating accuracy. Journal of Applied Psychology 2008, 93:711-719. 10.1037/0021-9010.93.3.711.
70. Urdan T., Mestas M. The goals behind performance goals. Journal of Educational Psychology 2006, 98:354-365. 10.1037/0022-0663.98.2.354.
71. Walberg H.J., Haertel G.D. Evaluation reflections. Validity and use of educational environment assessments. Studies in Educational Evaluation 1980, 6:225-238.
72. Walberg H.J., Paik S.J. Effective educational practices 2000, UNESCO International Academy of Education, Brussels.
73. Waxman H.C., Hilberg R.S., Tharp R.G. Future directions for classroom observation research. Observational research in U.S. classrooms. New approaches for understanding cultural and linguistic diversity 2004, 266-277. Cambridge University Press, Cambridge. H.C. Waxman, R.G. Tharp, R.S. Hilberg (Eds.).
74. Webb N.M., Schlackman J., Sugrue B. The dependability and interchangeability of assessment methods in science 2000, (Technical report no. 515), UCLA, Center for Research on Evaluation, Standards, and Student Testing, Los Angeles.
75. Willis G.B. Cognitive interviewing: A tool for improving questionnaire design 2005, Sage, Thousand Oaks, CA.