Rater effects in the measurement of quality of classroom teaching: The many-facet Rasch model and the generalizability theory as methods of quality assurance in the external evaluation of school;Beurteilereffekte bei der messung von unterrichtsqualität: Das multifacetten-Rasch-modell und die generalisierbarkeitstheorie als methoden der qualitätssicherung in der externen evaluation von schulen

Zeitschrift fur Erziehungswissenschaft

Volumn 11, Issue 3, 2008, Pages 430-452

  Pietsch, Marcus ^a   Tosana, Simone ^a  

^a Schulinspektion   (United Kingdom)

Author keywords

Generalizability theory; Many facet Rasch model; Quality of classroom teaching; Rater effects; School inspection

Indexed keywords

EID: 77249114257     PISSN: 1434663X     EISSN: 18625215     Source Type: Journal    
DOI: 10.1007/s11618-008-0021-7     Document Type: Article

References (57)

1. Andrich, D. (1978): Application of a psychometric model to ordered categories which are scores with successive integers. In: Applied Psychological Measurement, Vol. 2, pp. 581-594.
2. Bartz, A./Müller, S. (2005): Schulinspektion - Ziele, Funktionen und Qualitätsbereiche. In: Barz, A./Fabian, J./Huber, S. G./Kloft, C./Rosenbusch, H./Sassenscheidt, H. (Hrsg.): Praxiswissen Schulleitung (Stand: 15.11.2005). - München.
3. Behörde für Bildung und Sport (2006): Orientierungsrahmen: Qualitätsentwicklung an Hamburger Schulen. - Hamburg.
4. Berry, K. J./Mielke, P. W. (1988): A generalization of Cohen's kappa agreement to interval measurement and multiple raters. In: Educational and Psychological Measurement, Vol. 48(4), pp. 921-933.
5. Bock, R. D./Brennan, R. L./Muraki E. (2002): The information in multiple ratings. In: Applied Psychological Measurement, Vol. 26(4), pp. 364-375.
6. Bortz, J./Döring, N. (2002): Forschungsmethoden und Evaluation für Human- und Sozialwissenschaftler. - Berlin.
7. Bos, W./Holtappels, H. G./Rösner, E. (2006): Schulinspektionen in den deutschen Bundesländern - eine Baustellenbeschreibung. In: Bos, W./Holtappels, H.-G./Pfeiffer, H./Rolff, H.-G./Schulz-Zander, R. (Hrsg.): Jahrbuch der Schulentwicklung Band 14. Daten, Beispiele und Perspektiven. - Weinheim, S. 81-124.
8. Brennan, R. L. (2001a): Generalizability Theory. - New York.
9. Brennan, R. L. (2001b): Manual for urGENOVA Version 2.1. Iowa Testing Programs Occasional Papers 49. - Iowa City.
10. Briggs, D. C./Wilson, M. R. (2007): Generalizability in item response modeling. In: Journal of Educational Measurement, Vol. 44(2), pp. 131-155.
11. Clauser, B./Linacre, J. M. (1999): Relating Cronbach and Rasch rehabilites. In: Rasch Measurement Transactions, Vol. 13(2), p. 696.
12. Cohen, J. (1960): A coefficient of agreement for nominal scales. In: Educational and Psychological Measurement, Vol. 20, pp. 37-46.
13. Cronbach, L. J./Rajatranam, N./Gleser, C. G. (1963): Theory of generalizability: A liberalization of reliability theory. In: British Journal of Statistical Psychology, Vol. 16, pp. 137-163.
14. Cronbach et al. 1972 = Cronbach, L. J./Gleser, G. C./Nanda, H./Rajatranam, N. (1972): The Dependability of Behavioral Measurements: Theory of generalizability for scores and profiles. - New York.
15. Dobbelstein, P. (in Druck): Die Suche nach dem guten Unterricht. Qualitätsmaßstäbe in der Diskussion. In: Bos, W./Dedering, K./Holtappels, H.-G./Müller, S./Rösner, E. (Hrsg.): Schulische Qualitätsanalyse in NRW. - Köln.
16. Eckes, T. (2004): Facetten des Sprachtestens. Strenge und Konsistenz in der Beurteilung sprachlicher Leistung. In: Wolff, A./Ostermann, A./Closta, C. (Hrsg.): Integration durch Sprache. - Regensburg, S. 485-518.
17. Fisher, W. (1992): Reliability statistics. In: Rasch Measurement Transactions, Vol. 6(3), p. 238.
18. Gleser, G. C./Cronbach, L. J./Rajatranam, N. (1965): Generalizability of scores influenced by multiple sources of variance. In: Psychometrika, Vol. 30, pp. 395-418.
19. Helmke, A. (2003): Unterrichtsqualität erfassen, bewerten, verbessern. - Seelze.
20. Helmke, A. (2006): Was wissen wir über guten Unterricht? In: Pädagogik, 58. Jg., H. 2, S. 42-45.
21. Helmke, A./Helmke, T./Schrader, F.-W. (2007): Unterrichtsqualität: Brennpunkte und Perspektiven der Forschung. In: Arnold, K.-H. (Hrsg.): Unterrichtsqualität und Fachdidaktik. - Bad Heilbrunn, S. 51-72.
22. Hoyt, W T. (2000): Rater bias in psychological research: When is it a problem and what can we do about it? In: Psychological Methods, Vol. 5, pp. 64-86.
23. Hoyt, W. T./Kerns, M.-D. (1999): Magnitude and moderators of bias in observer ratings: A metaanalysis. In: Psychological Methods, Vol. 4, pp. 403-424.
24. Linacre, J. M. (1989): Many-faceted Rasch Measurement. - Chicago.
25. Linacre, J. M. (1997): KR-20/Cronbach alpha or Rasch reliabities: Which tells the truth? In: Rasch Measurement Transactions, Vol. 11(3), pp. 580-581.
26. Linacre, J. M. (1998): Linking constants with common items and judges. In: Rasch Measurement Transactions, Vol. 12(1), p. 621.
27. Linacre, J. M. (2001): Generalizability theory and Rasch measurement. In: Rasch Measurement Transactions, Vol. 15(1), pp. 806-807.
28. Linacre, J. M. (2002): What do infit and outfit, mean-square and standardized mean? In: Rasch Measurement Transactions, Vol. 16(2), p. 878.
29. Linacre, J. M. (2007): A User's Guide to Facets: Rasch model computer programs. - Chicago.
30. Lunz, M. E./Stahl, J. A./Wright, B. D. (1994): Interjudge reliability and decision reproducibility. In: Educational and Psychological Measurement, Vol. 54(4), pp. 913-925.
31. Lunz, M. E./Wright, B. D./Linacre, J. M. (1990): Measuring the impact of judge severity on examination scores. In: Applied Measurement in Education, Vol. 3(4), pp. 331-345.
32. Marcoulides, G. A./Drezner, Z. (2000): A procedure for detecting pattern clustering in measurement designs. In: Wilson, M. R./Engelhard, G. (Eds.): Objective Measurement: Theory into practice. - Stamford, pp. 287-303.
33. Maritzen, N. (2006): Eine Trendanalyse - Schulinspektion zwischen Aufsicht und Draufsicht. In: Buchen, H./Horster, L./Rolff, H.-G. (Hrsg.): Schulinspektion und Schulleitung. - Stuttgart, S. 7-26.
34. Masters, G. N. (1982): A Rasch model for partial credit scoring. In: Psychometrika, Vol. 47, pp. 149-174.
35. Meyer, H. (2004): Was ist guter Unterricht? - Berlin.
36. Myford, C. M./Wolfe, E. W. (2000): Strengthening the Ties that Bind: Improving the linking network in sparsely connected rater designs. - Princeton.
37. Myford, C. M./Wolfe, E. W. (2003): Detecting and measuring rater effects using many-facet Rasch measurement: Part 1. In: Journal of Applied Measurement, Vol. 4(4), pp. 386-422.
38. Myford, C. M./Wolfe, E. W. (2004): Detecting and measuring rater effects using many-facet Rasch measurement: Part 2. In: Journal of Applied Measurement, Vol. 5(2), pp. 189-227.
39. Myford, C. M./Marr, D. B./Linacre, J. M. (1996): Reader Calibration and its Potential Role in Equating for the Test of Written English. - Princeton.
40. Programme for educational research and development (2007): G-String II. A Windows Wrapper for urGENOVA. - Hamilton.
41. Rasch, G. (1960): Probabilistic Models for Some Intelligence and Attainment Tests. - Kopenhagen.
42. Rost, J. (2004): Lehrbuch Testtheorie - Testkonstruktion. - Bern.
43. Saal, F. E./Downey, R. G./Layhey, M. (1980): Rating the raters. Assessing the psychometric quality of rating data. In: Psychological Bulletin, Vol. 88, pp. 413-428.
44. Shavelson, R. J./Webb, N. M. (1991): Generalizability Theory: A primer. - Newbury Park.
45. Smith, E.V/Kulikovich, J. M. (2004): An application of generalizability theory and many-facet Rasch measurement using a complex problem-solving skills assessment. In: Educational and Psychological Measurement, Vol. 64, pp. 617-639.
46. Smith, R. M. (2004): Fit analysis in latent trait measurement models. In: Smith, E. V./Smith, R. M. (Eds.): Introduction to Rasch Measurement: Theory, models and applications. - Maple Grove, pp. 73-92.
47. Smith, R. M./Schumacker, R. E./Bush, M. J. (1998): Using item mean squares to evaluate fit to the Rasch model. In: Journal of Outcome Measurement, Vol. 2(1), pp. 66-78.
48. Stahl, J. A. (1994): What does generalizability theory offer that many-facet Rasch measurement cannot duplicate? In: Rasch Measurement Transactions, Vol. 8(1), pp. 342-343.
49. Webb, N. M./Shavelson, R. J./Haertel, E. H. (2006): Reliability coefficients and generalizability theory. In: Rao, C. R./Sinharay, S. (Eds.). Handbook of Statistics, Vol. 26: Psychometrics. - Amsterdam, pp. 81-124.
50. Wolfe, E. W./Chiu, C. W. T./Myford, C. M. (1999): The Manifestation of Common Rater Errors in Multi-faceted Rasch Analyses. - Princeton.
51. Wright, B. D. (1993): Logits? In: Rasch Measurement Transactions, Vol. 7(2), p. 288.
52. Wright, B. D./Linacre, J. M. (1994): Reasonable mean-square fit values. In: Rasch Measurement Transactions, Vol. 8(3), p. 370.
53. Wright, B. D./Masters, G. N. (1982): Rating Scale Analysis: Rasch measurement. - Chicago.
54. Wright, B. D./Masters, G. N. (2002): Number of person or item strata. In: Rasch Measurement Transactions, Vol. 16(3), p. 888.
55. Wright, B. D./Stone, M. (1979): Best Test Design. - Chicago.
56. Wright, B. D./Stone, M. (1999): Measurement Essentials. - Wilmington.
57. Wu, M. L./Adams, R. J./Wilson, M. R. (1998): ACER ConQuest. Generalised item response modelling software. - Melbourne.