Reliability and agreement of student ratings of the classroom environment: A reanalysis of TIMSS data

Learning Environments Research

Volumn 9, Issue 3, 2006, Pages 215-230

  Lüdtke, Oliver ^a   Trautwein, Ulrich ^a   Kunter, Mareike ^a   Baumert, Jürgen ^a  

^a MAX PLANCK INSTITUTE FOR HUMAN DEVELOPMENT   (Germany)

Author keywords

Agreement; Climate; Instructional quality; Intraclass correlation; Multilevel analysis; Reliability; Student ratings

Indexed keywords

EID: 33846988224     PISSN: 13871579     EISSN: None     Source Type: Journal    
DOI: 10.1007/s10984-006-9014-8     Document Type: Article

References (39)

1. Anderson, C. S. (1982). The search for school climate: A review of the research. Review of Educational Research, 52, 368-420.
2. Baumert, J., Lehmann, R. H., Lehrke, M., Schmitz, B., Clausen, M., Hosenfeld, I., et al. (1997). TIMSS: Mathematisch-Naturwissenschaftlicher Unterricht im internationalen Vergleich [TIMSS: Mathematics and science instruction in an international comparison]. Opladen, Germany: Leske and Budrich.
3. Beaton, A. E., Mullis, I. V. S., Martin, M. O., Gonzales, E. J., Kelly, D. L., & Smith, T. A. (1996). Mathematics achievement in the middle school years: IEA's third international mathematics and science study. Chestnut Hill: Boston College.
4. Bliese, P. D. (1998). Group size, ICC values, and group-level correlations: A simulation. Organizational Research Methods, 1, 355-373.
5. Bliese, P. D. (2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In K. J. Klein, & S. W. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations (pp. 349-381). San Francisco: Jossey-Bass.
6. WG indices. Organizational Research Methods, 8, 165-184.
7. Burke, M. J., & Dunlap, W. P. (2002). Estimating interrater agreement with the average deviation index: A user's guide. Organizational Research Methods, 5, 159-172.
8. Burke, M. J., Finkelstein, L. M., & Dusig, M. S. (1999). On average deviation indices for estimating interrater agreement. Organizational Research Methods, 2, 49-68.
9. Chan, D. (1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of compositional models. Journal of Applied Psychology, 83, 234-246.
10. Clausen, M. (2002). Unterrichtsqualität: Eine Frage der Perspektive? [Quality of instruction: A matter of perspective?] Münster, Germany: Waxmann.
11. WG(J) index of agreement. Psychological Methods, 6, 297-310.
12. WG and average deviation interrater agreement indexes. Journal of Applied Psychology, 88, 356-362.
13. Finn, R. H. (1970). A note on estimating the reliability of categorical data. Educational and Psychological Measurement, 30, 71-76.
14. Fraser, B. (1991). Two decades of classroom environment research. In H. J. Walberg (Eds.), Educational environments: Evaluation, antecedents and consequences (pp. 3-27). Elmsford, NY: Pergamon
15. Grawitch, M. J., & Munz, D. C. (2004). Are your data nonindependent? A practical guide to evaluating nonindependence and within-group agreement. Understanding Statistics, 3, 231-257.
16. Griffith, J. (2002). Is quality/effectiveness an empirically demonstrable school attribute? Statistical aids for determining appropriate levels of analysis. School Effectiveness and School Improvement, 13, 91-122.
17. Gruehn, S. (2000). Unterricht und schulisches Lernen: Schüler als Quellen der Unterrichtsbeschreibung [Instruction and learning in school: Students as sources of information]. Münster, Germany: Waxmann.
18. James, L. R., Demaree, R. G., & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85-98.
19. WG: An assessment of within-group interrater agreement. Journal of Applied Psychology, 78, 306-309.
20. Kane, M. T., & Brennan, R. L. (1977). The generalizability of class means. Review of Educational Research, 47, 267-292.
21. Klein, K. J., Conn, A. B., Smith, B. D., & Sorra, J. S. (2001). Is everyone in agreement? An exploration of within-group agreement in employee perceptions of the work environment. Journal of Applied Psychology, 86, 3-16.
22. Kozlowski, S. W., & Hattrup, K. (1992). A disagreement about within-group agreement: Disentangling issues of consistency versus consensus. Journal of Applied Psychology, 77, 161-167.
23. Kunter, M. (2005). Multiple Ziele im Mathematikunterricht [Multiple objectives in mathematics instruction]. Münster, Germany: Waxmann.
24. Kunter, M., Baumert, J., & Köller, O. (2005). Effective classroom management and the development of subject-related interest. Manuscript submitted for publication.
25. WG(J). Organizational Research Methods, 8, 128-138.
26. Lindell, M. K., & Brandt, C. J. (1997). Measuring interrater agreement for ratings of a single target. Applied Psychological Measurement, 21, 271-278.
27. WG(J) indexes. Journal of Applied Psychology, 84, 640-647.
28. Lindell, M. K., & Brandt, C. J. (2000). Climate quality and climate consensus as mediators of the relationship between organizational antecedents and outcomes. Journal of Applied Psychology, 85, 331-348.
29. Lindell, M. K., Brandt, C. J., & Whitney, D. J. (1999). A revised index of interrater agreement for multi-item ratings of a single target. Applied Psychological Measurement, 23, 127-135.
30. Lüdtke, O., Köller, O., Marsh, H. W., & Trautwein, U. (2005). Teacher frame of reference and the big-fish-little-pond effect. Contemporary Educational Psychology, 30, 263-285.
31. Lüdtke, O., Robitzsch, A., & Köller, O. (2002). Statistische Artefakte bei Kontexteffekten in der pädagogisch- psychologischen Forschung [Statistical artifacts in educational studies on context effects]. Zeitschrift für Pädagogische Psychologie, 16, 217-231.
32. Lüdtke, O., Trautwein, U., Kunter, M., & Baumert, J. (2006). Analyse von Lernumwelten: Ansätze zur Bestimmung der Reliabilität und Übereinstimmung von Schülerwahrnehmungen [Analysis of learning environments: Approaches to determining the reliability and agreement of student ratings]. Zeitschrift für Pädagogische Psychologie, 20, 85-96.
33. McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30-46.
34. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models (2nd ed.). Thousand Oaks, CA: Sage
35. Schmidt, F. L., & Hunter, J. E. (1989). Interrater reliability coefficients cannot be computed when only one stimulus is rated. Journal of Applied Psychology, 75, 322-327.
36. Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420-428.
37. Snijders, T. A. B., & Bosker, R. J. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. London: Sage.
38. Urdan, T., Midgley, C., & Anderman, E. M. (1998). The role of classroom goal structure in students' use of self-handicapping strategies. American Educational Research Journal, 35, 101-122.
39. Wong, A. F., Young, D. J., & Fraser, B. J. (1997). A multilevel analysis of learning environments and student attitudes. Educational Psychology, 17, 449-468.