Construct validity of student perceptions of instructional quality is high, but not perfect: Dimensionality and generalizability of domain-independent assessments

Learning and Instruction

Volumn 28, Issue , 2013, Pages 1-11

  Wagner, Wolfgang ^a   Göllner, Richard ^a   Helmke, Andreas ^b   Trautwein, Ulrich ^a   Lüdtke, Oliver ^c  

^a UNIVERSITY OF TÜBINGEN   (Germany)

^b University of Koblenz Landau   (Germany)

^c HUMBOLDT UNIVERSITY   (Germany)

Author keywords

Dimensionality; Generalizability; German and English lessons; Students' perceptions of instructional quality

Indexed keywords

EID: 84879520957     PISSN: 09594752     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.learninstruc.2013.03.003     Document Type: Article

References (51)

1. Abrami P.C., D'Apollonia S., Cohen P.A. Validity of student ratings of instruction: what we know and what we do not. Journal of Educational Psychology 1990, 82:219-231. 10.1037/0022-0663.82.2.219.
2. Abrami P.C., D'Apollonia S., Rosenfield S. The dimensionality of student ratings of instruction: what we know and what we do not. The scholarship of teaching and learning in higher education: An evidence-based perspective 2007, 385-456. Springer, Dordrecht, Netherlands. R.P. Perry, J.C. Smart (Eds.).
3. Anderson C.S. The search for school climate: a review of the research. Review of Educational Research 1982, 52(3):368-420. 10.2307/1170423.
4. Bejar I.I., Doyle K.O. Factorial invariance in student ratings on instruction. Applied Psychological Measurement 1981, 5(3):307-312. 10.1177/014662168100500303.
5. Bollen K.A. Structural equations with latent variables 1989, John Wiley & Sons, Oxford, England.
6. Bradburn N.M., Rips L.J., Shevell S.K. Answering autobiographical questions: the impact of memory and inference on surveys. Science 1987, 236(4798):157-161. 10.1126/science.3563494.
7. den Brok P., Brekelmans M., Wubbels T. Multilevel issues in research using students' perceptions of learning environments: the case of the questionnaire on teacher interaction. Learning Environments Research 2006, 9(3):199-213. 10.1007/s10984-006-9013-9.
8. Bryant F.B., Satorra A. Principles and practice of scaled difference chi-square testing. Structural Equation Modeling: A Multidisciplinary Journal 2012, 19(3):372-398. 10.1080/10705511.2012.687671.
9. Cheung G.W., Rensvold R.B. Testing factorial invariance across groups: a reconceptualization and proposed new method. Journal of Management 1999, 25(1):1-27. 10.1177/01492063990250010.
10. Clausen M. Qualität von Unterricht: Eine Frage der Perspektive? 2002, Waxmann, Münster, Germany.
11. Cronbach L.J. Research on classrooms and schools: Formulation of questions, design and analysis 1976, Stanford University, Stanford.
12. Cronbach L.J., Meehl P.E. Construct validity in psychological tests. Psychological Bulletin 1955, 52(4):281-302. 10.1037/h0040957.
13. D'Apollonia S., Abrami P.C. Navigating student ratings of instruction. American Psychologist 1997, 52(11):1198-1208. 10.1037/0003-066X.52.11.1198.
14. Fisicaro S.A., Lance C.E. Implications of three causal models for the measurement of halo error. Applied Psychological Measurement 1990, 14(4):419-429. 10.1177/014662169001400407.
15. Hattie J. Visible learning: A synthesis of meta-analyses relating to achievement 2009, Routledge, London.
16. Helmke A. Unterrichtsqualität und Lehrerprofessionalität. Diagnose, evaluation und Verbesserung des Unterrichts 2010, Klett-Kallmeyer, Seelze, Germany.
17. Horn J.L., McArdle J.J. Apractical and theoretical guide to measurement invariance in aging research. Experimental Aging Research 1992, 18(3):117-144. 10.1080/03610739208253916.
18. Hox J. Multilevel analysis: Techniques and applications 2010, Routledge, New York, NY. 2nd ed.
19. Kennedy J., Bush A. Overcoming some impediments to the study of teacher effectiveness. Journal of Teacher Education 1976, 27(1):14-17. 10.1177/002248717602700104.
20. Klieme E., Jude N., Rauch D., Ehlers H., Helmke A., Eichler W., et al. Alltagspraxis, Qualität und Wirksamkeit des Deutschunterrichts. Unterricht und Kompetenzerwerb in Deutsch und Englisch. Ergebnisse der DESI-Studie 2008, 319-344. Beltz, Weinheim, Germany. DESI-Konsortium (Ed.).
21. Klieme E., Rakoczy K. Empirische Unterrichtsforschung und Fachdidaktik. Outcome-orientierte Messung und Prozessqualität des Unterrichts. Zeitschrift für Pädagogik 2008, 54:222-237.
22. Klieme E., Schümer G., Knoll S. Mathematikunterricht in der Sekundarstufe I: Aufgabenkultur und Unterrichtsgestaltung. TIMSS-Impulse für Schule und Unterricht. Forschungsbefunde, Reforminitiativen, Praxisberichte und Video-Dokumente 2001, 43-57. Medienhaus Biering, Munich, Germany. E. Klieme, J. Baumert (Eds.).
23. Kunter M., Baumert J. Who is the expert? Construct and criteria validity of student and teacher ratings of instruction. Learning Environments Research 2006, 9(3):231-251. 10.1007/s10984-006-9015-7.
24. Kunter M., Tsai Y.-M., Klusmann U., Brunner M., Krauss S., Baumert J. Students' and mathematics teachers' perceptions of teacher enthusiasm and instruction. Learning and Instruction 2008, 18(5):468-482. 10.1016/j.learninstruc.2008.06.008.
25. Lubke G.H., Dolan C.V., Kelderman H., Mellenbergh G.J. On the relationship between sources of within- and between-group differences and measurement invariance in the common factor model. Intelligence 2003, 31(6):543-566. 10.1016/s0160-2896(03)00051-5.
26. Lüdtke O., Robitzsch A., Trautwein U., Kunter M. Assessing the impact of learning environments: how to use student ratings in multilevel modelling. Contemporary Educational Psychology 2009, 34:123-131. 10.1016/j.cedpsych.2008.12.001.
27. Lüdtke O., Trautwein U., Schnyder I., Niggli A. Simultane Analysen auf Schüler- und Klassenebene. Eine Demonstration der konfirmatorischen Mehrebenen-Faktorenanalyse zur Analyse von Schülerwahrnehmungen am Beispiel der Hausaufgabenvergabe. Zeitschrift für Entwicklungspsychologie und Pädagogische Psychologie 2007, 39(1):1-11. 10.1026/0049-8637.39.1.1.
28. Mainhard M.T., Brekelmans M., Wubbels T. Coercive and supportive teacher behaviour: within- and across-lesson associations with the classroom social climate. Learning and Instruction 2011, 21(3):345-354. 10.1016/j.learninstruc.2010.03.003.
29. Marsh H.W., Lüdtke O., Muthén B.O., Asparouhov T., Morin A.J.S., Trautwein U., et al. Anew look at the big-five factor structure through exploratory structural equation modelling. Psychological Assessment 2010, 22:471-491. 10.1037/a0019227.
30. Marsh H.W., Lüdtke O., Nagengast B., Trautwein U., Morin A.J.S., Abduljabbar A.S., et al. Classroom climate and contextual effects: conceptual and methodological issues in the evaluation of group-level effects. Educational Psychologist 2012, 47(2):106-124. 10.1080/00461520.2012.670488.
31. Marsh H.W., Roche L.A. Making students' evaluations of teaching effectiveness effective. American Psychologist 1997, 52:1187-1197. 10.1037/0003-066X.52.11.1187.
32. Meredith W., Teresi J.A. An essay on measurement and factorial invariance. Medical Care 2006, 44(11, Suppl 3):69-77. 10.1097/01.mlr.0000245438.73837.89.
33. Messick S. Validity of psychological assessment: validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist 1995, 50(9):741-749. 10.1037/0003-066X.50.9.741.
34. Muijs D., Reynolds D. Effective teaching: Evidence and practice 2007, Sage, London.
35. Muthén B.O. Multilevel covariance structure analysis. Sociological Methods and Research 1994, 22(3):376-398. 10.1177/0049124194022003006.
36. Muthén B.O., Muthén L.K. Mplus users' guide 1998-2010, Muthén & Muthén, Los Angeles, CA. 6th ed.
37. Nitsche S., Dickhäuser O., Fasching M.S., Dresel M. Rethinking teachers' goal orientations: conceptual and methodological enhancements. Learning and Instruction 2011, 21(4):574-586. 10.1016/j.learninstruc.2010.12.001.
38. Nunnally J.C. Psychometric theory 1978, McGraw-Hill, New York. 2nd ed.
39. Pianta R.C., La Paro K.M., Hamre B.K. Classroom assessment scoring System™ Manual K-3 2008, Paul H Brookes Publishing, Baltimore, MD, US.
40. Rabe-Hesketh S., Skrondal A., Pickles A. Generalized multilevel structural equation modeling. Psychometrika 2004, 68:167-190. 10.1007/BF02295939.
41. Roch S.G., Paqin A.R., Littlejohn T.W. Do raters agree more on observable items?. Human Performance 2009, 22:391-409. 10.1080/08959280903248344.
42. Rosenshine B. Evaluation of classroom instruction. Review of Educational Research 1970, 40:279-300. 10.2307/1169537.
43. Sälzer C., Trautwein U., Lüdtke O., Stamm M. Predicting adolescent truancy: the importance of distinguishing between different aspects of instructional quality. Learning and Instruction 2012, 22(5):311-319. 10.1016/j.learninstruc.2011.12.001.
44. Scheerens J., Bosker J. The foundations of educational effectiveness 1997, Elsevier, Oxford.
45. Seidel T., Shavelson R.J. Teaching effectiveness research in the past decade: the role of theory and research design in disentangling meta-analysis results. Review of Educational Research 2007, 77(4):454-499. 10.3102/0034654307310317.
46. Turner J.C., Meyer D.K. Studying and understanding the instructional contexts of classrooms: using our past to forge our future. Educational Psychologist 2000, 35(2):69-85. 10.1207/S15326985EP3502_2.
47. Deutsch Englisch Schülerleistungen International: Dokumentation der Erhebungsinstrumente für Schülerinnen und Schüler, Eltern und Lehrkräfte [German English student proficiency international: Documentation of assessment instruments for students, parents, and teachers] 2009, GFPF, DIPF, Frankfurt, Germany. W. Wagner, A. Helmke, E. Rösner (Eds.).
48. Wang M.C., Haertel G.D., Walberg H.J. Toward a knowledge base for school learning. Review of Educational Research 1993, 63(3):249-294. 10.2307/1170546.
49. West S.G., Finch J.F., Curran P.J. Structural equation models with nonnormal variables: problems and remedies. Structural equation modeling: Concepts, issues, and applications 1995, 56-75. Sage Publications, Thousand Oaks, CA. R.H. Hoyle (Ed.).
50. Yu C.-Y. Evaluating cutoff criteria of model fit indices for latent variable models with binary and continuous outcomes 2002, Unpublished doctoral dissertation, University of California, LA.
51. Zyphur M.J., Kaplan S.A., Christian M.S. Assumptions of cross-level measurement and structural invariance in the analysis of multilevel data: problems and solutions. Group Dynamics: Theory, Research, and Practice 2008, 12(2):127-140.