Assessing teachers' judgements of students' academic motivation and emotions across two rating methods

Educational Research and Evaluation

Volumn 20, Issue 5, 2014, Pages 411-427

  Zhu, Mingjing ^a   Urhahne, Detlef ^b  

^a UNIVERSITY OF MUNICH   (Germany)

^b UNIVERSITY OF PASSAU   (Germany)

Author keywords

rating methods; students motivation and emotions; teacher judgement accuracy

Indexed keywords

EID: 84908051492     PISSN: 13803611     EISSN: 17444187     Source Type: Journal    
DOI: 10.1080/13803611.2014.964261     Document Type: Article

References (65)

1. Auger, R. W. (2004). The accuracy of teacher reports in the identification of middle school students with depressive symptomatology. Psychology in the Schools, 41, 379–389. doi:10.1002/pits.10164
2. Barrett, P. T., & Paltiel, L. (1996). Can a single item replace an entire scale? POP vs. the OPQ 5.2. Selection and Development Review, 12, 1–4.
3. Bates, C., & Nettelbeck, T. (2001). Primary school teachers’ judgements of reading achievement. Educational Psychology, 21, 177–187. doi:10.1080/01443410020043878
4. Begeny, J. C., Eckert, T. L., Montarello, S. A., & Storie, M. S. (2008). Teachers’ perceptions of students’ reading abilities: An examination of the relationship between teachers’ judgments and students’ performance across a continuum of rating methods. School Psychology Quarterly, 23, 43–55. doi:10.1037/1045-3830.23.1.43
5. Begeny, J. C., Krouse, H. E., Brown, K. G., & Mann, C. M. (2011). Teacher judgments of students’ reading abilities across a continuum of rating methods and achievement measures. School Psychology Review, 40, 23–38.
6. Bergkvist, L., & Rossiter, J. R. (2007). The predictive validity of multiple-item versus single-item measures of the same constructs. Journal of Marketing Research, 44, 175–184.
7. Boehnke, K., Silbereisen, R. K., Reynolds, C. R., & Richmond, B. O. (1986). What I think and feel – German experience with the revised form of the children's manifest anxiety scale. Personality and Individual Differences, 7, 553–560. doi:10.1016/0191-8869(86)90132-7
8. Bong, M., & Skaalvik, E. M. (2003). Academic self-concept and self-efficacy: How different are they really? Educational Psychology Review, 15, 1–40. doi:10.1023/A:1021302408382
9. Brophy, J. E. (1983). Research on the self-fulfilling prophecy and teacher expectations. Journal of Educational Psychology, 75, 631–661. doi:10.1037/0022-0663.75.5.631
10. Carr, M., & Kurtz, B. E. (1991). Teachers’ perceptions of their students’ metacognition, attributions, and self-concept. British Journal of Educational Psychology, 61, 197–206. doi:10.1111/j.2044-8279.1991.tb00975.x
11. Chen C. S., & Stevenson, H. W. (1995). Motivation and mathematics achievement: A comparative study of Asian-American, Caucasian-American, and East Asian high school students. Child Development, 66, 1215–1234. doi:10.1111/j.1467-8624.1995.tb00932.x
12. Christ, T. J., Riley-Tillman, T. C., Chafouleas, S. M., & Jaffery, R. (2011). Direct Behavior Rating (DBR): An evaluation of alternate definitions to assess classroom behaviors. School Psychology Review, 40, 181–199.
13. Christophersen, T., & Konradt, U. (2011). Reliability, validity, and sensitivity of a single-item measure of online store usability. International Journal of Human-Computer Studies, 69, 269–280. doi:10.1016/j.ijhcs.2010.10.005
14. Coladarci, T. (1986). Accuracy of teacher judgments of student responses to standardized test items. Journal of Educational Psychology, 78, 141–146. doi:10.1037/0022-0663.78.2.141
15. Cronbach, L. (1955). Processes affecting scores on “understanding of others” and “assumed similarity”. Psychological Bulletin, 52, 177–193. doi:10.1037/h0044919
16. Demaray, M. K., & Elliott, S. N. (1998). Teachers’ judgments of students academic functioning: A comparison of actual and predicted performances. School Psychology Quarterly, 13, 8–24. doi:10.1037/h0088969
17. DeSalvo, K. B., Fisher, W. P., Tran, K., Bloser, N., Merrill, W., & Peabody, J. (2006). Assessing measurement properties of two single-item general health measures. Quality of Life Research, 15, 191–201. doi:10.1007/s11136-005-0887-2
18. Diamantopoulos, A., Sarstedt, M., Fuchs, C., Wilczynski, P., & Kaiser, S. (2012). Guidelines for choosing between multi-item and single-item scales for construct measurement: A predictive validity perspective. Journal of the Academy of Marketing Science, 40, 434–449. doi:10.1007/s11747-011-0300-3
19. Dicke, A.-L., Lüdtke, O., Trautwein, U., Nagy, G., & Nagy, N. (2012). Judging students’ achievement goal orientations: Are teacher ratings accurate? Learning and Individual Differences, 22, 844–849. doi:10.1016/j.lindif.2012.04.004
20. Eckert, T. L., Dunn, E. K., Codding, R. S., Begeny, J. C., & Kleinmann, A. E. (2006). Assessment of mathematics and reading performance: An examination of the correspondence between direct assessment of student performance and teacher report. Psychology in the Schools, 43, 247–265. doi:10.1002/pits.20147
21. Feinberg, A. B., & Shapiro, E. S. (2003). Accuracy of teacher judgments in predicting oral reading fluency. School Psychology Quarterly, 18, 52–65. doi:10.1521/scpq.18.1.52.20876
22. Feinberg, A. B., & Shapiro, E. S. (2009). Teacher accuracy: An examination of teacher-based judgments of students’ reading with differing achievement levels. Journal of Educational Research, 102, 453–462. doi:10.3200/JOER.102.6.453-462
23. Funder, D. C. (1995). On the accuracy of personality judgment: A realistic approach. Psychological Review, 102, 652–670. doi:10.1037/0033-295X.102.4.652
24. Funder, D. C. (2012). Accuracy of personality judgment. Current Directions in Psychological Sciences, 21, 177–182. doi:10.1177/0963721412445309
25. Gardner, D. G., Cummings, L. L., Dunham, R. B., & Pierce, J. L. (1998). Single-item versus multiple-item measurement scales: An empirical comparison. Educational and Psychological Measurement, 58, 898–915. doi:10.1177/0013164498058006003
26. Givvin, K. B., Stipek, D. J., Salmon, J. M., & MacGyvers, V. L. (2001). In the eyes of the beholder: Students’ and teachers’ judgments of students’ motivation. Teaching and Teacher Education, 17, 321–331. doi:10.1016/S0742-051X(00)00060-3
27. Greene, B. A., Miller, R. B., Crowson, H. M., Duke, B. L., & Akey, K. L. (2004). Predicting high school students’ cognitive engagement and achievement: Contributions of classroom perceptions and motivation. Contemporary Educational Psychology, 29, 462–482. doi:10.1016/j.cedpsych.2004.01.006
28. Grice, J., Mignogna, M., & Badzinski, S. (2011). The Dynamic Analog Scale: A generic method for single-item measurement. Personality and Individual Differences, 50, 481–485. doi:10.1016/j.paid.2010.11.012
29. Hardré, P. L., Davis, K. A., & Sullivan, D. W. (2008). Measuring teacher perceptions of the “how” and “why” of student motivation. Educational Research and Evaluation, 14, 155–179. doi:10.1080/13803610801956689
30. Hascher, T. (2011). Wellbeing. In S. Järvelä (Ed.), Social and emotional aspects of learning (pp. 99–105). Oxford, UK: Elsevier.
31. Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses related to achievement. London, UK: Routledge.
32. Hidi, S., & Harackiewicz, M. M. (2000). Motivating the academically unmotivated: A critical issue for the 21st century. Review of Educational Research, 70, 151–179. doi:10.3102/00346543070002151
33. Hoeppner, B. B., Kelly, J. F., Urbanoski, K. A., & Slaymaker, V. (2011). Comparative utility of a single-item versus multiple-item measure of self-efficacy in predicting relapse among young adults. Journal of Substance Abuse Treatment, 41, 305–312. doi:10.1016/j.jsat.2011.04.005
34. Hoge, R. D., & Coladarci, T. (1989). Teacher-based judgments of academic achievement: A review of literature. Review of Educational Research, 59, 297–313. doi:10.3102/00346543059003297
35. Hoge, R. D., & Cudmore, L. (1986). The use of teacher-judgment measures in the identification of gifted pupils. Teaching and Teacher Education, 2, 181–196. doi:10.1016/0742-051X(86)90016-8
36. Jordan, J. S., & Turner, B. A. (2008). The feasibility of single-item measures for organizational justice. Measurement in Physical Education and Exercise Science, 12, 237–257. doi:10.1080/10913670802349790
37. Jussim, L. (1989). Teacher expectations: Self-fulfilling prophecies, perceptual biases, and accuracy. Journal of Personality and Social Psychology, 57, 469–480. doi:10.1037/0022-3514.57.3.469
38. Kaiser, J., Retelsdorf, J., Südkamp, A., & Möller, J. (2013). Achievement and engagement: How student characteristics influence teacher judgments. Learning and Instruction, 28, 73–84. doi:10.1016/j.learninstruc.2013.06.001
39. Karing, C. (2009). Diagnostische Kompetenz von Grundschul- und Gymnasiallehrkräften im Leistungsbereich und im Bereich Interessen [Diagnostic competence of elementary and secondary school teachers in the domains of competence and interests]. Zeitschrift für Pädagogische Psychologie, 23, 197–209. doi:10.1024/10100652.23.34.197
40. Kenny, D. A., & West, T. V. (2010). Similarity and agreement in self- and other perception: A meta-analysis. Personality and Social Psychology Review, 14, 196–213. doi:10.1177/1088868309353414
41. Marsh, H. W., & Craven, R. G. (1991). Self-other agreement on multiple dimensions of preadolescent self-concept: Inferences by teachers, mothers, and fathers. Journal of Educational Psychology, 83, 393–404. doi:10.1037/0022-0663.83.3.393
42. Martin, A. (2006). The relationship between teachers’ perceptions of student motivation and engagement and teachers’ enjoyment of and confidence in teaching. Asia-Pacific Journal of Teacher Education, 34, 73–93. doi:10.1080/13598660500480100
43. Meng, X.-L., Rosenthal, R., & Rubin, D. B. (1992). Comparing correlated correlation coefficients. Psychological Bulletin, 111, 172–175. doi:10.1037/0033-2909.111.1.172
44. Pekrun, R., Frenzel, A. C., Goetz, T., & He, S. (2005). Achievement Emotions Questionnaire – Mathematics Chinese version (AEQ-M Chinese). User’s manual. Munich, Germany: University of Munich.
45. Pekrun, R., Goetz, T., Frenzel, A. C. (2005). Achievement Emotions Questionnaire – Mathematics (AEQ-M). User’s manual. Munich, Germany: University of Munich.
46. Pekrun, R., Goetz, T., Titz, W., & Perry, R. P. (2002). Academic emotions in students’ self-regulated learning and achievement: A program of quantitative and qualitative research. Educational Psychologist, 37, 91–106. doi:10.1207/S15326985EP3702_4
47. PISA-Konsortium Deutschland. (Eds.). (2006). PISA 2003. Dokumentation der Erhebungsinstrumente [PISA 2003. Documentation of measuring instruments]. Münster, Germany: Waxmann.
48. Praetorius, A.-K., Berner, V.-D., Zeinz, H., Scheunpflug, A., & Dresel, M. (2013). Judgment confidence and judgment accuracy of teachers in judging self-concepts of students. Journal of Educational Research, 106, 64–76. doi:10.1080/00220671.2012.667010
49. Salili, F. (1996). Learning and motivation: An Asian perspective. Psychology and Developing Societies, 8, 55–81.
50. Schrader, F. W., & Helmke, A. (1987). Diagnostische Kompetenz von Lehrern: Komponenten und Wirkungen [Diagnostic competence of teachers: Components and effects]. Empirische Pädagogik, 1, 27–52.
51. Schuck, L. A., Oehler-Stinnett, J., & Stinnett, T. A. (1995). Predictive validity of the Teacher Rating of Academic Achievement Motivation (TRAAM) with Hispanic students. Journal of Psychoeducational Assessment, 13, 143–156. doi:10.1177/073428299501300204
52. Seidel, T., & Shavelson, R. J. (2007). Teaching effectiveness research in the past decade: The role of theory and research design in disentangling meta-analysis results. Review of Educational Research, 77, 454–499. doi:10.3102/0034654307310317
53. Sharpley, C. F., & Edgar, E. (1986). Teachers’ rating vs. standardized test: An empirical investigation of agreement between two indices of achievement. Psychology in the Schools, 23, 106–111. doi:10.1002/1520-6807(198601)23:1<106::AID-PITS2310230117>3.0.CO;2-C
54. Shavelson, R. J. (1983). Review of research on teachers’ pedagogical judgment, plans, and decisions. The Elementary School Journal, 83, 392–413. doi:10.3102/00346543051004455
55. Shavelson, R. J., & Stern, P. (1981). Research on teachers’ pedagogical thoughts, judgments, decisions, and behavior. Review of Educational Research, 51, 455–498. doi:10.3102/00346543051004455
56. Spinath, B. (2005). Akkuratheit der Einschätzung von Schülermerkmalen durch Lehrer und das Konstrukt der diagnostischen Kompetenz [Accuracy of teacher judgments on student characteristics and the construct of diagnostic competence]. Zeitschrift für Pädagogische Psychologie, 19, 85–95. doi:10.1024/1010-0652.19.1.85
57. Stevenson, H. W., Lee, S., Chen, C., Lummis, M., Stigler, J., Liu, F., & Fang, G. (1990). Mathematics achievement of children in China and the United States. Child Development, 61, 1053–1066. doi:10.2307/1130875
58. Stevenson, H. W., Lee, S., & Stigler, J. W. (1986). Mathematics achievement of Chinese, Japanese, and American children. Science, 231, 693–699. doi:10.1126/science.3945803
59. Stipek, D., Givvin, K., Salmon, J., & MacGyvers, V. (1998). Can a teacher intervention improve classroom practices and student motivation in mathematics? Journal of Experimental Education, 66, 319–337. doi:10.1080/00220979809601404
60. Südkamp, A., Kaiser, J., & Möller, J. (2012). Accuracy of teachers’ judgments of students’ academic achievement: A meta-analysis. Journal of Educational Psychology, 104, 743–762. doi:10.1037/a0027627.supp
61. Südkamp, A., Möller, J., & Pohlmann, B. (2008). Der Simulierte Klassenraum: Eine experimentelle Untersuchung zur diagnostischen Kompetenz [The simulated classroom: An experimental study on diagnostic competence]. Zeitschrift für Pädagogische Psychologie, 22, 261–276. doi:10.1024/1010-0652.22.34.261
62. Urhahne, D., Chao, S. H., Florineth, M. L., Luttenberger, S., & Paechter, M. (2011). Academic self-concept, learning motivation, and test anxiety of the underestimated student. British Journal of Educational Psychology, 81, 161–177. doi:10.1348/000709910X504500
63. Urhahne, D., Zhou, J., Stobbe, M., Chao, S. H., Zhu, M. J., & Shi, J. N. (2010). Motivationale und affektive Merkmale unterschätzter Schüler. Ein Beitrag zur diagnostischen Kompetenz von Lehrkräften [Motivational and affective characteristics of underestimated students: A contribution to the diagnostic competence of teachers]. Zeitschrift für Pädagogische Psychologie, 24, 275–288. doi:10.1024/1010-0652/a000021
64. Woods, S. A., & Hampson, S. E. (2005). Measuring the Big Five with single items using a bipolar response scale. European Journal of Personality, 19, 373–390. doi:10.1002/per.542
65. Zhu, M. J., & Urhahne, D. (2014). Teachers’ judgements of students’ foreign-language achievement. European Journal of Psychology of Education. Advance online publication. doi:10.1007/s10212-014-0225-6