Meta-analysis of faculty's teaching effectiveness: Student evaluation of teaching ratings and student learning are not related

Studies in Educational Evaluation

Volumn 54, Issue , 2017, Pages 22-42

  Uttl, Bob ^a   White, Carmela A ^a,b   Gonzalez, Daniela Wong ^a,c  

^a MOUNT ROYAL UNIVERSITY   (Canada)

^b UNIVERSITY OF BRITISH COLUMBIA   (Canada)

^c UNIVERSITY OF WINDSOR   (Canada)

Author keywords

Evaluation of faculty; Meta analysis of student evaluation of teaching; Multisection studies; SET and learning correlations; Teaching effectiveness; Validity

Indexed keywords

EID: 84996659182     PISSN: 0191491X     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.stueduc.2016.08.007     Document Type: Article

References (90)

1. Abrami, P.C., d'Apollonia, S., Current concerns are past concerns. American Psychologist 54:7 (1999), 519–520, 10.1037/0003-066X.54.7.519.
2. Abrami, P.C., Cohen, P.A., d'Apollonia, S., Implementation problems in meta-analysis. Review of Educational Research 58:2 (1988), 151–179, 10.3102/00346543058002151.
3. Abrami, P.C., D'Apollonia, S., Cohen, P.A., Validity of student ratings of instruction: What we know and what we do not. Journal of Educational Psychology 82:2 (1990), 219–231, 10.1037/0022-0663.82.2.219.
4. Beleche, T., Fairris, D., Marks, M., Do course evaluations truly reflect student learning? Evidence from an objectively graded post-test. Economics of Education Review 31:5 (2012), 709–719, 10.1016/j.econedurev.2012.05.001.
5. Benbassat, J., Bachar, E., Validity of students’ ratings of clinical instructors. Medical Education 15:6 (1981), 373–376.
6. Bendig, A.W., Student achievement in introductory psychology and student ratings of the competence and empathy of their instructors. The Journal of Psychology: Interdisciplinary and Applied 36 (1953), 427–433, 10.1080/00223980.1953.9712910.
7. Bendig, A.W., The relation of level of course achievement to students’ instructor and course ratings in introductory psychology. Educational and Psychological Measurement 13 (1953), 437–448, 10.1177/001316445301300307.
8. Benton, S.L., Cashin, W.E., Student ratings of teaching: A summary of research and literature. 2012, IDEA Center Kansas State University, 1–19.
9. Benton, S.E., Scott, O., A Comparison of the Criterion Validity of Two Types of Student Response Inventories for Appraising Instruction. A paper presented at the Annual Meeting of the National Council on Measurement in Education, San Francisco, California, 1976.
10. Bolton, B., Bonge, D., Marr, J., Ratings of instruction, examination performance, and subsequent enrollment in psychology courses. Teaching of Psychology 6:2 (1979), 82–85, 10.1207/s15328023top0602_6.
11. Borg, W.R., Hamilton, E.R., Comparison between a performance test and criteria of instructor effectiveness. Psychological Reports 2 (1956), 111–116, 10.2466/PR0.2.3.
12. Braskamp, L.A., Caulley, D., Costin, F., Student ratings and instructor self-ratings and their relationship to student achievement. American Educational Research Journal 16:3 (1979), 295–306, 10.2307/1162782.
13. Bryson, R., Teacher evaluations and student learning: A reexamination. Journal of Educational Research 68:1 (1974), 12–14.
14. Capozza, D.R., Student evaluations, grades, and learning in economics. Western Economic Journal, 11(1), 1973, 127.
15. Centra, J.A., Student ratings of instruction and their relationship to student learning. American Educational Research Journal 14:1 (1977), 17–24, 10.2307/1162516.
16. Chase, C.I., Keene, J.M., Jr., Indiana Univ., B.B. of E. S. and T. (1979). Validity of Student Ratings of Faculty. Indiana Studies in Higher Education, Number Forty.
17. Clayson, D.E., Student evaluations of teaching: Are they related to what students learn? A meta-analysis and review of the literature. Journal of Marketing Education 31:1 (2009), 16–30, 10.1177/0273475308324086.
18. Cohen, S.H., Berger, W.G., Dimensions of students’ ratings of college instructors underlying subsequent achievement on course examinations. Proceedings of the annual convention of the american psychological association 5 (1970), 605–606 [Pt. 2].
19. Cohen, P.A., A meta-analysis of the relationship between student ratings of instruction and student achievement. Unpublished doctoral disertation, 1980, The University of Michigan.
20. Cohen, P.A., Student ratings of instruction and student achievement: A meta-analysis of multisection validity studies. Review of Educational Research 51:3 (1981), 281–309, 10.2307/1170209.
21. Cohen, P.A., Validity of student-ratings in psychology courses − A research synthesis. Teaching of Psychology 9:2 (1982), 78–82, 10.1207/s15328023top0902_3.
22. Cohen, P.A., A selective review of the validity of student-Ratings of teaching − comment. Journal of Higher Education 54:4 (1983), 448–458, 10.2307/1981907.
23. Costin, F., Do Student Ratings of College Teachers Predict Student Achievement?. Teaching of Psychology 5 (1978), 86–88.
24. Crooks, T., Smock, H., Student ratings of instructors related to student achievement. 1974, University of Illinois: Urbana, III: Office of Instructional Resources.
25. Davis, B.G., Tools for teaching. 2nd ed., 2009, Jossey-Bass, San Francisco, CA.
26. Dowell, D.A., Neal, J.A., A selective review of the validity of student ratings of teachings. The Journal of Higher Education 53:1 (1982), 51–62, 10.2307/1981538.
27. Doyle, K.O., Crichton, L.I., Student, peer, and self evaluations of college instructors. Journal of Educational Psychology 70:5 (1978), 815–826, 10.1037/0022-0663.70.5.815.
28. Doyle, K.O., Whitely, S.E., Student ratings as criteria for effective teaching. American Educational Research Journal 11:3 (1974), 259–274, 10.2307/1162199.
29. Drysdale M.J. (2010, January 1). Psychometric Properties of Postsecondary Students’ Course Evaluations. ProQuest LLC.
30. Duval, S., Tweedie, R., Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 56:2 (2000), 455–463, 10.1111/j.0006-341x.2000.00455.
31. Elliott, D.N., 1950. Characteristics and relationships of various criteria of college and university teaching. 70, 5–61.
32. Ellis, N.R., Rickard, H.C., Evaluating the teaching of introductory psychology. Teaching of Psychology, 4(3), 1977, 128.
33. Endo, G.T., Della-Piana, G., A validation study of course evaluation ratings. Improving College and University Teaching 24 (1976), 84–86.
34. Feldman, K.A., The association between student ratings of specific instructional dimensions and student achievement: Refining and extending the synthesis of data from multisection validity studies. Research in Higher Education 30:6 (1989), 583–645.
35. Fenderson, B.A., Damjanov, I., Robeson, M.R., Rubin, E., Relationship of students’ perceptions of faculty to scholastic achievement: Are popular instructors better educators?. Human Pathology 28:5 (1997), 522–525, 10.1016/S0046-8177(97)90072-1.
36. Frey, P.W., Leonard, D.W., Beatty, W.W., Student ratings of instruction: Validation research. American Educational Research Journal 12:4 (1975), 435–447.
37. Frey, P.W., Student ratings of teaching: validity of several rating factors. Science 182 (1973), 83–85.
38. Frey, P.W., Validity of student instructional ratings: Does timing matter?. Journal of Higher Education 47 (1976), 327–336.
39. Galbraith, C.S., Merrill, G.B., Predicting student achievement in university-Level business and economics classes: Peer observation of classroom instruction and student ratings of teaching effectiveness. College Teaching 60:2 (2012), 48–55, 10.1080/87567555.2011.627896.
40. Galbraith, C.S., Merrill, G.B., Kline, D.M., Are Student Evaluations of Teaching Effectiveness Valid for Measuring Student Learning Outcomes in Business Related Classes? A Neural Network and Bayesian Analyses. Research in Higher Education 53 (2012), 353–374, 10.1007/s11162-011-9229-0.
41. Gessner, P.K., Evaluation of instruction. Science 180:4086 (1973), 566–570, 10.2307/1736154.
42. Greenwood, G.E., Hazelton, A., Smith, A.B., Ware, W.B., A study of the validity of four types of student ratings of college teaching assessed on a criterion of student achievement gains. Research in Higher Education 5:2 (1976), 171–178, 10.1007/BF00992010.
43. Grush, J.E., Costin, F., The student as consumer of the teaching process. American Educational Research Journal 12:1 (1975), 55–66, 10.2307/1162580.
44. Hoffman, R.G., Variables affecting university student ratings of instructor behavior. American Educational Research Journal 15:2 (1978), 287–299, 10.2307/1162467.
45. Johnson, V.E., Grade inflation: A crisis in college education. 2003, Springer, New York.
46. Koon, J., Murray, H., Using multiple outcomes to validate student-ratings of overall teacher-effectiveness. Journal of Higher Education 66:1 (1995), 61–81, 10.2307/2943951.
47. Kruger, J., Dunning, D., Unskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology 77:6 (1999), 1121–1134, 10.1037/0022-3514.77.6.1121.
48. Marsh, H.W., Overall, J.U., Validity of students’ evaluations of teaching effectiveness: Cognitive and affective criteria. Journal of Educational Psychology 72:4 (1980), 468–475, 10.1037/0022-0663.72.4.468.
49. Marsh, H.W., Fleiner, H., Thomas, C.S., Validity and usefulness of student evaluations of instructional quality. Journal of Educational Psychology 67:6 (1975), 833–839, 10.1037/0022-0663.67.6.833.
50. Marsh, H.W., Students’ evaluations of university teaching: Dimensionality, reliability, validity, potential biases and usefulness. Perry, R.P., Smart, J.C., (eds.) The scholarship of teaching and learning in higher education: An evidence-based perspective, 2007, Springer, Netherlands, 319–383 Retrieved from: http://link.springer.com.ezproxy.library.ubc.ca/chapter/10.1007/1-4020-5742-3_9.
51. McCallum, L.W., A meta-analysis of course evaluation data and its use in the tenure decision. Research in Higher Education 21:2 (1984), 150–158, 10.1007/BF00975102.
52. McKeachie, W.J., et al. Student ratings of teacher effectiveness: Validity studies. American Educational Research Journal 8 (1971), 435–445.
53. McKeachie, W.J., Lin, Y., Mendelson, C.N., A small study assessing teacher effectiveness: Does learning last?. Contemporary Educational Psychology 3:4 (1978), 352–357, 10.1016/0361-476X(78)90037-1.
54. Mintzes, J.J., Field test and validation of a teaching evaluation instrument: The student opinion survey of teaching. A report submitted to the senate committee for teaching and learning, faculty senate, 1977, University of Windsor, Windsor, Ontario.
55. Moreno, S.G., Sutton, A.J., Ades, A., Stanley, T.D., Abrams, K.R., Peters, J.L., et al. Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation study. BMC Medical Research Methodology, 9, 2009, 2, 10.1186/1471-2288-9-2.
56. Morgan, W.D., Vasché, J.D., An educational production function approach to teaching effectiveness and evaluation. The Journal of Economic Education 9:2 (1978), 123–126.
57. Morsh, J.E., Burgess, G.G., Smith, P.N., Student achievement as a measure of instructor effectiveness. Journal of Educational Psychology 47:2 (1956), 79–88, 10.1037/h0043123.
58. Murdock R.P., Utah Univ., S.L. C. (1969). The effect of student ratings of their instructor on the student's achievement and rating. Final report. Utah University, Salt Lake City.
59. Murray, H.G., Low-inference classroom teaching behaviors and student ratings of college teaching effectiveness. Journal of Educational Psychology 75:1 (1983), 138–149, 10.1037/0022-0663.75.1.138.
60. Murray, H.G., Student evaluation of teaching: Has it made a difference?. Presented at the annual meeting of the society for teaching and learning in higher education, Charlottetown, PEI Canada, 2005 Retrieved from: https://www.stlhe.ca/wp-content/uploads/2011/07/Student-Evaluation-of-Teaching1.pdf.
61. Orpen, C., Student evaluation of lecturers as an indicator of instructional quality: A validity study. Journal of Educational Research 74:1 (1980), 5–7.
62. Ory, J.C., Ryan, K., How do student ratings measure up to a new validity framework?. New Directions for Institutional Research, 2001(109), 2001, 27.
63. Palmer, J., Leniency, learning, and evaluations. Journal of Educational Psychology 70:5 (1978), 855–863.
64. Prosser, M., Trigwell, K., Student evaluations of teaching and courses: Student learning approaches and outcomes as criteria of validity. Contemporary Educational Psychology 16:3 (1991), 293–301, 10.1016/0361-476X(91)90029-K.
65. Rücker, G., Schwarzer, G., Carpenter, J.R., Binder, H., Schumacher, M., Treatment-effect estimates adjusted for small-study effects via a limit meta-analysis. Biostatistics (Oxford England) 12:1 (2011), 122–142, 10.1093/biostatistics/kxq046.
66. Rankin, E.F., Greenmun, R., Tracy, R.J., Factors related to student evaluations of a college reading course. Journal of Reading 9:1 (1965), 10–15.
67. Remmers, H.H., Martin, F.D., Elliot, D.N., Are students’ ratings of instructors related to their grades?. Purdue University Studies in Higher Education 66 (1949), 17–26.
68. Reynolds, D.V., Hansvick, C., Graduate instructors who grade higher receive lower evaluations by students. In annual meeting of the american psychological association, Toronto, Ontario, 1978.
69. Rodin, M., Rodin, B., Student evaluations of teachers. Science 177 (1972), 1164–1166.
70. Rothstein, H.R., Sutton, A.J., Borenstein, M., Publication bias in meta-analysis: Prevention assessment and adjustments. 2006, John Wiley & Sons.
71. Rubenstein, J., Mitchell, H., Feeling free, student involvement, and appreciation. In proceedings of the 78th annual convention of the american psychological association 5 (1970), 623–624.
72. Sheets, D.F., Topping, E.E., Hoftyzer, J., The relationship of student evaluations of faculty to student performance on a common final examination in the principles of economics courses. Journal of Economics (MVEA) 21:2 (1995), 55–64 http://www.cba.uni.edu/economics/joe.htm.
73. Shmanske, S., On the measurement of teacher effectiveness. The Journal of Economic Education 19:4 (1988), 307–314, 10.1080/00220485.1988.10845278.
74. Solomon, D., Rosenberg, L., Bezdek, W.E., Teacher behavior and student learning. Journal of Educational Psychology 55:1 (1964), 23–30, 10.1037/h0040516.
75. Soper, J.C., Soft research on a hard subject: Student evaluations reconsidered. Journal of Economic Education 5 (1973), 22–26.
76. Sorge, D.H., Kline, C.E., Verbal behavior of college instructors and attendant effect upon student attitudes and achievement. College Student Journal 7:4 (1973), 24–29.
77. Spencer, R.E., Dick, W., Course evaluations questionnaire: Manual of interpretation (Research report No. 200)., 1965, Urbana, I11: Office of Instructional Resources, University of Illinois.
78. Spooren, P., Brockx, B., Mortelmans, D., On the validity of student evaluation of teaching the state of the art. Review of Educational Research 83:4 (2013), 598–642, 10.3102/0034654313496870.
79. Stanley, T.D., Doucouliagos, H., Meta-regression approximations to reduce publication selection bias. Research Synthesis Methods 5:1 (2014), 60–78, 10.1002/jrsm.1095.
80. Sullivan, A.M., Skanes, G.R., Validity of student evaluation of teaching and the characteristics of successful instructors. Journal of Educational Psychology 66:4 (1974), 584–590, 10.1037/h0036929.
81. Turner, R.L., Thompson, R.P., 1973. Relationships Between College Student Ratings of Instructors and Residual Learning. A paper presented at the annual meetings of the Amerian Educational Research Association, Chicago, Illinois..
82. Uttl, B., Eche, A., Fast, O., Mathison, B., Valladares Montemayor, H., Raab, V., Student evaluation of instruction/teaching (SEI/SET) review. 2012, Mount Royal Faculty Association, Calgary, AB, Canada Retrieved from: http://mrfa.net/files/MRFA_SEI_Review_v6. pdf.
83. Uttl, B., White, C.A., Morin, A., The numbers tell it all: Students don't like numbers. Plos One, 8(12), 2013, 10.1371/journal.pone.0083443 e83443-e83443.
84. Vul, E., Harris, C., Winkielman, P., Pashler, H., Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspectives on Psychological Science 4:3 (2009), 274–290, 10.1111/j.1745-6924.2009.01125.x.
85. Wachtel, H.K., Student evaluation of college teaching effectiveness: A brief review. Assessment & Evaluation in Higher Education, 23(2), 1998, 191.
86. Weinberg, B.A., Hashimoto, M., Fleisher, B.M., Evaluating teaching in higher education. Journal of Economic Education 40:3 (2009), 227–261.
87. Wherry, R.J., Control of bias in ratings (PRS reports, 914, 915, 919, 920, 921)., 1952, Deparment of Army, The Adjutant General's Office, Washington, D.C.
88. Whitely, S.E., Doyle, K.O. Jr., Validity and generalizability of student ratings from between-classes and within-class data. Journal of Educational Psychology 71 (1979), 117–124.
89. Wiviott, S.P., Pollard, D.S., Background, section, and student evaluation variables as predictors of achievement in a college course. The Journal of Educational Research 68:1 (1974), 36–42, 10.1080/00220671.1974.10884698.
90. Yunker, P.J., Yunker, J.A., Are student evaluations of teaching valid? Evidence from an analytical business core course. Journal of Education for Business 78:6 (2003), 313–317.