Evaluating course evaluations: An empirical analysis of a quasi-experiment at the stanford law school, 2000-2007

Journal of Legal Education

Volumn 58, Issue 3, 2008, Pages 388-412

  Ho, Daniel E ^a   Shapiro, Timothy H ^a  

^a Department of Law ^*  (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 63149198293     PISSN: 00222208     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (127)

1. See Pamela J. Eckard, Faculty Evaluation: The Basis for Rewards in Higher Education, 57 Peabody J. Educ. 04, 96 (1980);
2. Timothy J. Gallagher, Embracing Student Evaluations of Teaching: A Case Study, 28 Teaching Sociology 140 (2000);
3. and Gordon E. Greenwood and Howard J. Ramagli, Jr., Alternatives to Student Ratings of College Teaching, 51 J. Higher Educ. 673, 674 (1980).
4. See Curt J. Dommeyer, et al., Attitudes of Business Faculty Towards Two Methods of Collecting Teaching Evaluations: Paper v. Online, 27 Assessment & Evaluation in Higher Educ. 455 (2002);
5. and Keith G. Lumsden, Summary of an Analysis of Student Evaluations of Faculty and Courses, 5 J. Econ. Educ. 54 (1973).
6. See Jennifer R. Kogan and Judy A. Shea, Course Evaluation in Medical Education, 23 Teaching and Tchr. Educ. 251 (2007);
7. and Anthony M. Paolo et al., Response Rate Comparisons of E-mail- and Mail-Distributed Student Evaluations, 12 Teaching & Learning in Med. 81 (2000).
8. See William Roth, Student Evaluation of Law Teaching: Observations, Resource Materials, and a Proposed Questionnaire 1 (Ass'n of Am. L. Sch. Washington D.C. 1983);
9. Richard L. Abel, Evaluating Evaluations: How Should Law Schools Judge Teaching, 40 J. Legal Educ. 407, 412 (1990);
10. and Judith D. Fischer, How to Improve Student Ratings in Legal Writing Courses: Views from the Trenches, 34 U. Balt. L. Rev. 199 (2004).
11. See Daniel Gordon, Does Law Teaching Have Meaning? Teaching Effectiveness, Gauging Alumni Competence, and the MacCrate Report, 25 Fordham Urb. L.J. 43 (1998);
12. and Christopher T. Husbands, Variation in Students' Evaluations of Teachers' Lecturing in Different Courses on Which They Lecture: A Study at the London School of Economics and Political Science, 33 Higher Educ. 51 (1997).
13. See William E. Becker and Michael Watts, How Departments of Economics Evaluate Teaching, 89 Am. Econ. Rev. 334 (1999);
14. and Greenwood and Ramagli, Alternatives to Student Ratings, supra note 1, at 675.
15. See Charles F. Eiszler, College Students' Evaluations of Teaching and Grade Inflation, 43 Res. Higher Educ. 483 (2002) (discussing correlation between teacher ratings and grade inflation);
16. Laura I. Langbein, The Validity of Student Evaluations of Teaching, 27 PS: Pol. Sci. & Pol. 545, 552 (1994) (examining whether ratings are responsive to teacher popularity);
17. Theodore C. Wagenaar, Student Evaluation of Teaching: Some Cautions and Suggestions, 23 Teaching Sociology 64 (1995) (highlighting grade inflation and popularity as major sources of confounding);
18. and Donald H. Naftulin et al., The Doctor Fox Lecture: A Paradigm of Educational Seduction, 48 J. Med. Educ. 630, 634 (1973) (finding that "students can be effectively 'seduced' into an illusion of having learned if the lecturer simulates a style of authority and wit").
19. See Abel, Evaluating Evaluations, supra note 4, at 452;
20. Becker and Watts, How Departments of Economics Evaluate Teaching, supra note 6;
21. Sylvia d'Appolonia and Philip C. Abrami, Navigating Student Ratings of Instruction, 52 Am. Psychol. 1198, 1205 (1997);
22. John A. Centra, Research Productivity and Teaching Effectiveness, 18 Res. Higher Educ. 379 (1983);
23. Eiszler, College Students' Evaluations, supra note 7, at 499;
24. Gallagher, Embracing Student Evaluations, supra note 1, at 142;
25. Langbein, The Validity of Student Evaluations, supra note 7, at 552;
26. and Herbert W. Marsh and Lawrence A. Roche, Making Students' Evaluation of Teaching Effectiveness Effective: The Critical Issues of Validity, Bias, and Utility, 52 Am. Psychol. 1187,1193 (1997).
27. See, e.g., Roth, Student Evaluation, supra note 4;
28. Arthur Best, Student Evaluations of Law Teaching Work Well: Strongly Agree, Agree, Neutral, Disagree, Strongly Disagree 3 (June, 2006) (presentation at AALS conference "New Ideas for Law School Teachers: Teaching Intentionally");
29. Gerald F. Hess, Heads and Hearts: The Teaching and Learning Environment in Law School, 52 J. Legal Educ. 75, 97 (2002);
30. Robert M. Lloyd, On Consumerism in Legal Education, 45 J. Legal Educ. 551 (1995);
31. Richard S. Markovitz, The Professional Assessment of Legal Academics: On the Shift from Evaluator Judgment to Market Evaluations, 48 J. Legal Educ. 417 (1998);
32. Eric W. Orts, Quality Circles in Law Teaching, 47 J. Legal Educ. 425 (1997);
33. and Victor G. Rosenblum et al., Report of the AALS Special Committee on Tenure and the Tenuring Process, 42 J. Legal Educ. 477, 495 (1992).
34. See Vic Barnett, Sample Survey: Principles and Methods 176 (3d ed., New York, 2002);
35. William A Belson, The Design and Understanding of Survey Questions 240 (Aldershot, Eng., 1981);
36. Roger Tourangeau, Lance J. Rips, and Kenneth Rasinski, The Psychology of Survey Response (Cambridge, 2000);
37. Graham Kalton, Martin Collins and Linsay Brock, Experiments in Wording Opinion Questions, 27 Applied Stat. 149-161 (1978);
38. Sam G. McFarland, Effects of Question Order on Survey Responses, 45 Pub. Opinion Q. 208 (1981);
39. and Mary K. Serdula et al., Effects of Question Order on Estimates of the Prevalence of Attempted Weight Loss, 142 Am. J. Epidemiology 64 (1995).
40. See Mick P. Couper, Web Surveys: A Review of Issues and Approaches, 64 Pub. Opinion Q. 464, 465 (2000) (discussing how web surveys simplify the delivery of multimedia content);
41. and Benjamin H. Layne, Joseph R. DeCristoforo, and Dixie McGinty, Electronic Versus Traditional Student Ratings of Instruction, 40 Res. Higher Educ. 221, 229 (1999) (finding that students completing electronic evaluations were more likely to provide written comments).
42. See Kim B. Sheehan and Sally J. McMillan, Response Variation in E-Mail Surveys: An Exploration, 39 J. Advertising Res., July-Aug. 1999, at 45, 46 (1999) (conducting meta-analysis and finding shorter response time for e-mail compared with conventional mail).
43. See Cihan Cobanoglu, Bill Warde, and Patrick J. Moreo, A Comparison of Mail, Fax, and Web-Based Survey Methods, 43 Int'l J. Market Res. 441, 449 (2001);
44. Couper, Web Surveys, supra note 11, at 464;
45. and Michael D. Kaplowitz, Timothy D. Hadlock, and Ralph Levine, A Comparison of Web and Mail Survey Response Rates, 68 Pub. Opinion Q. 94, 98 (2004).
46. See Mick P. Couper, Johnny Blair, and Timothy Triplett, A Comparison of Mail and E-Mail for Survey of Employees in Federal Statistical Agencies, 15 J. Official Stat. 39, 46 (1999);
47. and Sheehan and McMillan, Response Variation in E-Mail Surveys, supra note 12, at 46.
48. Regarding online teaching evaluations in particular, see Curt J. Dommeyer et al., Gathering Faculty Teaching Evaluations By In-Class and Online Surveys: Their Effects on Response Rates and Evaluations, 29 Assessment & Evaluation in Higher Educ. 611, 618 (2004);
49. and Layne et al., Electronic versus Traditional Student Ratings, supra note 11, at 226 (60 percent response in-class vs. 48.7 percent online).
50. See Miguel A. Vallejo et al., Psychological Assessment via the Internet: A Reliability and Validity Study of Online (vs. Paper-and-Pencil) Versions of the General Health Questionnair-28 (GHQ-28) and the Symptoms Check-List-90-Revised (SCL-90-R), 9 J. Med. Internet Res. 2 (2007) (finding that "paper and pencil scores were higher than online ones" for standardized health questionnaires, and that a large bank of data will be necessary to anchor online responses to normative questions in the future).
51. See also Mei Alonzo and Milam Aiken, Flaming in Electronic Communication, 36 Decision Support Sys. 205 (2004);
52. Sara Keisler, Jane Siegel, and Timothy W. McGuire, Social Psychological Aspects of Computer-Mediated Communication, 39 Am. Psychologist 1123, 1129 (1984);
53. and John Suler, The Online Disinhibition Effect, 7 CyberPsychol. & Behav. 321 (2004).
54. See Stephen J. Sills and Chunyan Song, Innovations in Survey Research: An Application of Web-Based Surveys, 20 Soc. Sci. Computer Rev. 22, 26 (2002);
55. and Bruce Ravelli, Anonymous Online Teaching Assessments: Preliminary Findings 7 (June 14, 2000) (unpublished manuscript, at www.eric.ed.gov, #ED445069) ("[S]tudents expressed the belief that if they were content with their teacher's performance, there was no reason to complete the survey").
56. See Eiszler, College Students' Evaluations, supra note 7 (documenting evaluations' relationship to grade inflation);
57. Christopher J. Fries and R. James McNich, Signed Versus Unsigned Student Evaluations of Teaching: A Comparison, 31 Teaching Soc. 333, 341 (2003) (documenting effects of anonymity in course evaluations);
58. and Stephen Shmanske, On the Measurement of Teacher Effectiveness, 19 J. Econ. Educ. 307 (1988) (documenting evaluations' relationship to student performance).
59. See, e.g., Kenneth A. Feldman, College Students' Views on Male and Female College Teachers: Part II-Evidence from Students' Evaluations of Their Classroom Teachers, 34 Res. Higher Educ. 151 (1993);
60. Joan M. Krauskopf, Touching the Elephant: Perceptions of Gender Issues in Nine Law Schools, 44 J. Legal Educ. 311, 312 (1994);
61. Langbein, The Validity of Student Evaluations, supra note 7, at 552;
62. and Deborah L. Rhode, Taking Stock: Women of All Colors in Legal Education, 53 J. Legal Educ. 475, 480 (2003).
63. See, e.g., Abel, Evaluating Evaluations, supra note 4, at 407.
64. See, e.g., Centra, Research Productivity, supra note 8;
65. and James Lindgren and Allison Nagelberg, Are Scholars Better Teachers? 73 Chi.-Kent L. Rev. 823 (1998).
66. See Robert M. Groves et al., Survey Methodology 167 (New York, 2004);
67. Robert M. Groves et al., Survey Nonresponse (New York, 2002);
68. Sharon L. Lohr, Sampling: Design and Analysis 225 (Pacific Grove, Cal., 1999).
69. The focus on a single question of overall teaching effectiveness appears common across institutions. See, e.g., Roth, Student Evaluation, supra note 4, at II-3 ("for administrative purposes, the most useful item is a composite (or 'global') question pertaining to over-all teaching quality");
70. and Gallagher, Embracing Student Evaluations, supra note 1, at 142 ("Departments...tend to give greater weight to the global items in assessing teaching quality").
71. See generally Rensis Likert, A Technique for the Measurement of Attitudes, 140 Archives Psychol. 55 (1932).
72. Roth, Student Evaluation, supra note 4, at B-8 (reproducing seventy law schools' teaching evaluation forms, forty-two of which use the "poor" to "excellent" scale, thirteen of which use the "disagree" to "agree" scale, and two of which use the response categories "somewhat agree" and "somewhat disagree").
73. For empirical studies on the effects of response categories, see Alan J. Klokars and Midori Yamagishi, The Influence of Labels and Positions in Rating Scales, 25 J. Educ. Measurement 85 (1988);
74. Tony C.M. Lam and Alan J. Klokars, Anchor Point Effects on the Equivalence of Questionnaire Items, 19 J. Educ. Measurement 317 (1982);
75. and Tony C.M. Lam and Joseph J. Stevens, Effects of Content Polarization, Item Wording, and Rating Scale Width on Rating Response, 7 Applied Measurement in Educ. 141 (1994).
76. On incentivizing response, see Roth, Student Evaluation, supra note 4, at II-2;
77. M. Berlin et al., An Experiment in Monetary Incentives, in Proceeding of the Survey Research Methods Section of the American Statistical. Association 393 (Alexandria, Va., 1992);
78. and Dommeyer, Gathering Faculty Teaching Evaluations, supra note 14, at 619 (suggesting that early grade feedback significantly increases response rate compared with a control group).
79. See Belson, The Design and Understanding of Survey Questions, supra note 10;
80. and Norman Schwartz et al., Rating Scales: Numeric Values May Change the Meaning of Scale Labels, 55 Pub. Opinion Q. 570 (1991).
81. See William H. Angoff, Summary and Derivation of Equating Methods Used at ETS, in Test Equating 55 (Paul W. Holland and Donald B. Rubin eds., New York, 1982).
82. See Michael J. Kolen and Robert L. Brennan, Test Equating, Scaling, and Linking 15 (2nd ed., New York, 2004);
83. and Carl N. Morris, On the Foundations of Test Equating, in Test Equating 171 (Paul W. Holland and Donald B. Rubin eds., New York, 1982).
84. For a creative approach to creating scale linkages to remedy cultural incomparability, see Gary King, et al., Enhancing the Validity and Cross-Cultural Comparability of Measurement in Survey Research, 98 Am. Pol. Sci. Rev. 191 (2004).
85. See Kolen and Brennan, Test Equating, supra note 29, at 103;
86. Howard Wainer, et al., Computerized Adaptive Testing: A Primer 12 (Hillsdale, N.J., 1990);
87. and Nancy S. Petersen, Michael J. Kolen, and H.D. Hoover, et al., Scaling, Norming, and Equating, in Educational Measurement 221 (Robert B. Linn ed., 3d ed., New York, 1989).
88. See generally Donald B. Rubin, Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies, 66 J. Educ. Psychol. 688 (1974).
89. From the perspective of gathering maximum information, such a scale may be more useful, as it is likely to discriminate more between courses and instructors.
90. See Fries and McNich, Signed versus Unsigned Student Evaluations, supra note 17 (finding changes in evaluation response under perception of anonymity);
91. and Layne, Electronic Versus Traditional Student Ratings, supra note 11, at 229 ("most students felt that the traditional method would have afforded them a higher degree of anonymity than the electronic method, particularly since it had been necessary for them to use their student identification numbers to log in to the system").
92. See supra note 15.
93. See Couper, Web Surveys, supra note 11, at 464;
94. Kaplowitz, Hadlock, and Levine, A Comparison of Web and Mail Survey Response Rates, supra note 13, at 94;
95. and Sills and Song, Innovations in Survey Research, supra note 16, at 22.
96. See Barnett, Sample Survey, supra note 10; Belson, The Design and Understanding of Survey Questions, supra note 10;
97. Tourangeau et al., The Psychology of Survey Responses, supra note 10;
98. and Schwartz, Rating Scales, supra note 27.
99. See Dommeyer, Gathering Faculty Teaching Evaluations, supra note 14, at 618;
100. and Sheehan and McMillan, Response Variation in E-mail Surveys, supra note 12, at 46.
101. See Couper, Web Surveys, supra note 11, at 465;
102. Kaplowitz et al., A Comparison of Web and Mail Survey Response Rates, supra note 13, at 94;
103. Layne et al., Electronic Versus Traditional Student Ratings of Instruction, supra note 11, at 229;
104. and Sills and Song, Innovations in Survey Research, supra note 16, at 22.
105. The data includes information on courses and instructors for which student evaluations were solicited; while this is appropriate for assessing the impact of the new evaluations, one should be cautious about inferring too much about broader trends as the data was not validated against registrar and employment records.
106. See L. F. Jameson Boex, Attributes of Effective Economics Instructors, 31 J. Econ. Educ. 211, 219 (2000);
107. Albert L. Danielsen and Rudolph A. White, Some Evidence on the Variables Associated with Student Evaluations of Teachers, 7 J. Econ. Educ. 117, 118 (1976);
108. Herbert W. Marsh, Jesse U. Overall, and Stephen P. Kesler, Class Size, Students' Evaluations, and Instructional Effectiveness, 16 Am. Educ. Res. J. 57, 61 (1979);
109. and Kenneth Wood, Arnold S. Linsky, and Murray A. Straus, Class Size and Student Evaluation of Faculty, 45 J. Higher Educ. 524, 528 (1974).
110. See Trevor J. Hastie, Robert Tibshirani, and Jerome Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction 171-72 (New York, 2001);
111. and William S. Cleveland, Eric Grosse, and William M. Shyu, Local Regression Models, in Statistical Models in S 309 (John M. Chambers and Trevor J. Hastie eds., Pacific Grove, Cal., 1992).
112. See Daniel E. Ho, et al., Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference, 15 Pol. Analysis 199 (2007);
113. and Sander Greenland, James M. Robins, and Judea Pearl, Confounding and Collapsibility in Causal Inference, 14 Stat. Sci. 29 (1999).
114. Pertaining to class size specifically, see also sources cited in supra note 40.
115. See Ho, et al., Matching as Nonparametric Preprocessing, supra note 42;
116. Lee Epstein, et al., The Supreme Court During Crisis: How War Affects only Non-War Cases, 80 N.Y.U. L. Rev. 1 (2005);
117. Daniel E. Ho, Why Affirmative Action Does Not Cause Black Students to Fail the Bar, 114 Yale L.J. 1997 (2005).
118. See Andrew Gelman and Jennifer Hill, Data Analysis Using Regression and Multilevel/ Hierarchical Models (New York, 2007).
119. See Andrew Gelman et al., Bayesian Data Analysis (2d ed., Boca Raton, Fla., 2003);
120. Andrew Gelman, Exploratory Data Analysis for Complex Models (with Discussion), 13 J. Computational & Graphical Stat. 755 (2004);
121. and Andrew Gelman, Xiao-Li Meng, and Hal Stern, Posterior Predictive Assessment of Model Fitness via Realized Discrepancies (with Discussion), 6 Stat. Sinica 733 (1996).
122. See Charles F. Manski, Identification Problems in the Social Sciences 21 (Cambridge, Mass., 1996);
123. and Joel L. Horowitz and Charles F. Manski, Censoring of Outcomes and Regressors Due to Survey Nonresponse: Identification and Estimation Using Weights and Imputations, 84 J. Econometrics 38 (1998).
124. Gelman and Hill, Data Analysis, supra note 44.
125. R Core Development Team, R: A Language and Environment for Statistical Computing (Vienna, 2007) (available at 〈http://www.R-project.org〉(last visited Jan. 9, 2009));
126. Sibylle Sturtz, Uwe Ligges, and Andrew Gelman, R2WinBUGS: A Package for Running WinBUGS from R, 12 J. Stat. Software (3) 1 (2005);
127. and David J. Lunn et al., WinBUGS - A Bayesian Modelling Framework: Concepts, Structure and Extensibility, 10 Stat. & Computing 325 (2000).