Developing an empirical model to test whether required writing exercises or other changes in large-section law class teaching methodologies result in improved exam performance

Journal of Legal Education

Volumn 57, Issue 2, 2007, Pages 195-204

  Curcio, Andrea A ^a   Jones, Gregory Todd ^a   Washington, Tanya M ^a  

^a GEORGIA STATE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 38349018478     PISSN: 00222208     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (27)

1. Steven Friedland, A Critical Inquiry Into the Traditional Uses of Law School Evaluation, 23 Pace L. Rev. 147, 188, 208, 209 (2002); Gregory S. Munro, Outcomes Assessment for Law Schools 74 (Spokane, Wash., 2000); Philip C. Kissam, Law School Examinations, 42 Vand. L. Rev. 433, 473 (1989); Roy Stuckey et al., Best Practices for Legal Education 255 (2007) (encouraging the use of formative assessments throughout the semester); Mary Beth Beazley, Better Writing, Better Thinking: Using Legal Writing Pedagogy in the "Casebook" Classroom (Without Grading Papers), 10 Legal Writing: J. Legal Writing Institute 23,72-74 (2004); Sophie M. Sparrow, Describing the Ball: Improve Teaching By Using Rubrics-Explicit Grading Criteria, 2004 Mich. St. L. Rev 1 (2004); Greg Sergienko, New Modes of Assessment, 38 San Diego L. Rev. 463, 482 (2001); Mary A. Lundeberg, Metacognitive Aspects of Reading Comprehension: Studying Understanding in Legal Case Analysis, 22 Reading Res. Q. 407, 428 (1987) (finding that explicitly teaching strategies for reading and understanding legal opinions significantly improved comprehension of novice law students).
2. Stuckey, Best Practices for Legal Education, supra note 1; see also the information garnered during the Carnegie Foundation-funded, extensive study of law school education, William M. Sullivan et al., Educating Lawyers: Preparation for the Profession of Law (San Francisco, 2007).
3. Our research indicates that in the last fifty years, there have only been a few empirical examinations of law school classroom teaching methodology. See, e.g., Edward L. Kimball and Larry C. Farmer, Comparative Results of Teaching Evidence Three Ways, 30 J. Legal. Educ. 196 (1979) (finding little difference in learning outcomes between the case method, problem method, or self-instruction method); see also Paul F. Teich, Research on American Law Teaching: Is There a Case Against the Case System?, 36 J. Legal Educ. 167 (1986) (describing nine studies of law teaching methods, most with undergraduate law classes, virtually all of which concluded that the only methodologies that had any impact on learning outcomes were computer assisted methodologies or those geared toward individualized instruction). There have been a few other articles about classroom innovations that discuss the professors' subjective belief that their innovations had a positive impact upon students' performance. See, e.g., John M. Burman, Out-Of-Class Assignments as a Method of Teaching and Evaluating Law Students, 42 J. Legal Educ. 447 (1992) (finding that out-of-class writing assignments resulted in his subjective perception that the students performed better on the final exam after doing the assignments); Stephen J. Shapiro, The Use and Effectiveness of Various Learning Materials in an Evidence Class, 46 J. Legal Educ. 101 (1996) (finding that between casebook reading, hornbook reading, and CALI exercises, only the CALI exercises seemed to have an effect on student exam performance). One recent empirical study measured a particular lawyering skill. See Karen Barton et al., Valuing What Clients Think: Standardized Clients and the Assessment of Communicative Competence, 13 Clinical L. Rev. 1 (2006) (providing an excellent empirical analysis of the viability of using a standardized client as a method of measuring professional competence in communicating with clients).
4. There were fifty-five students in the non-intervention class and eighty students in the intervention class. Approximately 90 percent of the students in each class were first-year full-time students. The remaining 10 percent of students in each class were part-time second-year students. The study's data only included first-year full-time students.
5. As the semester progressed the questions became more complex either in terms of factual or legal analysis or both. However, all questions were single-issue questions dealing with a particular procedural rule or concept.
6. Students received a model annotated answer for all five essay questions. For one of the questions, students also were given a grading rubric so that they could see that most points were allocated to factual analysis. Students received raw score points towards their final grade for simply timely turning in their essay answers. The professor also graded one essay answer with grades varying between four to eight raw score points. The graded essay contained individualized comments. Students received their graded answer before they were asked to complete their final paper.
7. One question tested relation back of complaint amendments and the other involved a claim preclusion issue. To ensure students had the same basic substantive understanding of the subject matter, the professors used the same course material when teaching the substantive areas and collaborated to ensure that they were teaching from the same perspective and with similar emphasis. Also, to ensure they did not inadvertently "teach to the test" they waited until the end of the semester before jointly drafting the exam questions. Each question was worth one-third of the exam's total raw score and had a suggested one-hour time allocation. Each question contained about one page of facts and the overall issue was easily identifiable because the professors sought to test students' ability to break a rule into its component parts and analyze and apply the facts to the rule's various elements-the skills that were the focus of the practice writing exercises. In addition to the two co-authored essay questions, each professor drafted her own short answer questions.
8. To ensure they were applying the rubric in a similar manner the professors met and graded the first twenty-five answers at the same time for each essay question. They also checked in with each other throughout the grading process. Although each professor graded every answer, each student received his or her grade from the student's own professor. Each professor curved her final grades based upon the raw score points of the students in her individual class.
9. See Andrea A. Curcio, Gregory Todd Jones, and Tanya M. Washington, Does Practice Make Perfect? An Empirical Examination of the Impact of Practice Essays on Exam Performances, 35. Fla. St. U. L. Rev. (forthcoming 2008) (presenting more detailed findings from this study and exploring potential reasons for those findings).
10. The suggestion to employ the same rigor in an examination of our teaching as in our other areas of scholarship was first suggested by Ernest Boyer. Ernest L. Boyer, Scholarship Reconsidered: The Priorities of the Professoriate (Stanford, Cal., 1990) (urging universities to conceptualize research and scholarship more broadly to encompass scholarship of teaching and learning). The idea has been built upon by others. See, e.g., Charles E. Glassick, Mary Taylor Huber, and Gene I. Maeroff, Scholarship Assessed: Evaluation of the Professoriate (Stanford, Cal., 1997) (proposing guidelines for those seeking to engage in the scholarship of teaching and learning and for those seeking to assess the quality of scholarly work examining teaching and learning).
11. An independent sample t-test compares the mean scores of two groups on a given variable. The p value states the likelihood that we would find a difference in means as large as that found purely by chance. A low p value indicates that the difference is unlikely due to chance.
12. The traditional data used to predict first-year law school grades is UGPA and LSAT score. The validity of these predictors has been studied by many professors. See, e.g., David A. Thomas, Predicting Law School Academic Performance from LSAT Scores and Undergraduate Grade Point Averages: A Comprehensive Study, 35 Ariz. St. L. J. 1007 (2003) (discussing the results of twenty-seven years of admissions data at BYU law school that examined the correlation between LSAT and UGPA and law student grades); Jeffrey S. Kinsler, The LSAT Myth, 20 St. Louis Univ. Pub. L. Rev 393 (2001) (finding that although LSAT is claimed to be a better predictor than UGPA, at Marquette the opposite was true-UGPA was a better predictor of law school performance than LSAT). Studies indicate that LSAT scores and UGPAs are far from perfect predictors of first-year exam performance. Abiel Wong, Note, "Boalt-ing" Opportunity?: Deconstructing Elite Norms in Law School Admissions, 6 Geo. J. on Poverty Law & Pol'y 199, 208 (1999) (noting that the LSAT's correlation coefficient with first-year grades ranges from .01 to .62, depending on the law school, with a median correlation of .41; when the LSAT is used in conjunction with UGPA, predictive validity ranges from .11 to .68, with a median correlation coefficient of .49); William C. Kidder, The Rise of Testocracy: An Essay on the LSAT, Conventional Wisdom, and the Dismantling of Diversity, 9 Tex. J. Women & L. 167, 187 (2000) (reporting that "the LSAT accounts for only 16% of the variance in first year grades among students enrolled in ABA law schools"); see also Jerry R. Parkinson, Admissions After Grutter, 35 Univ. Toledo L. Rev. 159, 163 (2003) (noting that the LSAC does not dispute that the LSAT actually only accurately predicts first-year performance for a relatively small percentage of first-year students, but instead the LSAC "simply-and correctly-points out that there is no other measurement that 'comes close to matching the predictive qualities of the LSAT'"). We agree with those who argue that predicting exam performance is not a valid method for predicting who will be a good lawyer. See, e.g., Ian Weinstein, Testing Multiple Intelligences: Comparing Evaluation by Simulation and Written Exam, 8 Clinical L. Rev. 247, 248-50 (2001) (noting that the LSAT has limited utility in law school grade prediction and that law school grades do not correlate to or predict success in the practice of law); Phoebe A. Haddon and Deborah W. Post, Misuse and Abuse of the LSAT: Making the Case for Alternative Evaluative Efforts and a Redefinition of Merit, 80 St. John's L. Rev. 41, 53-54 (2006) (arguing that the LSAT and first-year exams do not purport to measure anything close to the wide range of skills necessary for minimally competent lawyering). However, given that LSAT and UGPA are the traditional method of predicting performance on first-year law school exams, we used those measures as control variables in this study.
13. The intervention professor had been teaching for twelve years while the non-intervention professor had taught for four years. One attempt to control for this variable was that the professors used the same cases and problems to teach the substantive areas that would be tested and collaborated to make sure that they were teaching the material from a similar perspective and with similar emphasis on various substantive issues.
14. The statistical significance of these results was confirmed by an independent sample t-test. Non-Intervention Professor [NIP] Essay 1: p = .000, NIP Essay 2: p = .000, Intervention Professor [IP] Essay 1: p= .000, IP Essay 2: p = .000.
15. NIP Essay 1: p = .679, NIP Essay 2: p = .123, IP Essay 1: p = .750, IP Essay 2: p = .264.
16. We found similar results using UGPA as a variable, with a statistically significant correlation showing that students with above-the-median UGPA benefited the most from the writing interventions. (NIP Essay 1: p = .005, NIP Essay 2: p = .005, IP Essay 1: p = .000, IP Essay 2: p = .009.) However we found that students with below-the-median UGPA in the intervention class showed a statistically significant benefit from the interventions on the second essay question. (NIP Essay 1: p = .244, NIP Essay 2: p = .001, IP Essay 1: p = .058, IP Essay 2: p = .004.) Possible explanations for our findings include the students' different metacognitive skill levels and perhaps the test-taking speed variable of both the LSAT and in-class timed tests. See Curcio et al., Does Practice Make Perfect?, supra note 9, for further discussion of these issues.
17. Although our findings lead us to believe that the results are probably not attributable to differences in teaching experience or style, we think a better way to control for this variable would be to have the same professor teach two different sections of the same class, using the interventions in one class and not in the other. If this is not possible, another control might be to have both professors give identical short answer questions and to compare students' scores on those questions as a way to test whether students' scores in the essay portion were a function of teacher effectiveness.
18. See Renee Oliver and Robert L. Williams, Direct and Indirect Effects of Completion Versus Accuracy Contingencies on Practice-Exam and Actual-Exam Performance, 14 J. Behav. Educ. 141, 142 (2005) (citing to studies which indicate that practice exams which are similar to regular exams may improve students' exam performance).
19. See, e.g., John L. Nietfeld, Li Cao and Jason W. Osborne, Metacognitive Monitoring Accuracy and Student Performance in the Postsecondary Classroom, 74 J. Experimental Educ. 22 (2005).
20. See Linda Baker, Metacognition, Comprehension Monitoring and the Adult Reader, 1 Educ. Psychol. Rev. 9, 10 (1989) (noting that one metacognitive strategy that has led to higher test scores is students' verbalization of their predictions regarding potential test questions). Because the intervention class had five essay assignments along with model answers, the students with more highly honed metacognitive skills in the intervention class may have predicted it was unlikely that the areas covered by the writing exercises would be tested on the exam. Given that there is a limited universe of substantive areas in civil procedure that lend themselves to essay questions, students in the intervention class with more highly developed metacognitive skills may have figured out the remaining areas most likely to be tested via an essay question and focused their studies on those areas. Students in the non-intervention class, regardless of metacognitive skills, had no idea what was likely to be tested and thus did not have the benefit of focusing their study for essay questions on just a few substantive areas. In a future study, to eliminate this factor, the professors could tell students the subject matter that would be tested on the essay exam questions and test broad knowledge via short answer questions.
21. See William D. Henderson, The LSAT, Law School Exams, and Meritocracy: The Surprising and Undertheorized Role of Test-Taking Speed, 82 Tex. L. Rev. 975 (2004) (discussing the impact of the test-taking speed variable on LSAT scores and law school exams).
22. Oliver and Williams, Direct and Indirect Effects of Completion Versus Accuracy, supra note 18, at 142.
23. See generally Henderson, The LSAT, supra note 21 (arguing that take-home essay exams without the speediness component may decrease the correlation between first-year exam grades and LSAT scores).
24. The literature on metacognition suggests that students' ability to use practice exercises is enhanced by providing the students with specific strategies to analyze why they performed well or poorly on the practice exercises. See, e.g., Nietfeld et al., Metacognitive Monitoring Accuracy, supra note 19, at 22; see generally Chris Masui and Erik De Corte, Learning to Reflect and to Attribute Constructively as Basic Components of Self-Regulated Learning, 75 Brit. J. Educ. Psychol. 351-72 (2005) (discussing how instruction in the metacognitive techniques of reflection upon what helps one's learning and attributing learning outcomes to particular causes has a positive impact upon students' overall academic achievement).
25. We found no correlation between whether students were in the intervention or non-intervention section and their grades in other spring semester courses. In retrospect, this is not that surprising given the different types of final examinations and the different emphases placed on various skill sets. It would be interesting to see if incorporating specific instruction in metacognitive skills during the feedback sessions changed this outcome. Studies suggest that students who learn specific metacognitive skills that help them monitor and self-regulate how they learn perform better in all their academic courses. See generally Michael Hunter Schwartz, Teaching Law Students to Be Self-Regulated Learners, 2003 Mich. St. L. Rev. 447 (discussing numerous studies which have found that highly developed metacognitive skills lead to overall better academic performance). It would be interesting to see what would happen if writing interventions accompanied by feedback as described here occurred in multiple first-year substantive courses. We suspect that the more classes in which the skills are explicitly taught, the stronger the skill acquisition.
26. See Stuckey, Best Practices, supra note 1; Sullivan, Educating Lawyers, supra note 2.
27. Most of the other empirical work on practice tests has involved multiple choice, rather than essay questions. See, e.g., Margaret K. Snooks, Using Practice Tests on a Regular Basis to Improve Student Learning, New Directions for Teaching and Learning, Winter 2004, at 109, 110 (multiple choice practice tests); William R. Balch, Practice Versus Review Exams and Final Exam Performance, 25 Teaching of Psychol. 181 (1998) (multiple choice practice tests); John A. Gretes and Michael Green, Improving Undergraduate Learning with Computer-Assisted Assessment, 33 J. Computing in Educ. 46 (2000) (computerized multiple choice practice tests). But see Linda Bol and Douglas Hacker, A Comparison of The Effects of Practice Tests and Traditional Review on Performance Calibration, 2 J. Experimental Educ. 133 (2001) (assessing the impact of practice tests on multiple choice and short answer essay questions). We searched for but did not find other studies involving an empirical examination of the impact of practice essay assignments on exam performance. This may be because most of the work in this area is in undergraduate classes, many of which test via multiple choice exams. However, we realize that the failure to find studies dealing with empirical assessments of essay practice tests or assignments on exam performance may also be due to our limited familiarity with methods of searching the social science literature. We did find one article in which a legal educator discussed his subjective perception that out-of-class writing assignments led to better performance on final exam essay questions. See Burman, Out-of-Class Assignments, supra note 3.