Revisiting law school mismatch: A comment on Barnes (2007, 2011)

Northwestern University Law Review

Volumn 105, Issue 2, 2011, Pages 813-828

  Williams, Doug ^a   Sander, Richard ^b   Luppino, Marc ^c   Bolus, Roger ^b  

^a University of the South   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c FEDERAL TRADE COMMISSION   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 81255209506     PISSN: 00293571     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (65)

1. Katherine Y. Barnes, Is Affirmative Action Responsible for the Achievement Gap Between Black and White Law Students?, 101 NW. U. L. REV. 1759 (2007).
2. Richard H. Sander, A Systemic Analysis of Affirmative Action in American Law Schools, 57 STAN. L. REV. 367 (2004).
3. The mismatch hypothesis implicitly posits that instructors aim the level of instruction at the middle student. Doug Williams, Does Affirmative Action Create Educational Mismatches in Law Schools? (Apr. 13, 2009) (paper presented at the 2010 ALEA meetings), http://econ.duke.edu/~hf14/ERID/Williams.pdf (reviewing the empirical literature that has appeared since Sander's original article and presenting several alternate empirical tests of law school mismatch, which consistently find strong support for mismatch effects on minority bar passage rates).
4. Barnes advanced a model in which outcomes for law students, such as graduating from law school and passing the bar exam, are modeled as functions of a few basic characteristics of each student: the student's credentials at the start of law school, such as LSAT score and undergraduate grades; the tier of law school the student attends; and the student's race.
5. See Barnes, supra note 1, at 1801. Barnes followed a novel approach by using a very flexible functional form that allows these basic characteristics to interact with one another and to take on higher-order powers; this imposes very few a priori assumptions on how the data and the model fit together.
6. Id.
7. See, e.g, Richard Lempert with William Kidder, Testimony for the U.S. Commission on Civil Rights, in U.S. COMMON ON CIVIL RIGHTS, AFFIRMATIVE ACTION IN AMERICAN LAW SCHOOLS 51, 66 (2007), available at http://www.usccr.gov/pubs/AALSreport.pdf.
8. Katherine Y. Barnes, Is Affirmative Action Responsible for the Achievement Gap Between Black and White Law Students? A Correction, A Lesson, and an Update, 105 NW. U. L. REV. 791 (2011).
9. Id. at 808.
10. If only 78.2% of previous blacks get admitted to law school without affirmative action, then blacks would have to graduate and pass the bar at a rate higher by a factor of more than 1.27 (100 ÷78.2) to have no change in the number of black lawyers. On the 21.8% figure, see infra note 33.
11. See infra Part IV.
12. See Barnes, supra note 1, at 1781 tbl.2A, 1788 tbl.5.
13. Id.
14. The principal data source used by Barnes, and by us in this Response, is the Bar Passage Study (BPS), a unique panel study sponsored by the Law School Admissions Council in the 1990s that tracked graduation and bar outcomes for some 27,000 students.
15. LINDA F. WIGHTMAN, LAW SCH. ADMISSION COUNCIL, LSAC NATIONAL LONGITUDINAL BAR PASSAGE STUDY 6 (1998), available at http://www. unc.edu/edp/pdf/NLBPS.pdf. Analysis of the entire BPS database reveals that blacks who graduate from "heavily minority" law schools have a first-time bar passage rate of 57%. Data Sets for Northwestern University Law Review 105:2, NW. U. L. REV. (Oct. 1, 2011)
16. http://www.law.northwestern.edu/ lawreview/issues/105.2.data.html (calculations by Doug Williams et al.). This is only slightly lower than the first-time bar passage rate of blacks at all other law schools in the BPS (62%)
17. id.
18. even though blacks at the heavily minority schools have, on average, significantly lower entering credentials than blacks at other law schools. Barnes, supra note 1, at 1781 tbl.2A, 1788 tbl.5.
19. When Bolus and Williams compared their codes, they found that different choices about how to code idiosyncratic observations (e.g., students who showed no graduation result but took a bar exam) accounted for the slight differences in their results. We present here the results based on what we think are the soundest coding choices. See infra Tables 1 & 2.
20. See infra Part II.
21. For example, consider Table 2A in Barnes's original essay, where she reports the likelihood of passing the bar for students credentialed at the fifth and tenth percentiles is lowered by 50.1 and 46.7 percentage points for students attending historically black schools compared to mid-range schools
22. See Barnes, supra note 1, at 1781 tbl.2A. These are the sort of large and unexpected effects that should invite the scrutiny of peer reviewers. Journals edited by academics generally make it a high priority to quickly and thoroughly provide notice when fundamental problems are discovered in a published article. They do so because they see the journal's credibility and reliability as its stock in trade, and they are perhaps better able to do so than student-edited journals because they have greater continuity of editorial control and have an immediate grasp of the significance of problems once those are pointed out to them.
23. See infra Tables 1 & 2. These tables show the results from Barnes's original tables
24. See Barnes, supra note 1, at 1777 tbl.1A, 1781 tbl.2A, 1787 tbl.4, 1788 tbl.5.
25. See, e.g., infra Table 2."Success" and "failure" in this context mean that someone does or does not achieve a particular outcome (or one of a set of outcomes) of interest.
26. The definition that Barnes uses in her revision appearing in this issue is the same variable that Yoon and Rothstein define as "Bar Passage If Attempt" in their 2006 working paper
27. Jesse Rothstein & Albert Yoon, Mismatch in Law School (Ctr. for Econ. Policy Studies, Working Paper No. 123, 2006), http://www.princeton.edu/ceps/workingpapers/123rothstein.pdf, whereas the definition that Barnes uses in her SSRN paper is the same variable that Yoon and Rothstein define as "Bar Passage."
28. See Katherine Y. Barnes, Is Affirmative Action Responsible for the Achievement Gap between Black and White Law Students? A Correction, a Lesson, and an Update (Aug. 12, 2011), http://ssrn.com/abstract=1908530
29. Neither of these definitions is an "error"; they are alternative reasonable definitions of what it means to successfully complete law school. Barnes states that we criticize her for omitting group b. Barnes, supra note 7, at 799 n.23. This is wrong. Our criticism is that she changed her method by including group a in her revision. Our criticism is not that this change in method from her original paper is an error but rather that it has the effect of masking the effect of mismatch on bar passage rates for actual test takers.
30. In any case, we think there is much to be said, in analyses of post-graduate performance, for counting as "failures" only those people who actually attempt the bar exam, and doing so does not substantially change the results in our model.
31. When we first set about replicating Barnes's analysis, we wrote to her to ask how her bar passage variable was constructed. We then (in February 2009) set up a conference call with Barnes, Williams, Bolus, and Sander and discussed with Barnes how she had defined her bar passage variable. She told us then that the bar passage rates listed in her Tables 2 and 5 were determined by dividing the number of persons who ever passed any bar exam by the number of persons who graduated from law school. A few days later, we sent her output and code showing our results based on this definition, and how the results differed from her original results. We had a number of conversations with Barnes after that point, and she never suggested that our analysis was faulty. Only twenty-two months later did Barnes dispute this definition.
32. Barnes, supra note 1, at 1774. Much of the law school mismatch literature, including Sander's original paper, combined LSAT scores and UGPA into a weighted "index" of credentials and then used that as an explanatory variable.
33. See Sander, supra note 2, at 393. As recently as February 2011, Barnes conceded in a draft that her original paper did not use such as index, but she now insists that she did.
34. Katherine Y. Barnes, Is Affirmative Action Responsible for the Achievement Gap Between Black and White Law Students? A Correction, a Lesson, and an Update 4 n.16 (Feb. 11, 2011) (earlier draft of Barnes's piece) (on file with the Northwestern University Law Review).
35. Barnes, supra note 7, at 796.
36. Barnes's fifth percentile student is close to our tenth percentile student in credentials.
37. The estimate of this "lawyer" rate is, conceptually, the product of the other two percentages: the graduation rate and bar passage rate. But because we used Barnes's modeling process to estimate lawyer production rates directly, the lawyer rates we report do not exactly match the product of the other two rates.
38. Because Barnes's model suffers from selection bias, see infra Part IV, we do not think it should be used to evaluate the point at which mismatch effects disappear. The important point is that the size of mismatch effects steadily declines as student credentials go up, and this is consistent with the mismatch hypothesis.
39. Barnes, supra note 1, at 1807.
40. Even in the absence of mismatch, Barnes's model shows blacks having worse outcomes than whites. This is probably because the selection bias in Barnes's model is worse for whites than it is for blacks. Whites with low credentials who are admitted to law school are more likely to have unusually positive unobservable characteristics than blacks with similar observable credentials because those characteristics would have accounted for their admission. Blacks with low credentials, in contrast, are likely to be admitted to relatively elite schools in pursuit of affirmative action goals. Low-credential whites will therefore appear to outperform low-credential blacks. Sander's original paper controlled for other student characteristics, and his regressions predicting graduation and bar outcomes showed no negative "black" effect. Sander, supra note 2, at 439, 444.
41. Barnes, supra note 7, at 809 tbl.7.
42. For her simulation, Barnes takes the pool of BPS underrepresented minority students (URMs) and drops the 14% with the lowest credentials. In doing this, she claims she is simply following Sander, who in turn used
43. Linda Wightman's 2003 finding that, without affirmative action, 14% fewer blacks would matriculate.
44. See Sander, supra note 2, at 473
45. citing Linda F. Wightman, The Consequences of Race-Blindness: Revisiting Prediction Models with Current Law School Data, 53 J. LEGAL EDUC. 229+243 tbl.7 (2003)). In applying the 14% figure to URMs as a whole
46. Barnes actually drops more than 400 blacks from the BPS data, which is over 21% of the blacks in the sample. Barnes's simulation means that 1141 out of the 1871 blacks in the BPS dataset become lawyers with affirmative action whereas 1134 out of 1463 blacks become lawyers without affirmative action. The overall number of black students who fail to become lawyers thus falls from 730+329.
47. Barnes, supra note 1, at 1807.
48. See Jesse Rothstein & Albert H. Yoon, Affirmative Action in Law School Admissions: What Do Racial Preferences Do?, 57 U. CHI. L. REV. 649 (2008);
49. Sander, supra note 2, at 418-25
50. Williams, supra note 3, at 11.
51. See Williams, supra note 3, at Part VII.
52. For example, Barnes writes that 15.1% of BPS respondents reported earnings of $40,000 or more.
53. Barnes, supra note 1, at 1775. The correct number is 35%.
54. See id.
55. Barnes notes the potential for nonresponse bias but does not characterize how such bias is likely to affect her results. Specifically, because students who experience worse outcomes are less likely to respond, a fact reflected in the data, nonresponse bias is likely to skew the results in Barnes's job market analysis upward.
56. There are other difficulties in using the BPS jobs data: for example, respondents are often reporting their earnings before they have learned their bar results. If mismatch causes low-credential graduates from elite schools to disproportionately fail the bar exam, these students will disproportionately see a reduction in their earnings. There are also high nonresponse rates on the jobs questions, which should be carefully analyzed when using the data. Barnes's revision suggests that correcting her original errors in her jobs analysis leaves her findings unchanged, but her analysis still ignores many problems with the data.
57. See Barnes, supra note 7.
58. Id. at 812.
59. supra note 1, at 1807.
60. This table presents three versions of two different outcomes, derived from data on students in the LSAC Bar Passage Study, which tracked roughly 27,000 students who entered law school in 1991 and the five following years. The first outcome is law school graduation, reflected in the first three columns. The second outcome is passing a bar examination, reflected in the last three columns. Within each outcome, we report the results from Barnes's 2007 essay
61. see supra note 1, Barnes's revised results
62. see Barnes, supra note 7, and our own estimates as calculated by
63. Roger Bolus and Doug Williams. Each number represents success rates in a particular outcome (e.g., a 76.9% success rate in passing the bar). The numbers are derived by first using the BPS to estimate logit models for graduation and bar passage and then using these logit model results to estimate outcomes for students who are at a particular point in the credential distribution and others who are attending a particular cohort of schools. As we discuss in the text, supra Part II, the revised Barnes results still differ from our own estimates because, we believe, Barnes is not faithfully replicating her original model.
64. This table is identical to Table 1 except that it reports estimates for blacks rather than whites. See supra note 42 for details.
65. The top number in each cell is the predicted proportion of students who will have the stated outcome, controlling for the student's race, credential level, and law school tier. Thus, in the cell at the upper left, Barnes's model estimates that 90.7% of whites attending historically black law schools who have credentials that place them at the fifth percentile of credentials in the BPS sample will graduate. Below each proportion is the standard error, estimated by standard bootstrap methods. For each estimate, we report whether the difference between a given tier and the top 30 tier outcome is statistically significant and mark positive results according to the legend at the bottom of the table. For definitions of the three outcomes, see supra Part II.