The relevance of results generated by human indexing and computer algorithms: A study of west's headnotes and key numbers and LexisNexis's headnotes and topics

Law Library Journal

Volumn 102, Issue 2, 2010, Pages 221-249

  Mart, Susan Nevelow ^a  

^a UC Hastings College of the Law   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 77953418002     PISSN: 00239283     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Review

References (67)

1. See infra 115-116 for a discussion of how the headnotes on the two systems are generated.
2. Stuart Allen Sutton, Managing Legal Information: A Model of Institutional Memory Based on User Cognitive Maps 10 (Mar. 11,1991) (unpublished Ph.D. dissertation, University of California at Berkeley) (on file with Hastings College of the Law Library). Sutton also calls this "level-one research." Id.
3. Patrick Meyer, Law Firm Legal Research Requirements for New Attorneys, 101 Law Libr. J. 297, 307, 314, 2009 Law Libr. J. 17 §§,51-52. Doing "cost-effective research" is frequently ranked number one, but in working with a roundtable of firm and academic law librarians in San Francisco to prepare a program on effective short training sessions for new summer associates, I learned that cost-effective usually means first knowing that you should begin your research in secondary sources. Cost-effective search techniques on LexisNexis and Westlaw are a separate category of cost-effectiveness.
4. Judith Lihosit, Research in the Wild: CALR and the Role of Informal Apprenticeship in Attorney Training. 101 Law Libr. J. 157,170,2009 Law Libr. J. 10 § 35. ("All the attorneys I interviewed stated that if they were researching an unfamiliar area of law, they would start. with an appropriate secondary source, such as a practice guide, a legal treatise, or an encyclopedia, or a document repository, in order to become familiar with what one termed the 'legal landscape'." (footnote omitted)).
5. Robert C. Berring, Legal Research for the 21st Century (2000) (set of 5 videotapes) (Tape 5: Legal Research on the Internet and Research Strategies).
6. See Sutton, supra note 2, at 11-12. This is the part of the research process where the researcher hopes to find cases whose facts are "sufficiendy similar to those of the client that a reasonable argument can be made that the court should similarly rule." Id. at 12.
7. This is often called "citation pearl growing" in the information science literature. Donald T. Hawkins & Robert Wagers, Online Bibliographic Search Strategy Development, Online, May, 1982, at 12,13. Because of the early development of citation systems in the retrieval of case law, this method is very highly developed for legal information.
8. See Eugene Garfield, Citation Indexes for Science: A New Dimension in Documentation Through Association of Ideas, 122 Science 108, 108 (1955).
9. Douglas K. Norman, The Art of Selecting Cases to Cite, 63 Tex. B.J. 340,340 (2000).
10. E-mail from Mark Cygnet, Thomson/West, to author (Mar. 11,2009) (on file with author).
11. E-mail from LexisNexis U.S. Legal Markets Corporate Communication to author (Mar. 3, 2010) (on file with author) (estimating that LexisNexis contains 9.7 million cases).
12. In the mid-nineteenth century, the concept of citators was a response to the growing body of case law. See Pattie Ogden, Mastering The Lawless Science of Our Law: A Story of Legal Citation Indexes, 85 Law Libr. J. 1, 12 (1993).
13. The earliest online legal databases, dating from the 1950s and 1960s, were similarly attempts to address the problems caused by the growing number of cases. See Jon Bing & Trygve Harvold, Legal Decisions and Information Systems 60 (1977).
14. This section is based in part on Susan Nevelow Mart, Cite Checking: A Brave New World, Legal Info. Alert, Apr. 2006, at 1.
15. The "editor's own language" does sometimes parallel the exact language of the court {see, e.g., Tarasoff v. Regents of the Univ. of Calif., 551 P.2d 334,335, 342 (Cal. 1976) (headnote 4)), but West editors are free to, and do, summarize legal concepts in their own words.
16. Morris L. Cohen, Robert C. Berring, & Kent C. Olson, How to Find the Law 84 (9th ed. 1989). Although the creation of headnotes is human-generated at this time, the process of matching the headnote of a target case with the language of citing cases is performed by computer algorithms.
17. Elizabeth M. McKenzie, Comparing KeyCite with Shepard's Online, Legal Reference Services Q., 1999, no.3, at 85,90,97 (stating that KeyCite uses automation or "computer programs" for headnote assignment).
18. Cohen, Berring & Olson, supra note 14, at 84.
19. Id.
20. Deborah Hackerson, Legal Research: A Guide to Online Tutorials for First Year Law Students, Legal Reference Services Q., 2006, no.2, at 153,159.
21. E-mail from Debra Myers, Account Executive, LexisNexis to author (Apr. 3, 2006) (on file with author). When this e-mail was sent, "Search by Topic or Headnote" was called Search Advisor. See Lexis Search Advisor Evolves; Search Across Multiple Content Types Simultaneously, LexisNexis InfoPro, Oct. 2007, available at http://law.lexisnexis.conT/infopro/Keeping-Current/ LexisNexis-Information-Professional-Update-Newslet//AVhats-New/ Lexis-Search-Advisor-Evolves-Search-Across-Multiple-Content-Types- Simultaiieously/archivelO-2007.
22. E-mail from Debra Myers, supra note 18.
23. Michael Ginsborg, Does Search Advisor Depend Too Litde on Classifiers, and Too Much on Algorithms, for Headnote Classification? Evidence on the Perils of Search Advisor's Automation and the Virtue of Thomson-West Classifiers 12-13 (Aug. 2007), http://ssrn.corn/abstract-1345465 (quoting System and Method for Classifying Legal Concepts Using Legal Topic Scheme, U.S. Patent 6,502,081 (issued December 31, 2002)).
24. Ginsborg identifies the other LexisNexis patents as: "Computer-based System for Classifying Documents Into a Hierarchy and Linking the Classifications to the Hierarchy," U.S. Patent 5,794,236 (issued August 11,1998). (describing algorithmic classification of cases by means of matching case citations with topically-linked citations);
25. and "Landmark Case Identification System and Method," U.S. Patent Application 20060041608 (February 23, 2006) (Search Advisor samples used to illustrate a system of cases arranged by topic in digest format, with a user option to rank cases by frequency of citation). Id. at 13 n. 14. LexisNexis did not acknowledge or deny that it uses these patents to create Lexis Topics. Id at 13.
26. Id. at 16-19.
27. Id. at 19. This is an extreme simplification of a process described in great detail by Ginsborg.
28. Id. See id. at 51-53.
29. Stuart A. Sutton, The Role of Attorney Mental Models of Law in Case Relevance Determinations: An Exploratory Analysis, 45 J. Am. Soc'y Info. Sci. 186, 196-97 (1994). Looking at the results of a broad search, such as all cases found in the chosen American Law Reports annotation or in the chosen key number, may return objectively relevant results, but return as well many cases that, although perhaps relevant "topically," will not help an attorney fill in a mental map of an area of law necessitated by a particular client's situation. Id. "[R]elevance is a complex notion of how a particular document relates to a given line of inquiry."
30. Scott F. Burson, A Reconstruction q/Thamus-Comments on the Evaluation of Legal Information Retrieval Systems, 79 Law Libr. J. 133,141 (1987).
31. Stephen M. Marx, Citation Networks in the Law, 10 Jurimetrics J. 121, 122-23 (1970). Marx thought computer-generated systems based on key words alone would suffer from similar defects and proposed a context-based and citation-based retrieval system he characterized as a form of "exhaustive shepardization" assisted by a technique for "automatically isolating the factual content of a case" Id. at 125,137. The article was written, of course, before Shepard's went online or KeyCite was launched.
32. Daniel P. Dabney, The Curse ofThamus: An Analysis of Full-Text Legal Document Retrieval, 78 Law Libr. J. 5,13-14 (1986).
33. Id at 14.
34. Whether finding all of the relevant documents is necessarily the holy grail of online research is an entirely different question. See, e.g., Burson, supra note 24, at 136-39. In fact, whether a researcher wants high recall or high precision may vary from search to search: for comprehensive research, a searcher may need every relevant document, while for a time-driven request, the user may want the system to return the documents that are most highly relevant first.
35. Anton Geist, Using Citation Analysis Techniques for Computer-Assisted Legal Research in Continental Jurisdictions 19 (May 1, 2009) (unpublished LL.M. thesis, University of Edinburgh), available at http://ssrn.com/abstract-1397674. One problem for inexperienced researchers seems to be a belief in the high quality of their research results. Novice searchers believe they have actually seen all the relevant documents and that the documents seen are the most relevant documents. Any legal research teacher can confirm this phenomenon.
36. Paul Callister's definitions of recall and precision are simple and understandable: "Essentially, there are two conflicting standards for measuring the success of your research. Precision measures how many documents were on point within your search results. In contrast, recall gauges the relevant documents in your results compared to what you could have found." Paul D. Callister, Working the Problem, Ill. B.J., Jan. 2003, at 43,44.
37. Geist, supra note 28, at 15,29.
38. Id. at 15.
39. Jo McDermott, Another Analysis of Full-Text Legal Document Retrieval, 78 Law Libr. J. 337, 343 (1986);
40. Craig E. Runde & William H. Lindberg, The Curse of Thamus: A Response, 78 Law Libr. J. 345,345-346(1986).
41. Kevin Gerson, Evaluating Legal Information Retrieval Systems: How Do the Ranked-Retrieval Systems ofWESTLAWand LEXIS Measure Up?, Legal Reference Services q., 1999, no.4, at 53.
42. Dean Mason, Legal Information Retrieval Study-Lexis Professional and Westlaw UK, 6 Legal Info. Mgmt. 246, 248 (2006);
43. see also Sutton, supra note 24, at 187 (discussing the concept of relevance in actual case law retrieval: "a relevant case is one that plays some cognitive role in the structuring of a legal argument or the framing of legal advice.").
44. Mason, supra note 34, at 248. Mason used research requests from lawyers as queries for his study, and examined the first ten results of fifty separate searches. Id. at 247.
45. Id. at 248 tbl.2. See also Burson, supra note 24, at 135 (discussing factors affecting precision and recall in legal information retrieval systems).
46. Callister, supra note 29, at 44 (2003). As far back as 1994, West's own study of the relationship between precision and recall in the Federal Supplement database showed that as precision went up, recall went down at almost the identical rate. Id. at 44 & fig.2.
47. "More Like This Headnote" is discussed in detail infra 129.
48. The relevance checking for this article was performed between Mar. 8,2009 and May 8,2009 for the first five citations; the rest of the cases were checked between July 5,2009 and Aug. 24,2009.
49. As always, determinations of relevance are subjective. As Scott Burson has noted, "we can profoundly disagree on the relevance of a particular document." Burson, supra note 24, at 141.1 reviewed these cases as I did when I was an attorney, looking for factors that would make me put it in the "this looks like it might be useful" file.
50. Because I was comparing results from headnotes (West Key Numbers) and full-text results (Lexis Topics), I tried to choose keyword searches that would not be too limiting when only searching headnotes and would also not be too broad when searching the full text of cases on LexisNexis (it is not possible to search only headnotes on LexisNexis). When it was unclear from the headnote whether or not the case was relevant, I would review the full text of the case, and use the summary of the case and, if necessary, "Locate in Result," to determine relevance.
51. In the West digest results, there may be multiple entries for one case, if there is more than one headnote in the case with the same key number. Where multiple entries were retrieved, each entry was counted separately.
52. See infra figure 8.
53. When I limited my results of my accidental search by "Show Headnotes Only," the set of 678 results at least contained my headnote. At this point I could have used the Focus bar, or clicked on an "In Depth Discussion" link. When I chose Focus, my results were reduced to 292 cases, but the "In Depth Discussion" link was no longer available. The first ten results were not relevant to my research. The second choice was to first choose the "In Depth Discussion" link, which the mouse-over tag informs me will "retrieve up to 15 discussion cases on this topic." With those fifteen cases, when I added my search string ((police or law or peace) 12 officer), I retrieved two cases, neither of which was relevant.
54. It appears from the results that LexisNexis automation ranks these result sets first according to the occurrence of terms matching the limiting search string that occur anywhere in the text of retrieved cases, and then according to headnotes that match the topic. See infra 128 (discussing the location of relevant cases found using the West key number search in LexisNexis results sets). Thus, results are not limited to cases that include the relevant headnotes.
55. For LexisNexis results sets that returned hundreds of documents, if the Westlaw results set was less than ten, I limited my review to the first ten results; for Westlaw results sets of more than ten, I reviewed the same number of LexisNexis results. The limitations of time prevented me from reviewing the entire results sets. A researcher with a real-world problem would have to determine the number of results to review based on client need.
56. LexisNexis, Research Help: More Like This, http://web.lexis.com/help/ research/gh-browsing.asp#MoreLikeThis (last visited Feb. 14,2010).
57. I have my preferences set in LexisNexis as follows: the default for natural language results in LexisNexis is set to 250 results and the results are returned in order of relevance.
58. This search was run originally in May 2009 and rerun on July 5, 2009, and there were a few differences in the results. The second original results set was 250, rather than 249. Also, the "sort by closest match" button was new; that button had not been there in May. "Sort by closest match" confirms the algorithmic nature of the search and the ordering of the results; it does not explain why the results are so different from the Lexis Topic search. After searching using my Focus terms, I reviewed all sixty-four cases I found in July, and compared them to the ten I reviewed in May; of the 64, 9 were relevant (14%). The additional relevant cases were numbers 18, 21, 35,39,45, and 46 in the list ordered by "closest match."
59. Relevance was again determined by whether cases discussed a situation in which police have been "sharply attacked."
60. To make the results as comparable as possible, and because a default of 250 was simply too many cases to review, where there were up to ten Westlaw results, I reviewed the first ten results on "More Like This Headnote" and the Lexis Topic; if there were more than ten results on Wesdaw, I reviewed the same number of cases in "More Like This Headnote" and the Lexis Topic.
61. Of course, the headnotes themselves are created differently, with West editors reframing the legal issues in each case in their own language and LexisNexis taking the headnotes directly from the language of the case (see supra 115-6 ), but because the headnotes being compared for this test were chosen for their similarity, the method of generation should not really matter.
62. McKenzie, supra note 14, at 85,86-87.
63. E-mail from Debra Myers, supra note 18.
64. The citation checking for this article was performed between Mar. 8,2009 and May 8,2009 for the first five pairs of headnotes; the rest of the research was done between July 5,2009 and Aug. 24,2009.
65. Generally speaking, law school is a jurisdiction-free zone. The use of multi-jurisdictional, edited cases in law students' case books masks the importance of jurisdiction, making a discussion of the importance of jurisdiction (and level of court) to a determination of the relevance of a case for articulating a legal argument a key lesson for new lawyers.
66. In addition to missing relevant cases, reviewing irrelevant cases wastes time. In this example, if you used Shepard's, 44% of the cases you looked at would be irrelevant (4/9). Using KeyCite, 23% of the cases you looked at would be irrelevant (3/13).
67. While these unique cases did not appear in the headnote number- and jurisdiction-limited citator results of the other system, checking the unlimited citatory results was beyond the scope of this article.