Evaluation of information retrieval for E-discovery

Artificial Intelligence and Law

Volumn 18, Issue 4, 2010, Pages 347-386

  Oard, Douglas W ^a   Baron, Jason R ^b   Hedin, Bruce ^c   Lewis, David D ^d   Tomlinson, Stephen ^e  

^a UNIVERSITY OF MARYLAND   (United States)

^b United States National Archives and Records Administration   (United States)

^c David D Lewis Consulting ^*  (United States)

^d David D Lewis Consulting   (United States)

^e Open Text Corporation Canada   (Canada)

Author keywords

E discovery; Evaluation; Information retrieval; Interactive search

Indexed keywords

E-DISCOVERY; ELECTRONIC DISCOVERIES; EVALUATION; EVALUATION METHOD; INFORMATION-RETRIEVAL TECHNOLOGY; INTERACTIVE SEARCH; LARGE DATASETS; NEW APPROACHES; RECALL AND PRECISION; RESEARCH COMMUNITIES; RETRIEVAL EVALUATION; RETRIEVAL METHODS; SEARCH TECHNOLOGY; STATISTICAL INFORMATION; TECHNICAL COMMUNITY; TEXT RETRIEVAL CONFERENCES;

INFORMATION RETRIEVAL;

EID: 79951773558     PISSN: 09248463     EISSN: 15728382     Source Type: Journal    
DOI: 10.1007/s10506-010-9093-9     Document Type: Article

References (66)

1. American Institute of Certified Public Accountants (2009) Statement on auditing standards no. 70: Service organizations. SAS 70
2. Aslam JA, Pavlu V, Yilmaz E (2006) A statistical method for system evaluation using incomplete judgments. In: Proceedings of the 29th annual international ACM SIGIR conference on research and development in information retrieval, pp 541-548
3. Bales S, Wang P (2006) Consolidating user relevance criteria: A meta-ethnography of empirical studies. In: Proceedings of the 42nd annual meeting of the American society for information science and technology
4. Baron JR (2005) Toward a federal benchmarking standard for evaluation of information retrieval products used in E-discovery. Sedona Conf J 6:237-246
5. Baron JR (2007) The TREC legal track: Origins and reports from the first year. Sedona Conf J 8:237-246
6. Baron JR (2008) Towards a new jurisprudence of information retrieval: What constitutes a 'reasonable' search for digital evidence when using keywords?. Digit Evid Electronic Signature Law Rev 5:173-178
7. Baron JR (2009) E-discovery and the problem of asymmetric knowledge. Mercer Law Rev 60:863
8. Baron JR, Thompson P (2007) The search problem posed by large heterogeneous data sets in litigation: Possible future approaches to research. In: Proceedings of the 11th international conference on artificial intelligence and law, pp 141-147 (Pubitemid 47281672)
9. Baron JR, Lewis DD, Oard DW (2007) TREC-2006 Legal Track overview. In: The fifteenth text retrieval conference proceedings (TREC 2006), pp 79-98
10. Bauer RS, Brassil D, Hogan C, Taranto G, Brown JS (2009) Impedance matching of humans and machines in high-Q information retrieval systems. In: Proceedings of the IEEE international conference on systems, man and cybernetics, pp 97-101
11. Blair D (2006) Wittgenstein, language and information: Back to the rough ground. Springer, New York
12. Blair D, Maron ME (1985) An evaluation of retrieval effectiveness for a full-text document-retrieval system. Commun ACM 28(3):289-299 (Pubitemid 15599385)
13. Brin S, Page L (1998) The anatomy of a large-scale hypertextual Web search engine. Computer Netw ISDN Syst 30(1-7):107-117
14. Buckley C, Voorhees EM (2004) Retrieval evaluation with incomplete information. In: Proceedings of the 27th annual international conference on research and development in information retrieval, pp 25-32
15. Buckley C, Voorhees EM (2005) Retrieval system evaluation. In: Voorhees EM, Harman DK (eds) TREC: Experiment and evaluation in information retrieval. MIT Press, Cambridge, pp 53-75
16. Buckley C, Dimmick D, Soboroff I, Voorhees E (2006) Bias and the limits of pooling. In: Proceedings of the 29th annual international ACM SIGIR conference on research and development in information retrieval, pp 619-620
17. Büttcher S, Clarke CLA, Yeung PCK, Soboroff I (2007) Reliable information retrieval evaluation with incomplete and biased judgements. In: Proceedings of the 30th annual international ACM SIGIR conference on research and development in information retrieval, pp 63-70 (Pubitemid 350164945)
18. Carmel D, Yom-Tov E, Darlow A, Pelleg D (2006) What makes a query difficult? In: Proceedings of the 29th annual international ACM SIGIR conference on research and development in information retrieval, pp 390-397 (Pubitemid 350164806)
19. Carterette B, Pavlu V, Kanoulas E, Aslam JA, Allan J (2008) Evaluation over thousands of queries. In: Proceedings of the 31st annual international ACM SIGIR conference on research and development in information retrieval, pp 651-658
20. Clarke C, Craswell N, Soboroff I (2005) The TREC terabyte retrieval track. SIGIR Forum 39(1):25-25
21. Cleverdon C (1967) The Cranfield tests on index language devices. Aslib Proceed 19(6):173-194
22. Cormack GV, Lynam TR (2006) TREC 2005 spam track overview. In: The fourteenth text retrieval conference (TREC 2005), pp 91-108
23. Cormack GV, Palmer CR, Clarke CLA (1998) Efficient construction of large test collections. In: Proceedings of the 21st annual international ACM SIGIR conference on research and development in information retrieval, pp 282-289
24. Dumais ST, Belkin NJ (2005) The TREC interactive tracks: Putting the user into search. In: Voorhees EM, Harman DK (eds) TREC: Experiment and evaluation in information retrieval. MIT Press, Cambridge, pp 123-152
25. Fox EA (1983) Characterization of two new experimental collections in computer and information science containing textual and bibliographic concepts. Tech. rep. TR83-561, Cornell University
26. Harman DK (2005) The TREC test collections. In: Voorhees EM, Harman DK (eds) TREC: Experiment and evaluation in information retrieval. MIT Press, Cambridge, pp 21-52
27. Hedin B, Oard DW (2009) Replication and automation of expert judgments: Information engineering in legal E-discovery. In: SMC'09: Proceedings of the 2009 IEEE international conference on systems, man and cybernetics, pp 102-107
28. Hedin B, Tomlinson S, Baron JR, Oard DW (2010) Overview of the TREC 2009 Legal Track. In: The eighteenth text retrieval conference (TREC 2009)
29. Ingwersen P (1992) Information retrieval interaction. Taylor Graham, London
30. Ingwersen P, Järvelin K (2005) The turn: Integration of information seeking and retrieval in context. Springer
31. International Organization for Standards (2005) Quality management systems-fundamentals and vocabulary. ISO 9000:2005
32. Jensen JH (2000) Special issues involving electronic discovery. Kansas J Law Public Policy 9:425
33. Kando N, Mitamura T, Sakai T (2008) Introduction to the NTCIR-6 special issue. ACM Trans Asian Lang Inform Process 7(2):1-3
34. Kazai G, Lalmas M, Fuhr N, Gövert N (2004) A report on the first year of the initiative for the evaluation of XML retrieval (INEX'02). J Am Soc Inform Sci Technol 55(6):551-556
35. Lewis D, Agam G, Argamon S, Frieder O, Grossman D, Heard J (2006) Building a test collection for complex document information processing. In: Proceedings of the 29th annual international ACM SIGIR conference on research and development in information retrieval, pp 665-666
36. Lewis DD (1996) The TREC-4 filtering track. In: The fourth text retrieval conference (TREC-4), pp 165-180
37. Lynam TR, Cormack GV (2009) Multitext legal experiments at TREC 2008. In: The sixteenth text retrieval conference (TREC 2008)
38. Majumder P, Mitra M, Pal D, Bandyopadhyay A, Maiti S, Mitra S, Sen A, Pal S (2008) Text collections for FIRE. In: Proceedings of the 31st annual international ACM SIGIR conference on research and development in information retrieval, pp 699-700
39. Moffat A, Zobel J (2008) Rank-biased precision for measurement of retrieval effectiveness. ACM Trans Inf Syst 27(1)
40. Oard DW, Hedin B, Tomlinson S, Baron JR (2009) Overview of the TREC 2008 Legal Track. In: The seventeenth text retrieval conference (TREC 2008)
41. Oot P, Kershaw A, Roitblat HL (2010) Mandating reasonableness in a reasonable inquiry. Denver Univ Law Rev 87:533
42. Paul GL, Baron JR (2007) Information inflation: Can the legal system adapt? Richmond J Law Technol 13(3)
43. PCI Security Standards Council (2009) Payment card industry (PCI) data security standard: Requirements and security assessment procedures, version 1.2.1. http://www.pcisecuritystandards.org
44. Peters C, Braschler M (2001) European research letter: Cross-language system evaluation: The CLEF campaigns. J Am Soc Inf Sci Technol 52(12):1067-1072 (Pubitemid 32936226)
45. Roitblat HL, Kershaw A, Oot P (2010) Document categorization in legal electronic discovery: Computer classification vs. manual review. J Am Soc Inf Sci Technol 61(1):70-80
46. Sakai T, Kando N (2008) On information retrieval metrics designed for evaluation with incomplete relevance assessments. Inf Retr 11(5):447-470
47. Sanderson M, Joho H (2004) Forming test collections with no system pooling. In: Proceedings of the 27th annual international ACM SIGIR conference on research and development in information retrieval, pp 33-40
48. Sanderson M, Zobel J (2005) Information retrieval system evaluation: Effort, sensitivity, and reliability. In: Proceedings of the 28th annual international ACM SIGIR conference on research and development in information retrieval, pp 162-169
49. Schmidt H, Butter K, Rider C (2002) Building digital tobacco document libraries at the University of California, San Francisco Library/Center for Knowledge Management. D-Lib Mag 8(2)
50. Singhal A, Salton G, Buckley C (1995) Length normalization in degraded text collections. In: Proceedings of fifth annual symposium on document analysis and information retrieval, pp 15-17
51. Soboroff I (2007) A comparison of pooled and sampled relevance judgments. In: Proceedings of the 30th annual international ACM SIGIR conference on research and development in information retrieval, pp 785-786 (Pubitemid 350165069)
52. Solomon RD, Baron JR (2009) Bake offs, demos & kicking the tires: A practical litigator's brief guide to evaluating early case assessment software & search & review tools. http://www.kslaw.com/ Library/publication/ BakeOffs-Solomon.pdf
53. Spärck Jones K, van Rijsbergen CJ (1975) Report on the need for and provision of an ideal information retrieval test collection. Tech. Rep. 5266, Computer Laboratory, University of Cambridge, Cambridge (UK)
54. Taghva K, Borsack J, Condit A (1996) Effects of OCR errors on ranking and feedback using the vector space model. Inf Process Manage 32(3):317-327 (Pubitemid 126359367)
55. The Sedona Conference (2007a) The Sedona Principles, second edition: Best practice recommendations and principles for addressing electronic document production. http://www.thesedonaconference.org
56. The Sedona Conference (2007b) The Sedona Conference best practices commentary on the use of search and information retrieval methods in E-discovery. The Sedona Conf J 8:189-223
57. The Sedona Conference (2009) The Sedona Conference commentary on achieving quality in the E-discovery process. The Sedona Conf J 10:299-329
58. Tomlinson S (2007) Experiments with the negotiated Boolean queries of the TREC 2006 legal discovery track. In: The fifteenth text retrieval conference (TREC 2006)
59. Tomlinson S, Oard DW, Baron JR, Thompson P (2008) Overview of the TREC 2007 Legal Track. In: The sixteenth text retrieval conference (TREC 2007)
60. Turpin A, Scholer F (2006) User performance versus precision measures for simple search tasks. In: Proceedings of the 29th annual international ACM SIGIR conference on research and development in information retrieval, pp 11-18
61. Voorhees EM, Garofolo JS (2005) Retrieving noisy text. In: Voorhees EM, Harman DK (eds) TREC: Experiment and evaluation in information retrieval. MIT Press, Cambridge, pp 183-197
62. Voorhees EM, Harman DK (2005) The text retrieval conference. In: Voorhees EM, Harman DK (eds) TREC: Experiment and evaluation in information retrieval. MIT Press, Cambridge, pp 3-19
63. Wayne CL (1998) Multilingual topic detection and tracking: Successful research enabled by corpora and evaluation. In: Proceedings of the first international conference on language resources and evaluation
64. Yilmaz E, Aslam JA (2006) Estimating average precision with incomplete and imperfect judgments. In: Proceedings of the 15th international conference on information and knowledge management (CIKM), pp 102-111 (Pubitemid 47203557)
65. Zhao FC, Oard DW, Baron JR (2009) Improving search effectiveness in the legal E-discovery process using relevance feedback. In: ICAIL 2009 DESI III Global E-Discovery/E-Disclosure Workshop. http://www.law.pitt.edu/DESI3- Workshop/DESI-III-papers.htm
66. Zobel J (1998) How reliable are the results of large-scale information retrieval experiments? In: Proceedings of the 21st annual international ACM SIGIR conference on research and development in information retrieval, pp 307-314