Assessor disagreement and text classifier accuracy

SIGIR 2013 - Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2013, Pages 929-932

  Webber, William ^a   Pickens, Jeremy ^b  

^a UNIVERSITY OF MARYLAND   (United States)

^b Catalyst Repository Systems   (United States)

Author keywords

Assessor disagreement; Evaluation; Text classification

Indexed keywords

ASSESSOR DISAGREEMENT; BINARY CLASSIFICATION; E DISCOVERIES; EVALUATION; LARGE CORPORA; TEXT CLASSIFICATION; TEXT CLASSIFIERS;

CLASSIFICATION (OF INFORMATION); TEXT PROCESSING;

INFORMATION RETRIEVAL;

EID: 84883096802     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2484028.2484156     Document Type: Conference Paper

References (9)

1. Carla E. Brodley and Mark A. Friedl. Identifying mislabeled training data. Journal of Artificial Intelligence Research, 11:131-167, 1999.
2. Ben Carterette and Ian Soboroff. The effect of assessor errors on IR system evaluation. In Proc. 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 539-546, Geneva, Switzerland, July 2010.
3. Chih-Chung Chang and Chih-Jen Lin. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1-27:27, 2011. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm.
4. Gordon V. Cormack, Maura R. Grossman, Bruce Hedin, and Douglas W. Oard. Overview of the TREC 2010 legal track. In Ellen Voorhees and Lori P. Buckland, editors, Proc. 19th Text REtrieval Conference, pages 1:2:1-45, Gaithersburg, Maryland, USA, November 2010.
5. Maura R. Grossman and Gordon V. Cormack. Technology-assisted review in e-discovery can be more effective and more efficient than exhaustive manual review. Richmond Journal of Law and Technology, 17 (3):11:1-48, 2011.
6. John C. Platt. Probabilistic outputs for support vector machines and comparison to regularized likelihood methods. In Alexander J. Smola, Peter Bartlett, Bernhard Schölkopf, and Dale Schuurmans, editors, Advances in Large Margin Classifiers, pages 61-74. MIT Press, 1999.
7. Ganesh Ramakrishnan, Krishna Prasad Chitrapura, Raghu Krishnapu-ram, and Pushpak Bhattarcharyy. A model for handling approximate, noisy or incomplete labeling in text classification. In Proc. 22nd International Conference on Machine Learning, pages 681-688, Bonn, Germany, August 2005.
8. Ellen Voorhees. Variations in relevance judgments and the measurement of retrieval effectiveness. Information Processing & Management, 36(5):697-716, September 2000.
9. William Webber. Re-examining the effectiveness of manual review. In Proc. SIGIR Information Retrieval for E-Discovery Workshop, pages 2:1-8, Beijing, China, July 2011.