Towards minimizing the annotation cost of certified text classification

International Conference on Information and Knowledge Management, Proceedings

Volumn , Issue , 2013, Pages 989-998

  Bagdouri, Mossaab ^a   Webber, William ^a   Lewis, David D ^b   Oard, Douglas W ^a  

^a UNIVERSITY OF MARYLAND   (United States)

^b David D Lewis Consulting   (United States)

Author keywords

E discovery; Evaluation; Supervised learning; Text categorization

Indexed keywords

ALLOCATION POLICIES; ANALYTIC APPROXIMATION; CONFIDENCE INTERVAL; E DISCOVERIES; EVALUATION; STATISTICAL VALIDITY; TEXT CATEGORIZATION; TEXT CLASSIFICATION;

KNOWLEDGE MANAGEMENT; SUPERVISED LEARNING; TEXT PROCESSING;

CLASSIFICATION (OF INFORMATION);

EID: 84889598884     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2505515.2505708     Document Type: Conference Paper

References (32)

1. Y. Bengio and Y. Grandvalet. No unbiased estimator of the variance of k-fold cross-validation. Journal of Machine Learning Research, 5:1089-1105, September 2004.
2. A. Borodin and R. El-Yaniv. Online Computation and Competitive Analysis. Cambirdge University Press, 2005.
3. T. T. Cai. One-sided confidence intervals in discrete distributions. Journal of Statistical Planning and Inference, 131:63-88, 2005.
4. A. M. Cohen, W. R. Hersh, K. Peterson, and P.-Y. Yen. Reducing workload in systematic review preparation using automated citation classification. Journal of the American Medical Informatics Association, 13:206-219, 2006.
5. J. Cohen. Statistical Power Analysis for the Behavioral Sciences. Lawrence Erlbaum Associates, 2nd edition, 1988.
6. A. Isaksson, M. Wallman, H. Göransson, and M. G. Gustafsson. Cross-validation and bootstrapping are unreliable in small sample classification. Pattern Recognition Letters, 29:1960-1965, 2008.
7. T. Joachims. Text categorization with support vector machines: Learning with many relevant features. In ECML, pages 137-142, 1998.
8. T. Joachims. A support vector method for multivariate performance measures. In ICML, pages 377-384, 2005.
9. T. Joachims. Training linear svms in linear time. In KDD, pages 217-226, 2006.
10. T. Joachims and C.-N. J. Yu. Sparse kernel svms via cutting-plane training. In ECML PKDD: Part I, 2009.
11. L. P. Kaelbling, M. L. Littman, and A. W. Moore. Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4:237-285, 1996.
12. A. Kershaw and J. Howie. eDiscovery institute survey on predictive coding. Technical report, Electronic Discovery Institute (http://www. ediscoveryinstitute.org/pubs/PredictiveCodingSurvey.pdf), October 2010.
13. J. Langford. Combining train set and test set bounds. In ICML, pages 331-338, 2002.
14. J. Langford. Tutorial on practical prediction theory for classification. Journal of Machine Learning Research, 6(1):273-306, 2005.
15. E. L. Lehmann and J. P. Romano. Testing Statistical Hypotheses. Springer, 3rd edition, 2005.
16. D. D. Lewis and W. A. Gale. A sequential algorithm for training text classifiers. In SIGIR, pages 3-12, 1994.
17. D. D. Lewis, Y. Yang, T. G. Rose, and F. Li. RCV1: A new benchmark collection for text categorization research. Journal of Machine Learning Research, 5:361-397, December 2004.
18. A. M. Mood, F. A. Graybill, and D. C. Boes. Introduction to the Theory of Statistics. McGraw-Hill, 3rd edition, 1974.
19. C. Nadeau and Y. Bengio. Inference for the generalization error. Machine Learning, 52:239-281, 2003.
20. D. W. Oard, J. R. Baron, B. Hedin, D. D. Lewis, and S. Tomlinson. Evaluation of information retrieval for e-discovery. Artificial Intelligence and Law, 18(4):347-386, 2010.
21. N. M. Pace and L. Zakaras. Where the money goes: Understanding litigant expenditures for producing electronic discovery. Technical report, RAND Institute for Civil Justice, Santa Monica, CA, 2012.
22. F. Provost, D. Jensen, and T. Oates. Efficient progressive sampling. In KDD, pages 23-32, 1999.
23. G. Salton and C. Buckley. Term-weighting approaches in automatic text retrieval. Information Processing and Management, 24(5):513-523, 1988.
24. R. Simon, M. D. Radmacher, K. Dobbin, and L. M. McShane. Pitfalls in the use of DNA microarray data for diagnostic and prognostic classification. Journal of the National Cancer Institute, 95(1):14-18, January 2003.
25. J. R. Taylor. Introduction to error analysis. University Science Books, 2nd edition, 1997.
26. S. K. Thompson. Sampling. Wiley, 2nd edition, 2002.
27. G. T. Toussaint. Bibliography on estimation of misclassification. IEEE Transactions on Information Theory, IT-20(4):472-479, July 1974.
28. C. J. van Rijsbergen. Information Retrieval. Butterworth, 1979.
29. W. Webber. Approximate recall confidence intervals. ACM Transactions on Information Systems, 31(1):2:1-33, 2013.
30. W. Webber, M. Bagdouri, D. D. Lewis, and D. W. Oard. Sequential testing in classifier evaluation yields biased estimates of effectiveness. In SIGIR, pages 933-936, July 2013.
31. U. Wickenberg-Bolin, H. Göransson, M. Fryknäs, M. G. Gustafsson, and A. Isaksson. Improved variance estimation of classification performance via reduction of bias caused by small sample size. BMC Bioinformatics, 2006.
32. Y. Yang and X. Liu. A re-examination of text categorization methods. In SIGIR, pages 42-49, 1999.