Sequential testing in classifier evaluation yields biased estimates of effectiveness

SIGIR 2013 - Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2013, Pages 933-936

  Webber, William ^a   Bagdouri, Mossaab ^a   Lewis, David D ^b   Oard, Douglas W ^a  

^a UNIVERSITY OF MARYLAND   (United States)

^b David D Lewis Consulting   (United States)

Author keywords

Evaluation; F measure; Supervised learning; Text categorization

Indexed keywords

BIASED ESTIMATES; CLASSIFIER EVALUATION; EVALUATION; F-MEASURE; OPTIMISTIC ESTIMATES; SEQUENTIAL TESTING; TEXT CATEGORIZATION; TEXT CLASSIFICATION;

ESTIMATION; INFORMATION RETRIEVAL; RESEARCH; SUPERVISED LEARNING; TEXT PROCESSING;

CLASSIFICATION (OF INFORMATION);

EID: 84883057552     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2484028.2484159     Document Type: Conference Paper

References (10)

1. Cyril Goutte and Eric Gaussier. A probabilistic interpretation of precision, recall and F-score, with implication for evaluation. In ECIR '05, pages 345-359, March 2005.
2. David Haussler, Michael Kearns, H Sebastian Seung, and Naftali Tishby. Rigorous learning curve bounds from statistical mechanics. Machine Learning, 25(2):195-236, 1996.
3. Thorsten Joachims. Training linear SVMs in linear time. In ACM KDD '06, pages 217-226, 2006. ISBN 1-59593-339-5.
4. Michael Kearns and Dana Ron. Algorithmic stability and sanity-check bounds for leave-one-out cross-validation. Neural Computation, 11(6): 1427-1453, 1999.
5. David D Lewis. Evaluating and optimizing autonomous text classification systems. In ACM SIGIR '95, pages 246-254, 1995.
6. David D. Lewis, Yiming Yang, Tony G. Rose, and Fan Li. RCV1: A new benchmark collection for text categorization research. Journal of Machine Learning Research, 5:361-397, December 2004.
7. David Siegmund. Sequential Analysis: Tests and Confidence Intervals. Springer-Verlag, 1985.
8. Godfried T. Toussaint. Bibliography on estimation of misclassification. IEEE Transactions on Information Theory, IT-20(4):472-479, July 1974.
9. William Webber. Approximate recall confidence intervals. ACM TOIS, 31 (1):2:1-2:33, January 2013.
10. G. Barrie Wetherill and Kevin D. Glazebrook. Sequential Methods in Statistics. Chapman and Hall, 3rd edition, 1986.