The ineffectiveness of within-document term frequency in text classification

Information Retrieval

Volumn 12, Issue 5, 2009, Pages 509-525

  Wilbur, W John ^a   Kim, Won ^a  

^a NATIONAL INSTITUTES OF HEALTH   (United States)

Author keywords

Bag of words; Within document frequency; Word burstiness

Indexed keywords

EID: 80052839148     PISSN: 13864564     EISSN: 15737659     Source Type: Journal    
DOI: 10.1007/s10791-008-9069-5     Document Type: Article

References (25)

1. Baeza-Yates, R., & Ribeiro-Neto, B. (1999). Modern information retrieval. Harlow, England: Addison-Wesley Longman Ltd.
2. Church, K. W. (1995). One term or two? In Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. Seattle, Washington: ACM Press.
3. Craven, M., DiPasquo, D., et al. (1998). Learning to extract symbolic knowledge from the World Wide Web. AAAI-98.
4. Elkan, C. (2006). Clustering documents with an exponential-family approximation of the Dirichlet compound multinomial distribution. In 23rd International Conference on Machine Learning. Pittsburgh, Pennsylvania: ACM Press.
5. Eyheramendy, S., Lewis, D., et al. (2003). On the Naive Bayes model for text categorization. In Ninth International Workshop on Artificial Intelligence & Statistics, Key West, FL.
6. Joachims, T. (1997). A probabilistic analysis of the Rocchio algorithm with TFIDF for text categorization. ICML-97.
7. Joachims, T. (2006). Training linear SVMs in linear time. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Philadelphia, PA, USA: ACM.
8. Kalt, T. (1996). A new probabilistic model of text classification and retrieval (1-9). Tech Report CIIR-TR-78, Univ. of Massachusetts.
9. Lewis, D. D. (1998). Naive (Bayes) at forty: The independence assumption in information retrieval. ECML.
10. Madsen, R. E., & Kauchak, D., et al. (2005). Modeling word burstiness using the Dirichlet distribution. In 22nd International Conference on Machine Learning. Bonn, Germany: ACM Press.
11. McCallum, A. K. (1996). Bow: A toolkit for statistical language modeling, text retrieval, classification and clustering. http://www.cs.cmu.edu/*mccallum/bow.
12. McCallum, A., & Nigam, K. (1998). A comparison of event models for naive bayes text classification. AAAI-98 Workshop on Learning for Text Categorization.
13. McEntyre, J., & Lipman, D. (2001). PubMed: Bridging the information gap. Canadian Medical Association Journal, 164(9), 1317-1319.
14. Minka, T. P. (2003). Estimating a Dirichlet distribution. Retrieved September 18, 2007, from https://research.microsoft.com/*minka/papers/dirichlet/
15. Mitchell, T. M. (1997). Machine learning. Boston: WCB/McGraw-Hill.
16. Pavlov, D., & Balasubramanyan, R., et al. (2004). Document preprocessing for naive Bayes classification and clustering with mixture of multinomials. KDD'04, Seattle, Washington.
17. Rennie, J. D. M., & Shih, L., et al. (2003). Tackling the poor assumptions of naive Bayes text classifiers. In Twentieth International Conference on Machine Learning, Washington, DC.
18. Salton, G. (1989). Automatic text processing. Reading, Massachusetts: Addison-Wesley Publishing Company.
19. Schneider, K.-M. (2003). A comparison of event models for Naive Bayes anti-spam e-mail filtering. EACL'03.
20. Schneider, K.-M. (2005). Techniques for improving the performance of Naive Bayes for text classification. In Computational Linguistics and Intelligent Text Processing, 6th International Conference, Mexico City, Springer.
21. Section, N. L. o. M. U. S. M. S. H. (2004). MeSH tree structures [electronic resource]/United States National Library of Medicine, National Institutes of Health, Medical Subject Headings. N. I. o. H. United States National Library of Medicine, U.S. National Library of Medicine, National Institutes of Health, Health & Human Services.
22. Teevan, J., & Karger, D. R. (2003). Empirical development of an exponential probabilistic model for text retrieval: Using textual analysis to build a better model. In ACM SIGIR Conference on Research and Development in Information Retrieval, Toronto, Canada.
23. Witten, I. H., Moffat, A., et al. (1999). Managing gigabytes. San Francisco: Morgan-Kaufmann Publishers, Inc.
24. Yang, Y., & Pedersen, J. O. (1997). A comparative study on feature selection in text categorization. Machine Learning: Proceedings of the Fourteenth International Conference (ICML'97).
25. Zhang, T., & Oles, F. J. (2001). Text categorization based on regularized linear classification methods. Information Retrieval, 4(1), 5-31. doi:10.1023/A:1011441423217.