Comparison of term frequency and document frequency based feature selection metrics in text categorization

Expert Systems with Applications

Volumn 39, Issue 5, 2012, Pages 4760-4768

  Azam, Nouman ^a   Yao, Jingtao ^a  

^a UNIVERSITY OF REGINA   (Canada)

Author keywords

Document frequency; Feature selection metrics; Term frequency; Text categorization

Indexed keywords

DATA SETS; DOCUMENT FREQUENCY; FEATURE SETS; GINI INDEX; RELATIVE IMPORTANCE; REUTERS; TERM FREQUENCY; TEXT CATEGORIZATION;

TEXT PROCESSING;

FEATURE EXTRACTION;

EID: 84855881106     PISSN: 09574174     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.eswa.2011.09.160     Document Type: Article

References (34)

1. Azam, N.; Yao, J. T. (2011). Incorporating game theory in feature selection for text categorization. In Proceedings of 13th international conference on rough sets, fuzzy sets, data mining and granular computing (RSFDGrC'11), Lecture notes in computer science (Vol. 6743, pp. 215-222).
2. L. Breiman, J. Friedman, C.J. Stone, and R. Olshen Classification and regression trees 1984 Wadsworth International Group Montery, CA
3. C.-M. Chen, H.-M. Lee, and Y.-J. Chang Two novel feature selection approaches for web page classification Expert Systems with Applications 36 1 2009 260 272
4. K. Coussement, and D. Van den Poel Integrating the voice of customers through call center emails into a decision support system for churn prediction Information Management 45 3 2008 164 174
5. F. Debole, and F. Sebastiani An analysis of the relative hardness of reuters-21,578 subsets Journal of the American Society for Information Science and Technology 56 6 2005 584 596
6. G. Forman An extensive empirical study of feature selection metrics for text classification Journal of Machine Learning Research 3 2003 1289 1305
7. A. Genkin, D.L. David, and D. Madigan Large-scale Bayesian logistic regression for text categorization Technometrics 49 3 2007 291 304
8. Kou, H.; Napoli, A.; Toussaint, Y. (2005). Application of text categorization to astronomy field. In Proceedings of 10th international conference on applications of natural language to information systems (NLDB'05), Lecture notes in computer science (Vol. 3513, pp. 32-43).
9. C. Lee, and G.G. Lee Information gain and divergence-based feature selection for machine learning-based text categorization Information Processing and Management 42 1 2006 155 165
10. Lewis, D. D. (1999). Reuters-21,578 text categorization collection. Retrieved 06.10.
11. H. Liang, J. Wang, and Y.Y. Yao User-oriented feature selection for machine learning The Computer Journal 50 2007 421 434
12. B. Li, Q. Lu, and S. Yu An adaptive k-nearest neighbor text categorization strategy ACM Transactions on Asian Language Information Processing 3 4 2004 215 226
13. H. Liu, and L. Yu Toward integrating feature selection algorithms for classification and clustering IEEE Transactions on Knowledge and Data Engineering 17 4 2005 491 502
14. López, F. R.; Jiménez-Salazar, H.; Pinto, D. (2007). A competitive term selection method for information retrieval. In Proceedings of 8th international conference on computational linguistics and intelligent text processing, (CICLing'07), Lecture notes in computer science (Vol. 4394, pp. 468-475).
15. Mladenic, D.; Grobelnik, M. (1999). Feature selection for unbalanced class distribution and Naive Bayes. In Proceedings of the 16th international conference on machine learning (ICML'99) (pp. 258-267).
16. Moyotl-Hernández, E.; Jiménez-Salazar, H. (2005). Enhancement of dtp feature selection method for text categorization. In Proceedings of 6th international conference on computational linguistics and intelligent text processing (CICLing'05), Lecture notes in computer science (Vol. 3406, pp. 719-722).
17. H. Ogura, H. Amano, and M. Kondo Feature selection with a measure of deviations from poisson in text categorization Expert Systems with Applications 36 3 2009 6826 6832
18. H. Ogura, H. Amano, and M. Kondo Distinctive characteristics of a metric using deviations from poisson for feature selection Expert Systems with Applications 37 3 2010 2273 2281
19. S. Piramuthu Evaluating feature selection methods for learning in data mining applications European Journal of Operational Research 156 2 2004 483 494
20. M. Porter An algorithm for suffix stripping Program 14 3 1980 130 137
21. X. Qi, and B.D. Davison Web page classification: Features and algorithms ACM Computing Surveys 41 2 2009
22. Y. Saeys, I. Inza, and P. Larranaga A review of feature selection techniques in bioinformatics Bioinformatics 23 19 2007 2507 2517
23. G. Sakkis, I. Androutsopoulos, G. Paliouras, V. Karkaletsis, C.D. Spyropoulos, and P. Stamatopoulos A memory-based approach to anti-spam filtering for mailing lists Information Retrieval 6 1 2003 49 73
24. F. Sebastiani Machine learning in automated text categorization ACM Computing Surveys 34 1 2002 1 47
25. W. Shang, H. Huang, H. Zhu, Y. Lin, Y. Qu, and Z. Wang A novel feature selection algorithm for text categorization Expert Systems with Applications 33 1 2007 1 5
26. Stopwords (2010). Retrieved 06.10.
27. Tang, B.; Shepherd, M.; Milios, E.; Heywood, M. I. (2005). Comparing and combining dimension reduction techniques for efficient text clustering. In Proceeding of 2005 international workshop on feature selection for data mining (ICDM'05) (pp. 17-26).
28. S. Wang, D. Li, X. Song, Y. Wei, and H. Li A feature selection method based on improved fisher's discriminant ratio for text sentiment classification Expert Systems with Applications 38 7 2011 8696 8702
29. Yang, Y.; Pedersen, J. O. (1997). A comparative study on feature selection in text categorization. In Proceedings of the 14th international conference on machine learning (ICML'97) (pp. 412-420).
30. Yang, S. M.; Wu, X.-B.; Deng, Z.-H.; Zhang, M.; Yang, D.-Q. (2002). Relative term frequency based feature selection for text categorization. In Proceedings of the 1st international conference on machine learning and cybernatics (ICMLC'02) (pp. 1432-1436).
31. Yao, J. T.; Zhang, M. (2005). Feature selection with adjustable criteria. In Proceedings of 10th international conference on rough sets, fuzzy sets, data mining, and granular computing (RSFDGrC'05), Lecture notes in computer science (Vol. 3641, pp. 204-213).
32. Y.Y. Yao, and Y. Zhao Attribute reduction in decision-theoretic rough set models Information Science 178 17 2008 3356 3373
33. Y.Y. Yao, Y. Zhao, and J. Wang On reduct construction algorithms Transactions on Computational Science 2 2008 100 117
34. Zhou, B.; Yao, Y. Y, & Luo, J. (2010). A three-way decision approach to email spam filtering. In Proceedings of 23rd canadian conference on artificial intelligence (Canadain AI'10), Lecture notes in computer science (Vol. 6085, pp. 28-39).