A Repetition Based Measure for Verification of Text Collections and for Text Categorization

SIGIR Forum (ACM Special Interest Group on Information Retrieval)

Volumn , Issue SPEC. ISS., 2003, Pages 104-110

  Khmelev, Dmitry V ^a,b   Teahan, William J ^c  

^a UNIVERSITY OF TORONTO   (Canada)

^b MOSCOW STATE UNIVERSITY   (Russian Federation)

^c BANGOR UNIVERSITY   (United Kingdom)

Author keywords

Cross entropy; Language modeling; Text categorization; Text compression

Indexed keywords

ALGORITHMS; ARRAYS; COMPUTATIONAL METHODS; COMPUTER SOFTWARE SELECTION AND EVALUATION; DATA STRUCTURES; FORMAL LANGUAGES; NATURAL LANGUAGE PROCESSING SYSTEMS; STANDARDS;

CROSS-ENTROPY; LANGUAGE MODELING; TEXT CATEGORIZATION; TEXT COMPRESSION;

TEXT PROCESSING;

EID: 1542377547     PISSN: 01635840     EISSN: None     Source Type: Journal    
DOI: 10.1145/860454.860456     Document Type: Conference Paper

References (22)

1. E. L. Allwein, R. E. Schapire, and Y. Singer. Reducing multiclass to binary: A unifying approach for margin classifiers. In Proc. 17th International Conf. on Machine Learning, pages 9-16. Morgan Kaufmann, San Francisco, CA, 2000.
2. M. Burrows and D. J. Wheeler. A block-sorting lossless data compression algorithm. Technical Report 124, Digital SRC, Palo Alto, 1994.
3. K. Crammer and Y. Singer. On the learnability and design of output codes for multiclass problems. In Computational Learning Theory, pages 35-46, 2000.
4. A. Crauser and P. Ferragina. A theoretical and experimental study on the construction of suffix arrays in external memory. Algorithmica, 32(1):1-35, 2002.
5. J. Diederich, J. Kindermann, E. Leopold, and G. Paass. Authorship attribution with support vector machines.
6. P. Ferragina and G. Manzini. Opportunistic data structures with applications. In 41st Ann. Symp. on Found, of Comput. Sc., pages 390-398. IEEE Comput. Soc. Press, Los Alamitos, CA, 2000.
7. E. Frank, C. Chui, and I. H. Witten. Text categorization using compression models. In Proceedings of DCC-00, IEEE DCC, pages 200-209, Snowbird. US, 2000. IEEE Computer Society Press.
8. T. Joachims. Learning to Classify Text Using Support Vector Machines. Kluwer, New Jersey, 2002.
9. D. Khmelev. Disputed Authorship Resolution through Using Relative Empirical Entropy for Markov Chains of Letters in Human Language Text. J. of Quantitative Linguistics, 7(3):201-207, 2000.
10. D. V. Khmelev and W. J. Teahan. Verification of text collections for text categorization and natural language processing. Technical Report AIIA 03.1, School of Informatics, Univ. of Wales, Bangor, 2003.
11. O. Kukushkina, A. Polikarpov, and D. Khmelev. Using Letters and Grammatical Statistics for Authorship Attribution. Problems of Information Transmission, 37(2): 172-184, 2001.
12. U. Manber and G. Myers. Suffix arrays: a new method for on-line string searches. SIAM J. Comput., 22(5):935-948, 1993.
13. M. Porter. An algorithm for suffix stripping. Program, 14(3):130-137, 1980.
14. Reuters-21578 Text Categorization Collection. Available at http://kdd.ics.uci.edu/databases/reuters21578/reuters21578.html.
15. T. Rose, M. Stevenson, and M. Whitehead. The Reuters Corpus Volume 1 - from yesterday's news to tomorrow's language resources. In Proceedings of the Third International Conference on Language Resources and Evaluation, pages 29-31, Las Palmas de Gran Canaris, 2002. IEEE Computer Society Press.
16. M. Sanderson. Duplicate detection in the Reuters collection. Technical report. Department of Computer Science, Univ. of Glasgow, 1997.
17. C. E. Shannon. A mathematical theory of communication. Bell System Technical Journal, 27:379-423,623-656,1948,1948.
18. D. Shkarin. Improving the efficiency of the PPM algorithm. Problems of Information Transmission. 37(3):226-235, 2001.
19. W. J. Teahan. Text classification and segmentation using minimum cross-entropy. In Proc. RIAO'2000, volume 2, pages 943-961, Paris, France, 2000.
20. W. J. Teahan and D. J. Harper. Using compression- based language models for text categorization. In Workshop on Lang. Modeling and Inform. Retrieval, pages 83-88. Carnegie Mellon Univ., May 2001.
21. The 20 Newsgroups data set. Available at http://www.ai.mit.edu/people/jrennie/20Newsgroups/.
22. J. Weston and C. Watkins. Multi-class support vector machines, 1998.