Distribution of multi-words in Chinese and English documents

International Journal of Information Technology and Decision Making

Volumn 8, Issue 2, 2009, Pages 249-265

  Zhang, Wen ^a,b   Yoshida, Taketoshi ^a   Tang, Xijin ^c  

^a JAPAN ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY   (Japan)

^b INSTITUTE OF SOFTWARE   (China)

^c INSTITUTE OF SYSTEMS SCIENCE   (China)

Author keywords

G distribution; Multi word; Poisson distribution; Term distribution; Zero inflated distribution

Indexed keywords

EID: 67749101286     PISSN: 02196220     EISSN: None     Source Type: Journal    
DOI: 10.1142/S0219622009003399     Document Type: Article

References (25)

1. J. Teevan and D. R. Karger, Empirical development of an exponential probabilistic model for text retrieval, in Proc. 26th Int. ACM SIGIR Conf. (Toronto, Canada, 2003), pp. 18-25.
2. R. E. Madsen, D. Kauchak and C. Elkan, Modeling word burstiness using the Dirichlet distribution, in Proc. 22nd Int. Conf. Machine Learning (Bonn, Germany, 2005), pp. 545-552.
3. Q. Mei, H. Fang and C. Zhai, A study of poisson query generation model for information retrieval, in Proc. 30th Int. ACM SIGIR Conf. (Amsterdam, The Netherlands, 2007), pp. 319-326.
4. Y. Peng, G. Kou, Y. Shi and Z. Chen, A descriptive framework for the field of data mining and knowledge discovery, Int. J. Inform. Technol. Decision Making 7(4) (2008).
5. F. J. Ren, Automatic abstracting important sentences, Int. J. Inform. Technol. Decision Making,4 (2005) 141-152.
6. R. Sakamoto, Y. Sumi and K. Kogure, Hyperlinked comic strips for sharing personal contexts, Int. J. Inform. Technol. Decision Making 6 (2007) 443-458.
7. C. D. Manining and S. Scheutze, Foundations of Statistical Natural Language Processing (MIT Press, Cambridge, Massachusetts, 1999), pp. 548-561
8. W. Zhang, T. Yoshida and X. J. Tang, Text classification based on multi-word with support vector machine, Knowl.-Based Syst., in press.
9. D. Wang, M. Li and Z. Di, True reason for Zipf's law in language, Physica A 358 (2005) 545-550.
10. R. F. Cancho and R. V. Sole, Zipf's law and random texts, Adv. Complex Syst. 5(1) (2002) 1-6.
11. S. Katz, Distribution of content words and phrases in texts and language modeling, Nat. Language Eng. 2(1) (1996) 15-59.
12. K. W. Church and W. A. Gale, Poisson mixtures, Nat. Language Eng. 1(2) (1995) 163-190.
13. A. D. Roeck, A. Sarkar and P. Garthwaite, Frequent term distribution measures for dataset profiling, in Proc. 4th Int. Conf. Language Resources and Evaluation (Lisbon, Portugal, 2004), pp. 30-37.
14. M. Jansche, Parametric models of linguistic count data, in Proc. 41th Annual Meeting on Association for Computational Linguistics (Sapporo, Japan, 2003), pp. 288-295.
15. C. M. Bishop, Neural Network for Pattern Recognition (Oxford University Press, New York, 2003), pp. 65-73.
16. W. Zhang, X. J. Tang and T. Yoshida, Web text mining on a scientific forum, Int. J. Knowl. Syst. Sci. 3(4) (2006) 51-59.
17. W. Zhang, X. J. Tang and T. Yoshida, Text classification toward a scientific forum, J. Syst. Sci. Syst. Eng. 16(3) (2007) 356-369.
18. M. Yamamoto and K. W. Church, Using suffix arrays to compute term frequency and document frequency for all substrings in a corpus, in Proc. 6th Workshop on very Large Corpora (Montreal, Canada, 1998), pp. 285-313.
19. W. Zhang, T. Yoshida and X. J. Tang, Augmented mutual information for multi-word extraction, Int. J. Innovative Comput. Inform. Contr. (2009), in press.
20. K. W. Church and P. Hanks, Word association norms, mutual information, and lexicography, Comput. Linguist. 16(1) (1990) 22-29.
21. F. Jueston and S. M. Katz, Technical terminology: Some linguistic properties and an algorithm for identification in text, Nat. Language Eng. 1(1) (1995) 9-27.
22. G. K. Zipf, Human Behaviour and the Principle of Least Effort (Addison-Wesley, Cambridge, Massachusetts, 1949).
23. F. Mosteller and D. L. Wallance, Applied Bayesian and Classical Inference: The Case of the Federalist Papers, 2nd edn. (Springer-Verlag, New York, 1984).
24. M. Fuketa, Y, Kadoya, E. Atlam, T. Kunikata, K. Morita, S. Kashiji and J. Aoe, A method of extracting and evaluating good and bad reputations for natural language expressions, Int. J. Inform. Technol. Decision Making 4 (2005) 177-196.
25. B. Boutsinas and S. Athanasiadis, On merging classification rules, Int. J. Inform. Technol. Decision Making 7(3) (2008) 431-450.