Using suffix arrays to compute term frequency and document frequency for all substrings in a corpus

Computational Linguistics

Volumn 27, Issue 1, 2001, Pages 1-30

  Yamamoto, Mikio ^a   Church, Kenneth W ^b  

^a Inst of Info Sci and Electronics   (Japan)

^b NONE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EQUIVALENCE CLASSES; NATURAL LANGUAGE PROCESSING SYSTEMS; SEMANTICS; TERMINOLOGY; TEXT PROCESSING;

DOCUMENT FREQUENCY; INDEPENDENCE ASSUMPTION; INVERSE DOCUMENT FREQUENCY; MUTUAL INFORMATIONS; STATISTICAL NATURAL LANGUAGE PROCESSING; TECHNICAL TERMINOLOGY; TERM FREQUENCY; WALL STREET JOURNAL;

COMPUTATIONAL LINGUISTICS;

EID: 0038632285     PISSN: 08912017     EISSN: None     Source Type: Journal    
DOI: 10.1162/089120101300346787     Document Type: Article

References (17)

1. Charniak, Eugene. 1993. Statistical Language Learning. MIT Press.
2. Church, Kenneth W. and William A. Gale. 1995. Poisson mixtures. Natural Language Engineering, 1(2):163-190.
3. Gonnet, Gaston H., Ricardo A. Baeza-Yates, and Tim Snider. 1992. New indices for text: PAT trees and PAT arrays. In Information Retrieval: Data Structure & Algorithms, pages 66-82. Prentice Hall PTR.
4. Gusfield, Dan. 1997. Algorithms on Strings, Trees, and Sequences: Computer Science and Computational Biology. Cambridge University Press.
5. Hui, Lucas Chi Kwong. 1992. Color set size problem with applications to string matching. In Lecture Notes in Computer Science, Volume 644. Springer (CPM92: Combinatorial Pattern Matching, 3rd Annual Symposium), pages 230-243.
6. Jelinek, Frederick. 1997. Statistical Methods for Speech Recognition. MIT Press.
7. Katz, Slava M. 1987. Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE Transactions on Acoustics, Speech, and Signal Processing, ASSP-35(3):400-401.
8. Kita, Kenji, Yasuhiko Kato, Takashi Omoto, and Yoneo Yano. 1994. A comparative study of automatic extraction of collocations from corpora: Mutual information vs. cost criteria. Journal of Natural Language Processing, 1(1):21-33.
9. Manber, Udi and Gene Myers. 1990. Suffix arrays: A new method for on-line string searches. In Proceedings of The First Annual ACM-SIAM Symposium on Discrete Algorithms, pages 319-327. URL=http:// glimpse.cs.arizona.edu/udi.html.
10. McCreight, Edward M. 1976. A space-economical suffix tree construction algorithm. Journal of the ACM, 23:262-272.
11. Nagao, Makoto and Shinsuke Mori. 1994. A new method of n-gram statistics for large number of n and automatic extraction of words and phrases from large text data of Japanese. In Proceedings of the 15th International Conference on Computational Linguistics, pages 611-615.
12. Nagata, Masaaki. 1996. Automatic extraction of new words from Japanese texts using generalized forward-backward search. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 48-59.
13. Paul, Douglas B. and Janet M. Baker. 1992. The design for the Wall Street Journal-based CSR corpus. In Proceedings of DARPA Speech and Natural Language Workshop, pages 357-361.
14. Shibatani, Masayoshi. 1990. The Languages of Japan. Cambridge Language Surveys. Cambridge University Press.
15. Sparck Jones, Karen. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28(1):11-21.
16. Sproat, Richard and Chilin Shih. 1990. A statistical method for finding word boundaries in Chinese text. Computer Processing of Chinese and Oriental Languages, 4(4):336-351.
17. Ukkonen, Esko. 1995. On-line construction of suffix trees. Algorithmica, 14(3):249-260.