Level statistics of words: Finding keywords in literary texts and symbolic sequences

Physical Review E - Statistical, Nonlinear, and Soft Matter Physics

Volumn 79, Issue 3, 2009, Pages

  Carpena, P ^a   Bernaola Galvan P ^a   Hackenberg, M ^b   Coronado, A V ^a   Oliver, J L ^c  

^a UNIVERSITY OF MÁLAGA   (Spain)

^b CIC BIOGUNE   (Spain)

^c UNIVERSITY OF GRANADA   (Spain)

Author keywords

[No Author keywords available]

Indexed keywords

A-PRIORI; DISORDERED SYSTEMS; LEVEL STATISTICS; LITERARY TEXTS; SPATIAL DISTRIBUTIONS; SYMBOLIC SEQUENCES;

EID: 63649110040     PISSN: 15393755     EISSN: 15502376     Source Type: Journal    
DOI: 10.1103/PhysRevE.79.035102     Document Type: Article

References (28)

1. WordNet: An Electronic Lexical Database, edited by, C. Fellbaum, (MIT Press, Cambridge, MA, 1998);
2. E. Frank, Proceedings of the 16th International Joint Conference on Artificial Intelligence, 1999, p. 668;
3. L. van der Plas, in Proceedings of the 4th International Conference on Language Resources and Evaluation, edited by, M. T. Lino, M. F. Xavier, F. Ferreira, R. Costa, and, R. Silva, European Language Resource Association, 2004, p. 2205;
4. K Cohen and L. Hunter, PLOS Comput. Biol. 4, e20 (2008). 10.1371/journal.pcbi.0040020
5. H. P. Luhn, IBM J. Res. Dev. 2, 157 (1958).
6. Y. Matsuo and M. Ishizuka, Int. J. Artif. Intell. 13, 157 (2004); 10.1142/S0218213004001466
7. G. Palshikar, in Proceedings of the Second International Conference on Pattern Recognition and Machine Intelligence (PReMI07), Vol. 4815 of Lecture Notes on Computer Science, edited by, A. Ghosh, R. K. De, and, S. K. Pal, (Springer-Verlag, Berlin, 2007), p. 503.
8. G. Salton and M. J. McGill, Introduction to Modern Information Retrieval (McGraw-Hill, New York, 1983);
9. K. Sparck Jones, J. Document. 28, 11 (1972); 10.1108/eb026526
10. S. E. Robertson, in Overview of the Third Text REtrieval Conference (TREC-3), NIST SP 500-225, edited by, D. K. Harman, (National Institute of Standards and Technology, Gaithersburg, MD, 1995), pp. 109-126.
11. A. Bookstein and D. R. Swanson, J. Am. Soc. Inf. Sci. 25, 312 (1974); 10.1002/asi.4630250505
12. A. Bookstein and D. R. Swanson, J. Am. Soc. Inf. Sci. 26, 45 (1975). 10.1002/asi.4630260107
13. S. P. Harter, J. Am. Soc. Inf. Sci. 26, 197 (1975). 10.1002/asi. 4630260402
14. A. Berger and J. Lafferty, Proc. ACM SIGIR99, 1999, p. 222;
15. J. Ponte and W. Croft, Proc. ACM SIGIR98, 1998, p. 275.
16. T. A. Brody, Rev. Mod. Phys. 53, 385 (1981); 10.1103/RevModPhys.53.385
17. M. L. Mehta, Random Matrices (Academic Press, New York, 1991).
18. M. Ortuño, Europhys. Lett. 57, 759 (2002). 10.1209/epl/i2002-00528- 3
19. P. Carpena, P. Bernaola-Galvan, and P. C. Ivanov, Phys. Rev. Lett. 93, 176804 (2004). 10.1103/PhysRevLett.93.176804
20. H. D. Zhou and G. W. Slater, Physica A 329, 309 (2003); 10.1016/S0378-4371(03)00625-3
21. M. J. Berryman, A. Allison, and D. Abbott, Fluct. Noise Lett. 3, L1 (2003). 10.1142/S021947750300104X
22. V. T. Stefanov, J. Appl. Probab. 40, 881 (2003). 10.1239/jap/1067436088
23. S. Robin and J. J. Daudin, Ann. Inst. Stat. Math. 53, 895 (2001). 10.1023/A:1014633825822
24. http://bioinfo2.ugr.es/TextKeywords
25. H. J. Bussemaker, H. Li, and E. D. Siggia, Proc. Natl. Acad. Sci. U.S.A. 97, 10096 (2000). 10.1073/pnas.180265397
26. Now, the nearest neighbors distances are measured in letters, not in words, since we advance letter by letter exploring the text. Thus we ignore the "reading frame," which would be relevant only in texts with fixed-length words consecutively placed, as the words of length 3 (codons) in the coding regions of DNA. In addition, we consider only nonoverlapping occurrences of a word. Although the overlapping is not very likely in text without spaces, it can happen in other symbolic sequences, such as DNA, and the expected P (d) is different for overlapping or nonoverlapping randomly placed words (see S. Robin, F. Rodolphe, and S. Schbath, DNA, Words and Models (Cambridge University Press, Cambridge, England, 2005).
27. T. H. Cormen, Introduction to Algorithms (The MIT Press, Cambridge, MA, 1990), pp. 485-488.
28. The algorithm proceeds as follows: we consider an initial l0, and we start with a given l0 word for which C> C0, where C0 corresponds to a p value=0.05. We then find the child (l0 +1) word with the highest C value, and proceed the same in the following generations: we find the successive child (l0 +i) word with the highest C value; i=1,2,.... This process stops at a previously chosen maximal word length lmax. Finally, we choose as the representative of the lineage the longest word for which C> C0 and define it as the extracted semantic unit. We repeat this algorithm for all the l0 words with C> C0, and we repeat also all the processes by changing the initial l0 value. Finally, we remove the remaining redundancies (repeated words or semantic units) due to explore all lineages from different initial l0.