Analyzing word frequencies in large text corpora using inter-arrival times and bootstrapping

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Volumn 6912 LNAI, Issue PART 2, 2011, Pages 341-357

  Lijffijt, Jefrey ^a,b   Papapetrou, Panagiotis ^a,b   Puolamäki, Kai ^a,b   Mannila, Heikki ^a,b  

^a AALTO UNIVERSITY   (Finland)

^b HELSINKI INSTITUTE FOR INFORMATION TECHNOLOGY   (Finland)

Author keywords

burstiness; natural language modeling; sequence analysis

Indexed keywords

BAG OF WORDS; BERNOULLI PROCESS; BURSTINESS; FALSE POSITIVE; FIXED PARAMETERS; FREQUENCY COUNTS; GOLD STANDARDS; INTER-ARRIVAL TIME; METHOD MODEL; NATURAL LANGUAGES; NULL MODEL; OCCURRENCE PATTERN; SCIENTIFIC DISCIPLINE; SEQUENCE ANALYSIS; SIGNIFICANCE TESTING; SPATIAL PATTERNS; TEXT CORPORA; WORD FREQUENCIES;

COMPUTATIONAL LINGUISTICS; LEARNING SYSTEMS; NATURAL LANGUAGE PROCESSING SYSTEMS; SOFTWARE AGENTS;

DATA MINING;

EID: 80052405328     PISSN: 03029743     EISSN: 16113349     Source Type: Book Series    
DOI: 10.1007/978-3-642-23783-6_22     Document Type: Conference Paper

References (26)

1. Altmann, E.G., Pierrehumbert, J.B., Motter, A.E.: Beyond word frequency: Bursts, lulls, and scaling in the temporal distributions of words. PLoS ONE 4(11), e7678 (2009)
2. Baeza-Yates, R., Ribeiro-Neto, B.: Modern Information Retrieval. Addison-Wesley Longman, Amsterdam (1999)
3. Barabási, A.-L.: The origin of bursts and heavy tails in human dynamics. Nature 435, 207-211 (2005)
4. Biber, D.: Dimensions of register variation: A cross-linguistic comparison. Cambridge University Press, Cambridge (1995)
5. Dunning, T.: Accurate methods for the statistics of surprise and coincidence. Computational Linguistics 19, 61-74 (1993)
6. Efron, B., Tibshirani, R.J.: An Introduction to the Bootstrap. Chapman and Hall/CRC (1994)
7. Faloutsos, M., Faloutsos, P., Faloutsos, C.: On power-law relationships of the internet topology. In: ACM SIGCOMM, pp. 251-262 (1999)
8. Fung, G.P.C., Pui, G., Fung, C., Yu, J.X., Yu, P.S., Yu, S., Lu, H.: Parameter free bursty events detection in text streams. In: VLDB, pp. 181-192 (2005)
9. Gries, S.T.: Null-hypothesis significance testing of word frequencies: a follow-up on Kilgarriff. Corpus Linguistics and Linguistic Theory 12, 277-294 (2005)
10. Gries, S.T.: Syntactic priming: A corpus-based approach. Journal of Psycholinguistic Research 34(4), 365-399 (2005)
11. He, Q., Chang, K., Lim, E.-P.: Analyzing feature trajectories for event detection. In: ACM SIGIR, pp. 207-214 (2007)
12. He, Q., Chang, K., Lim, E.-P.: Using burstiness to improve clustering of topics in news streams. In: IEEE ICDM, pp. 493-498 (2007)
13. He, Q., Chang, K., Lim, E.-P., Zhang, J.: Bursty Feature Representation for Clustering Text Streams. In: SIAM SDM, pp. 491-496 (2007)
14. Kilgarriff, A.: Language is never, ever, ever, random. Corpus Linguistics and Linguistic Theory 1-2, 263-275 (2005)
15. Kleinberg, J.: Bursty and hierarchical structure in streams. DMKD 7, 373-397 (2003)
16. Kumar, R., Novak, J., Raghavan, P., Tomkins, A.: On the bursty evolution of blogspace. World Wide Web 8(2), 159-178 (2005)
17. Lappas, T., Arai, B., Platakis, M., Kotsakos, D., Gunopulos, D.: On burstiness-aware search for document sequences. In: ACM SIGKDD, pp. 477-486 (2009)
18. Leskovec, J., Kleinberg, J., Faloutsos, C.: Graph evolution: Densification and shrinking diameters. ACM TKDD 1(1) (2007)
19. North, B.V., Curtis, D., Sham, P.C.: A note on the calculation of empirical p-values from Monte Carlo procedures. The American Journal of Human Genetics 71(2), 439-441 (2002)
20. Rayson, P., Garside, R.: Comparing corpora using frequency profiling. In: 38th ACL Workshop on Comparing Corpora, pp. 1-6 (2000)
21. Rayson, P., Leech, G., Hodges, M.: Social differentiation in the use of English vocabulary: some analyses of the conversational component of the British National Corpus. International Journal of Corpus Linguistics 2(1), 133-152 (1997)
22. Robertson, S.E., Walker, S.: Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval. In: ACM SIGIR, pp. 232-241 (1994)
23. Szmrecsanyi, B.: Language users as creatures of habit: A corpus-based analysis of persistence in spoken English. Corpus Linguistics and Linguistic Theory 1(1), 113-149 (2005)
24. The British National Corpus, version 3, BNC XML edn. (2007)
25. Vlachos, M.: Identifying similarities, periodicities and bursts for online search queries. In: ACM SIGMOD, pp. 131-142 (2004)
26. Zipf, G.K.: Human behavior and the principle of least effort. Addison-Wesley, Reading (1949)