Randomness versus specifics for word-frequency distributions

Physica A: Statistical Mechanics and its Applications

Volumn 444, Issue , 2016, Pages 828-837

  Yan, Xiaoyong ^a,b   Minnhagen, Petter ^c  

^a BEIJING JIAOTONG UNIVERSITY   (China)

^b UNIVERSITY OF ELECTRONIC SCIENCE AND TECHNOLOGY OF CHINA   (China)

^c UMEÅ UNIVERSITY   (Sweden)

Author keywords

Maximum entropy; Random Group Formation; Word frequency distributions; Zipf's law

Indexed keywords

CURVE FITTING; MAXIMUM ENTROPY METHODS; RANDOM PROCESSES;

POWER LAW INDEX; RANDOM GROUPS; SCALING PROPERTIES; TEXT LENGTH; WORD FREQUENCIES; ZIPF'S LAW;

LINGUISTICS;

EID: 84946594721     PISSN: 03784371     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.physa.2015.10.082     Document Type: Article

References (34)

1. J.B. Estroup Les Gammes Sténographiques fourth ed. 1916 Institut Stenographique de France Paris
2. G.K. Zipf Selective Studies of the Principle of Relative Frequency in Language 1932 Harvard University Press Cambridge
3. G.K. Zipf The Psycho-Biology of Language: An Introduction to Dynamic Philology 1935 Mifflin Boston
4. G.K. Zipf Human Bevavior and the Principle of Least Effort 1949 Addison-Wesley Reading
5. B. Mandelbrot An Informational Theory of the Statistical Structure of Languages 1953 Butterworth Woburn
6. W. Li Random texts exhibit Zipf's-law-like word frequency distribution IEEE Trans. Inform. Theory 38 1992 1842 1845
7. R.H. Baayen Word Frequency Distributions 2001 Kluwer Academic Dordrecht
8. R.F. i Cancho, and R.V. Solé Least effort and the origins of scaling in human language Proc. Natl. Acad. Sci. USA 100 2003 788 791
9. M.A. Montemurro Beyond the Zipf-Mandelbrot law in quantitative linguistics Physica A 300 2001 567 578
10. F. Font-Close, G. Boleda, and A. Corral A scaling law beyond Zipf's law and its relation to Heaps' law New J. Phys. 15 2013 093033
11. H. Simon On a class of skew distribution functions Biometrika 42 1955 425 440
12. I. Kanter, and D.A. Kessler Markov processes: linguistics and Zipf's Law Phys. Rev. Lett. 74 1995 4559 4562
13. S.N. Dorogovtsev, and J.F.F. Mendes Languague as an evolving word web Proc. R. Soc. Lond. Ser. B 268 2001 2603 2606
14. D.H. Zanette, and M.A. Montemurro Dynamics of text generation with realistic Zipf's distribution J. Quant. Linguist. 12 2005 29 40
15. A. Masucci, and G. Rodgers Networks properties of written human language Phys. Rev. E 74 2006 26102
16. C. Cattuto, V. Loreto, and V.D.P. Servedio A Yule-Simon process with memory Europhys. Lett. 76 2006 208 214
17. L. Lü, Z.-K. Zhang, and T. Zhou Deviation of Zipf's and Heaps' laws in human languages with limited dictionary sizes Sci. Rep. 3 2013 1082
18. G.A. Miller Some effects of intermittance silence Am. J. Psychol. 70 1957 311 314
19. S. Bernhardsson, S.K. Baek, and P. Minnhagen A paradoxical property of the monkey book J. Stat. Mech. 7 2011 PO7013
20. S. Bernhardsson, L.E.C. da Rocha, and P. Minnhagen Size dependent word frequencies and the translational invariance of books Physica A 389 2010 330 341
21. S. Bernhardsson, L.E.C. da Rocha, and P. Minnhagen The meta book and size-dependent properties of written language New J. Phys. 11 2009 123015
22. S.K. Baek, S. Bernhardsson, and P. Minnhagen Zipf's law unzipped New J. Phys. 13 2011 043004
23. X. Yan, and P. Minnhagen Maximum entropy, word-frequency, Chinese characters, and multiple meanings PLoS One 10 2015 e0125592
24. A.A. Markov An example of statistical investigations of the text Eugene Onegin concerning the connection of samples in chains Sci. Context 19 2006 595 600
25. B. Hayes First links in Markov chains Am. Sci. 101 2013 92 97
26. S.H. Lee, S. Bernhardsson, P. Holme, B.J. Kim, and P. Minnhagen Neutral theory of chemical reaction networks New J. Phys. 14 2012 033032
27. S.K. Baek, P. Minnhagen, and B.J. Kim The ten thousands Kims New J. Phys. 13 2011 073036
28. F. Bokma, S.K. Baek, and P. Minnhagen 50 years of inordinate fondness Syst. Biodivers. 63 2013 251 256
29. A. Bazzani, B. Giorgini, S. Rambaldi, R. Gallotti, and L. Giovannini Statistical laws in urban mobility from microscopic GPS data in the area of Florence J. Stat. Mech. 5 2010 P05001
30. X.-Y. Yan, X.-P. Han, B.-H. Wang, and T. Zhou Diversity of individual mobility patterns and emergence of aggregated scaling laws Sci. Rep. 3 2013 2678
31. D.T. Wijaya, R. Yeniterzi, Understanding semantic change of words over centuries, in: Proc. DETECT'11, 2011, pp. 35-40.
32. A.M. Petersen, J. Tenenbaum, S. Havlin, H. Stanley, and M. Perc Language cool as the expand: Allometric scaling and decreasing need for new words Sci. Rep. 2 2012 00943
33. L. Lü, Z.-K. Zhang, and T. Zhou Zipf's law leads to Heaps' law: Analyzing their relation in finite-size systems PLoS One 5 2010 e14139
34. http://storage.googleapis.com/books/ngrams/books/datasetsv2.html (Google Books Ngram Viewer. Data version: 20090715).