Random texts do not exhibit the real Zipf's law-like rank distribution

PLoS ONE

Volumn 5, Issue 3, 2010, Pages

  Ferrer I Cancho, Ramon ^a   Elvevåg, Brita ^b  

^a UNIVERSITAT POLITÈCNICA DE CATALUNYA   (Spain)

^b NATIONAL INSTITUTE OF MENTAL HEALTH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHM; ARTICLE; AUTOMATION; LAW; PROBABILITY; RANDOM SAMPLE; RANK SUM TEST; STATISTICAL ANALYSIS; COMPUTER; COMPUTER PROGRAM; HUMAN; LANGUAGE; SEMANTICS; STATISTICAL MODEL; THEORETICAL MODEL;

ALGORITHMS; COMPUTERS; HUMANS; LANGUAGE; MODELS, STATISTICAL; MODELS, THEORETICAL; PROBABILITY; SEMANTICS; SOFTWARE;

EID: 77949686094     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0009411     Document Type: Article

References (35)

1. Zipf GK (1972) Human behaviour and the principle of least effort. An introduction to human ecology. New York: Hafner reprint. 1st edition: Cambridge, MA: Addison-Wesley, 1949.
2. Li W (1992) Random texts exhibit Zipf's-law-like word frequency distribution. IEEE T Inform Theory 38: 1842-1845.
3. Rapoport A (1982) Zipf's law re-visited. Quantitative Linguistics 16: 1-28.
4. Miller GA, Chomsky N (1963) Finitary models of language users. In: Luce RD, Bush R, Galanter E, eds. Handbook of Mathematical Psychology. New York: Wiley, volume 2. pp 419-491.
5. Miller GA (1957) Some effects of intermittent silence. Am J Psychol 70: 311-314.
6. Mitzenmacher M (2003) A brief history of generative models for power law and lognormal distributions. Internet Mathematics 1: 226-251.
7. Suzuki R, Tyack PL, Buck J (2005) The use of Zipf's law in animal communication analysis. Anim Behav 69: 9-17.
8. Mandelbrot B (1953) An informational theory of the statistical structure of language. In: Jackson W, ed. Communication theory. London: Butterworths. 486 p.
9. Nicolis JS (1991) Chaos and information processing. Singapore: World Scientific.
10. Ferrer-i-Cancho R, Gavaldà R (2009) The frequency spectrum of finite samples from the intermittent silence process. Journal of the American Association for Information Science and Technology 60: 837-843.
11. Herdan G (1964) Quantitative linguistics. London: Butterworths.
12. Miller DW (1995) Fitting frequency distributions: philosophy and practice. Volume I: discrete distributions. New York: Book Resource.
13. Miller DW (1995) Fitting frequency distributions: philosophy and practice. Volume I: continous distributions. New York: Book Resource.
14. Clauset A, Shalizi CR, Newman MEJ (2009) Power-law distributions in empirical data. SIAM Review 51: 661-703.
15. Popescu II, Altmann G, Köhler R (2009) Zipf's law - another view. Quality and Quantity: in press.
16. Conrad B, Mitzenmacher M (2004) Power laws for monkeys typing randomly: the case of unequal probabilities. IEEE Transactions on Information Theory 50: 1403-1414.
17. Manning CD, Schütze H (1999) Foundations of statistical natural language processing. CambridgeMA: MIT Press, chapter Introduction.
18. Mandelbrot B (1966) Information theory and psycholinguistics: a theory of word frequencies. In: Lazarsfield PF, Henry NW, eds. Readings in mathematical social sciences. Cambridge: MIT Press. pp 151-168.
19. Cohen A, Mantegna RN, Havlin S (1997) Numerical analysis of word frequencies in artificial and natural language texts. Fractals 5: 95-104.
20. Zipf GK (1935) The psycho-biology of language. Boston: Houghton Mifflin.
21. Balasubrahmanyan VK, Naranan S (1996) Quantitative linguistics and complex system studies. J Quantitative Linguistics 3: 177-228.
22. Ferrer i Cancho R (2007) On the universality of Zipf's law for word frequencies. In: Grzybek P, Köhler R, eds. Exact methods in the study of language and text. To honor Gabriel Altmann. Berlin: Gruyter. pp 131-140.
23. Newman MEJ (2005) Power laws, Pareto distributions and Zipf's law. Contemporary Physics 46: 323-351.
24. Manin DY (2008) Zipf's law and avoidance of excessive synonymy. Cognitive Science 32: 1075-1098.
25. Wolfram S (2002) A new kind of science. Champaign: Wolfram Media.
26. Ferrer i Cancho R, Solé RV (2002) Zipf's law and random texts. Advances in Complex Systems 5: 1-6.
27. Ferrer i Cancho R (2005) Zipf's law from a communicative phase transition. European Physical Journal B 47: 449-457.
28. Bell TC, Cleary JG, Witten IH (1990) Text Compression. Englewood Cliffs, NJ: Prentice-Hall.
29. DeGroot MH (1989) Probability and statistics. Reading, MA: Addison-Wesley. 2nd edition.
30. Levelt WJM (2001) Spoken word production: a theory of lexical access. Proc Natl Acad Sci USA 98: 13464-13471.
31. Ferrer i Cancho R, Solé RV (2003) Least effort and the origins of scaling in human language. Proceedings of the National Academy of Sciences USA 100: 788-791.
32. Zanette D, Montemurro MA (2005) Dynamics of text generation with realistic Zipf's distribution. J Quantitative Linguistics 12: 29-40.
33. Devroye L (1986) Non-uniform random variate generation. New York: Springer-Verlag.
34. Dagpunar JS (1988) Principles of random variate generation. Oxford: Clarendon Oxford Science Publications.
35. Cormen TH, Leiserson CE, Rivest RL (1990) Introduction to algorithms. Cambridge, MA: The MIT Press.