Zipfs law holds for phrases, not words

Scientific Reports

Volumn 5, Issue , 2015, Pages

  Ryland Williams, Jake ^a   Lessard, Paul R ^b   Desu, Suma ^c   Clark, Eric M ^a   Bagrow, James P ^a   Danforth, Christopher M ^a   Sheridan Dodds, Peter ^a  

^a Department of Mathematics AndStatistics Vermont Complex Systems Center Computational Story Lab Andthe Vermont Advanced Computing Core University of Vermont ^*  (United States)

^b UNIVERSITY OF COLORADO   (United States)

^c Massachusetts Institute of Technology   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DATA MINING; HUMAN; LANGUAGE; PROCEDURES; THEORETICAL MODEL;

DATA MINING; HUMANS; LANGUAGE; MODELS, THEORETICAL;

EID: 84939141504     PISSN: None     EISSN: 20452322     Source Type: Journal    
DOI: 10.1038/srep12209     Document Type: Article

References (29)

1. Zipf, G. K. The Psycho-Biology of Language (Houghton-Mifflin, New York, NY, 1935).
2. Zipf, G. K. Human Behaviour and the Principle of Least-Effort (Addison-Wesley, Cambridge, MA, 1949).
3. Simon, H. A. On a class of skew distribution functions. Biometrika 42, 425-440 (1955).
4. Batty, M. The size, scale, and shape of cities. Science Magazine 319, 769-771 (2008).
5. Axtell, R. Zipf distribution of U.S. firm sizes. Science 293, 1818-1820 (2001).
6. Zanette, D. H. and Manrubia, S. C. Vertical transmission of culture and the distribution of family names. Physica A 295, 1-8 (2001).
7. Mandelbrot, B. B. An informational theory of the statistical structure of languages. In Jackson, W. (ed.) Communication Theory, 486-502 (Butterworth, Woburn, MA, 1953).
8. Miller, G. A. Some effects of intermittent silence. American Journal of Psychology 70, 311-314 (1957).
9. Ferrer-i Cancho, R.andElvevag, B. Random texts do not exhibit the real Zipf s law-like rank distribution. PLoS ONE 5, e9411 (2010).
10. DSouza, R. M., Borgs, C., Chayes, J. T., Berger, N.andKleinberg, R. D. Emergence of tempered preferential attachment from optimization. Proc. Natl. Acad. Sci. 104, 6112-6117 (2007).
11. Coromina-Murtra, B.and Sole, R. Universality of Zifps law. Phsyical Revew E 82, 011102 (2010).
12. Bornholdt, S.andEbel, H. World Wide Web scaling exponent from Simons 1955 model. Phys. Rev. E 64, 035104(R) (2001).
13. Maillart, T., Sornette, D., Spaeth, S.andvon Krogh, G. Empirical tests of Zipf s law mechanism in open source Linux distribution. Phys. Rev. Lett. 101, 218701 (2008).
14. de Solla Price, D. J. A general theory of bibliometric and other cumulative advantage processes. J. Amer. Soc. Inform. Sci. 27, 292-306 (1976).
15. Barabasi, A.-L.andAlbert, R. Emergence of scaling in random networks. Science 286, 509-511 (1999).
16. Sag, I. A., Baldwin, T., Bond, F., Copestake, A. A.and Flickinger, D. Multiword expressions: A pain in the neck for NLP. In Proceedings of the Third International Conference on Computational Linguistics and Intelligent Text Processing, CICLing 02, 1-15 (Springer-Verlag, London, UK, 2002).
17. (2012). Google Labs n-gram viewer. Available at http://ngrams.googlelabs.com/. Accessed May 15, 2014.
18. I should have known it ( Broadcast January 29, 2013). Cougar Town, Season 4, Episode 4, http://www.imdb.com/title/tt2483134/. Accessed March 15, 2015.
19. Goldenfeld, N. Lectures on Phase Transitions and the Renormalization Group, vol. 85 of Frontiers in Physics (Addison-Wesley, Reading, Massachusetts, 1992).
20. Ferrer-i Cancho, R.and Sole, R. V. Two regimes in the frequency of words and the origins of complex lexicons: Zipf s law revisited. Journal of Quantitative Linguistics 8, 165-173 (2001).
21. Gerlach, M.and Altmann, E. G. Stochastic model for the vocabulary growth in natural languages. Phys. Rev. X 3, 021006 (2013).
22. Williams, J. R., Bagrow, J. P., Danforth, C. M.and Dodds, P. S. Text mixing shapes the anatomy of rank-frequency distributions. Phys. Rev. E 91, 052811 (2015).
23. Project Gutenberg (2010). http://www.gutenberg.org. Accessed May 15, 2014.
24. Ha, L. Q., Sicilia-Garcia, E. I., Ming, J.and Smith, F. J. Extension of Zipf s law to words and phrases. In Proceedings of the 19th International Conference on Computational Linguistics (COLING), 315-320 (2002).
25. Wikipedia database dump, 2010, https://meta.wikimedia.org/wiki/Data-dumps. Accessed March 15, 2015.
26. Sandhaus, E. The New York Times Annotated Corpus. Linguistic Data Consortium, Philadelphia (2008).
27. Dodds, P. S.and Danforth, C. M. Measuring the happiness of large-scale written expression: Songs, blogs, and presidents. Journal of Happiness Studies 11, 441-456 (2009).
28. Clauset, A., Shalizi, C. R.and Newman, M. E. J. Power-law distributions in empirical data. SIAM Review 51, 661-703 (2009).
29. Rayner, J. M. V. Linear relations in biomechanics: the statistics of scaling functions. J. Zool. Lond. (A) 206, 415-439 (1985).