Statistical mechanics of letters in words

Physical Review E - Statistical, Nonlinear, and Soft Matter Physics

Volumn 81, Issue 6, 2010, Pages

  Stephens, Greg J ^a   Bialek, William ^a  

^a PRINCETON UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ENERGY LANDSCAPE; LOCAL MINIMUMS; MAXIMUM ENTROPY MODELS; PAIRWISE CORRELATION;

ENTROPY; PROBABILITY DENSITY FUNCTION; STATISTICAL MECHANICS;

PROBABILITY DISTRIBUTIONS;

EID: 77953980096     PISSN: 15393755     EISSN: 15502376     Source Type: Journal    
DOI: 10.1103/PhysRevE.81.066119     Document Type: Article

References (27)

1. See, for example, D. Mumford, in New Directions in Statistical Signal Processing: From Systems to Brain, edited by, S. Haykin, (MIT Press, Cambridge, MA, 2005), pp. 3-34.
2. E. T. Jaynes, Phys. Rev. 106, 620 (1957). 10.1103/PhysRev.106.620
3. E. Schneidman, M. J. Berry II, R. Segev, and W. Bialek, Nature (London) 440, 1007 (2006). 10.1038/nature04701
4. J. Shlens, J. Neurosci. 26, 8254 (2006). 10.1523/JNEUROSCI.1282-06.2006
5. See the presentations at the Society for Neuroscience (http://www.sfn.org/am2007/)
6. I. E. Ohiorhenuan and J. D. Victor, Annual Meeting of the Society for Neuroscience (unpublished), 615.8/O01
7. S. Yu, D. Huang, W. Singer, and D. Nikolić, Annual Meeting of the Society for Neuroscience (unpublished), 615.14/O07
8. M. A. Sacek, Annual Meeting of the Society for Neuroscience (unpublished), 790.1/J12
9. A. Tang, Annual Meeting of the Society for Neuroscience (unpublished), 792.4/K27.
10. J. J. Hopfield, Proc. Natl. Acad. Sci. U.S.A. 79, 2554 (1982). 10.1073/pnas.79.8.2554
11. G. Tkačik, Ph.D. thesis, Princeton University, 2007.
12. T. R. Lezon, Proc. Natl. Acad. Sci. U.S.A. 103, 19033 (2006). 10.1073/pnas.0609152103
13. W. Bialek and R. Ranganathan, e-print arXiv:0712.4397.
14. G. Tkačik, E. Schneidman, M. J. Berry II, and W. Bialek, e-print arXiv:q-bio.NC/0611072.
15. E. Schneidman, S. Still, M. J. Berry II, and W. Bialek, Phys. Rev. Lett. 91, 238701 (2003). 10.1103/PhysRevLett.91.238701
16. M. Mezard, G. Parisi, and M. Virasoro, Spin Glass Theory and Beyond (World Scientific, Singapore, 1987).
17. A. L. Berger, S. Della Pietra, and V. J. Della Pietra, Comput. Linguist. 22, 39 (1996).
18. N. Chomsky, IRE Trans. Inf. Theory 2, 113 (1956). 10.1109/TIT.1956. 1056813
19. "In one's introductory linguistics course, one learns that Chomsky disabused the field once and for all of the notion that there was anything of interest to statistical models of language. But one usually comes away a little fuzzy on the question of what, precisely, he proved;" S. Abney, in The Balancing Act: Combining Statistical and Symbolic Approaches to Language, edited by, J. L. Klavans, and, P. Resnik, (MIT Press, Cambridge, MA, 1996), pp. 1-26.
20. F. Pereira, Philos. Trans. R. Soc. London, Ser. A 358, 1239 (2000). 10.1098/rsta.2000.0583
21. The Austen word corpus was created via Project Guttenberg (www.gutenberg.org), combining text from Emma, Lady Susan, Love and Friendship, Mansfield Park, Northhanger Abbey, Persuassion, Pride and Prejudice, and Sense and Sensibility. Out of 676302 total words in our Austen corpus there were 7114 unique words, 763 of which were four-letter words; the four-letter words occurred in the corpus a total of 135441 times. We used the second release of the ANC (www.americannationalcorpus.org) with ∼ 2 × 10 7 words and restricted ourselves to words used more than 100 times, providing 798 unique four-letter words occurring 2 179108 times. These numbers indicate that we can sample the distribution of four-letter words with reasonable confidence. To control for potential typographic errors, words were also checked against a large dictionary database (http://wordlist.sourceforge.net/12dicts-readme.html).
22. For technical points about finite sample sizes and error bars, see N. Slonim, G. S. Atwal, G. Tkačik, and W. Bialek, e-print arXiv:cs/0502017v1.
23. T. Broderick, M. Dudik, G. Tkačik, R. Schapire, and W. Bialek, e-print arXiv:0712.2437.
24. J. N. Darroch and D. Ratcliff, Ann. Math. Stat. 43, 1470 (1972). 10.1214/aoms/1177692379
25. G. K. Zipf, Selected Studies of the Principle of Relative Frequency in Language (Harvard University Press, Cambridge, MA, 1932).
26. W. Li, IEEE Trans. Inf. Theory 38, 1842 (1992). 10.1109/18.165464
27. S. Dehaene, L. Cohen, M. Sigman, and F. Vinckier, Trends Cogn. Sci. 9, 335 (2005). 10.1016/j.tics.2005.05.004