Coincidences and estimation of entropies of random variables with large cardinalities

Entropy

Volumn 13, Issue 12, 2011, Pages 2013-2023

  Nemenman, Ilya ^a  

^a EMORY UNIVERSITY   (United States)

Author keywords

Bias variance tradeoff; Coincidences; Entropy estimation; Model selection

Indexed keywords

EID: 84855371905     PISSN: None     EISSN: 10994300     Source Type: Journal    
DOI: 10.3390/e13122013     Document Type: Article

References (25)

1. Paninski, L. Estimation of entropy and mutual information. Neural Comp. 2003, 15, 1191-1253.
2. Panzeri, S.; Treves, A. Analytical estimates of limited sampling biases in different information measures. Netw. Comput. Neural Syst. 1996, 7, 87-107.
3. Strong, S.; Koberle, R.; de Ruyter van Steveninck, R.; Bialek, W. Entropy and information in neural spike trains. Phys. Rev. Lett. 1998, 80, 197-200.
4. Victor, J. Binless strategies for estimation of information from neural data. Phys. Rev. E 2002, 66, 051903.
5. Antos, A.; Kontoyiannis, I. Convergence properties of functional estimates for discrete distributions. Random Struct. Algorithm. 2002, 19, 163-193.
6. Batu, T.; Dasgupta, S.; Kumar, R.; Rubinfeld, R. The complexity of approximating the entropy. SIAM J. Comput. 2005, 35, 132-150.
7. Grassberger, P. Entropy estimates from insufficient samples. arXiv 2003, physics/0307138v2.
8. Wyner, A.; Foster, D. On the lower limits of entropy estimation. Available online: http://www-stat.wharton.upenn.edu/~ajw/lowlimitsentropy.pdf (accessed on 16 December 2011).
9. Kennel, M.; Shlens, J.; Abarbanel, H.; Chichilnisky, E. Estimating entropy rates with bayesian confidence intervals. Neural Comp. 2005, 17, 1531-1576.
10. Ma, S. Calculation of entropy from data of motion. J. Stat. Phys. 1981, 26, 221-240.
11. Orlitsky, A.; Santhanam, N.; Vishwanathan, K. Population estimation with performance guarantees. In Proceedings of the IEEE International Symposium on Information Theory, Nice, France, 24th-29th June 2007; pp. 2026-2030.
12. Orlitsky, A.; Santhanam, N.; Vishwanathan, K.; Zhang, J. Limit results on pattern entropy. IEEE Trans. Inf. Ther. 2006, 52, 2954-2964.
13. Nemenman, I.; Shafee, F.; Bialek, W. Entropy and inference, revisited. In Advances in Neural Information Processing Systems 14; Dietterich, T.G., Becker, S., Ghahramani, Z., Eds.; MIT Press: Cambridge, MA, USA, 2002.
14. Nemenman, I.; Bialek, W. de Ruyter van Steveninck, R. Entropy and information in neural spike trains: Progress on the sampling problem. Phys. Rev. E 2004, 69, 056111.
15. Nemenman, I.; Lewen, G.; Bialek, W.; de Ruyter van Steveninck, R. Neural coding of natural stimuli: Information at sub-millisecond resolution. PLoS Comput. Biol. 2008, 4, e1000025.
16. NSB Entropy Estimation. Available online: http://nsb-entropy.sourceforge.net/ (accessed on 16 December 2011).
17. Wolpert, D.;Wolf, D. Estimating functions of probability distributions from a finite set of samples. Phys. Rev. E 1995, 52, 6841-6854.
18. Sj̈olander, K.; Karplus, K.; Brown, M.; Hughey, R.; Krogh, A.; Mian, I.S.; Haussler, D. Dirichlet mixtures: A method for improved detection of weak but significant protein sequence homology. Comput. Appl. Biosci. 1996, 12, 327-345.
19. Jeffreys, H. Further significance tests. Proc. Camb. Phil. Soc. 1936, 32, 416-445.
20. Schwartz, G. Estimating the dimension of a model. Ann. Stat. 1978, 6, 461-464.
21. Clarke, B.; Barron, A. Information-Theoretic asymptotics of Bayes methods. IEEE Trans. Inf. Thy. 1990, 36, 453-471.
22. Balasubramanian, V. Statistical inference, Occam's razor, and statistical mechanics on the space of probability distributions. Neural Comp. 1997, 9, 349-368.
23. Nemenman, I. Fluctuation-dissipation theorem and models of learning. Neural Comp. 2005, 17, doi:10.1162/0899766054322982.
24. Gradshteyn, I.; Ryzhik, I. Tables of Integrals, Series and Products, 6th ed.; Academic Press: Burlington, MA, USA, 2000.
25. Schurmann, T.; Grassberger, P. Entropy estimation of symbol sequences. Chaos 1996, 6, 414-427.