Bayesian entropy estimation for countable discrete distributions

Journal of Machine Learning Research

Volumn 15, Issue , 2014, Pages 2833-2868

  Archer, Evan ^a,b   Park, Il Memming ^a   Pillow, Jonathan W ^a  

^a UNIVERSITY OF TEXAS AT AUSTIN   (United States)

^b MAX PLANCK INSTITUTE FOR BIOLOGICAL CYBERNETICS   (Germany)

Author keywords

Bayesian estimation; Bayesian nonparametrics; Dirichlet process; Entropy; Information theory; Neural coding; Pitman Yor process

Indexed keywords

ARTIFICIAL INTELLIGENCE; BAYESIAN NETWORKS; ESTIMATION; INFORMATION THEORY; LEARNING SYSTEMS;

BAYESIAN ESTIMATIONS; BAYESIAN NONPARAMETRICS; CLASS OF DISTRIBUTIONS; DIRICHLET PROCESS; LEAST SQUARES ESTIMATE; NEURAL CODING; NON-PARAMETRIC STATISTICS; POSTERIOR DISTRIBUTIONS;

ENTROPY;

EID: 84919763884     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (46)

1. A. Antos and I. Kontoyiannis. Convergence properties of functional estimates for discrete distributions. Random Structures & Algorithms, 19(3-4):163-193, 2001.
2. E. Archer, I. M. Park, and J. Pillow. Bayesian estimation of discrete entropy with mixtures of stick-breaking priors. In P. Bartlett, F. Pereira, C. Burges, L. Bottou, and K. Weinberger, editors, Advances in Neural Information Processing Systems, pages 2024-2032. MIT Press, Cambridge, MA, 2012.
3. E. Archer, I. M. Park, and J. W. Pillow. Bayesian entropy estimation for binary spike train data using parametric prior knowledge. In Advances in Neural Information Processing Systems, 2013.
4. R. Barbieri, L. Frank, D. Nguyen, M. Quirk, V. Solo, M. Wilson, and E. Brown. Dynamic analyses of information encoding in neural ensembles. Neural Computation, 16:277-307, 2004.
5. J. Boyd. Exponentially convergent Fourier-Chebshev quadrature schemes on bounded and infinite intervals. Journal of Scientific Computing, 2(2):99-109, 1987.
6. A. Chao and T. Shen. Nonparametric estimation of Shannon's index of diversity when there are unseen species in sample. Environmental and Ecological Statistics, 10(4):429-443, 2003.
7. C. Chow and C. Liu. Approximating discrete probability distributions with dependence trees. IEEE Transactions on Information Theory, 14(3):462-467, 1968.
8. T. Dudok de Wit. When do finite sample effects significantly affect entropy estimates? Eur. Phys. J. B - Cond. Matter and Complex Sys., 11(3):513-516, October 1999.
9. W. J. Ewens. Population genetics theory-the past and the future. In Mathematical and Statistical Developments of Evolutionary Theory, pages 177-227. Springer, 1990.
10. M. Farach, M. Noordewier, S. Savari, L. Shepp, A. Wyner, and J. Ziv. On the entropy of DNA: Algorithms and measurements based on memory and rapid convergence. In ACM-SIAM Symposium on Discrete Algorithms, pages 48-57. Society for Industrial and Applied Mathematics, 1995.
11. S. Favaro, A. Lijoi, R. H. Mena, and I. Prünster. Bayesian non-parametric inference for species variety with a two-parameter Poisson-Dirichlet process prior. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 71(5):993-1008, November 2009.
12. T. S. Ferguson. A Bayesian analysis of some nonparametric problems. The Annals of Statistics, 1(2):209-230, 1973.
13. A. Gnedin, B. Hansen, and J. Pitman. Notes on the occupancy problem with infinitely many boxes: general asymptotics and power laws. Probability Surveys, 4:146-171, 2007.
14. S. Goldwater, T. Griffiths, and M. Johnson. Interpolating between types and tokens by estimating power-law generators. In Advances in Neural Information Processing Systems, page 459. MIT Press, Cambridge, MA, 2006.
15. P. Grassberger. Entropy estimates from insufficient samplings. arXiv preprint, January 2008.
16. J. Hausser and K. Strimmer. Entropy inference and the James-Stein estimator, with application to nonlinear gene association networks. The Journal of Machine Learning Research, 10:1469-1484, 2009.
17. M. Hutter. Distribution of mutual information. In Advances in Neural Information Processing Systems, pages 399-406. MIT Press, Cambridge, MA, 2002.
18. H. Ishwaran and L. James. Generalized weighted Chinese restaurant processes for species sampling mixture models. Statistica Sinica, 13(4):1211-1236, 2003.
19. H. Ishwaran and M. Zarepour. Exact and approximate sum representations for the Dirichlet process. Canadian Journal of Statistics, 30(2):269-283, 2002.
20. H. Ishwaran and L. F. James. Gibbs sampling methods for stick-breaking priors. Journal of the American Statistical Association, 96(453):161-173, March 2001.
21. J. Kingman. Random discrete distributions. Journal of the Royal Statistical Society. Series B (Methodological), 37(1):1-22, 1975.
22. K. C. Knudson and J. W. Pillow. Spike train entropy-rate estimation using hierarchical dirichlet process priors. In C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, editors, Advances in Neural Information Processing Systems, pages 2076-2084. Curran Associates, Inc., 2013.
23. C. Letellier. Estimating the shannon entropy: recurrence plots versus symbolic dynamics. Physical Review Letters, 96(25):254102, 2006.
24. G. Miller. Note on the bias of information estimates. Information Theory in Psychology: Problems and Methods, 2:95-100, 1955.
25. T. Minka. Estimating a Dirichlet distribution. Technical report, MIT, 2003.
26. I. Nemenman. Coincidences and estimation of entropies of random variables with large cardinalities. Entropy, 13(12):2013-2023, 2011.
27. I. Nemenman, F. Shafee, and W. Bialek. Entropy and inference, revisited. In Advances in Neural Information Processing Systems, pages 471-478. MIT Press, Cambridge, MA, 2002.
28. I. Nemenman, W. Bialek, and R. Van Steveninck. Entropy and information in neural spike trains: Progress on the sampling problem. Physical Review E, 69(5):056111, 2004.
29. M. Newman. Power laws, Pareto distributions and Zipf's law. Contemporary Physics, 46(5):323-351, 2005.
30. L. Paninski. Estimation of entropy and mutual information. Neural Computation, 15:1191-1253, 2003.
31. S. Panzeri and A. Treves. Analytical estimates of limited sampling biases in di erent information measures. Network: Computation in Neural Systems, 7:87-107, 1996.
32. M. Perman, J. Pitman, and M. Yor. Size-biased sampling of poisson point processes and excursions. Probability Theory and Related Fields, 92(1):21-39, March 1992.
33. J. W. Pillow, L. Paninski, V. J. Uzzell, E. P. Simoncelli, and E. J. Chichilnisky. Prediction and decoding of retinal ganglion cell responses with a probabilistic spiking model. The Journal of Neuroscience, 25:11003-11013, 2005.
34. J. Pitman. Random discrete distributions invariant under size-biased permutation. Advances in Applied Probability, pages 525-539, 1996.
35. J. Pitman and M. Yor. The two-parameter Poisson-Dirichlet distribution derived from a stable subordinator. The Annals of Probability, 25(2):855-900, 1997.
36. E. T. Rolls, M. J. Tovée, and S. Panzeri. The neurophysiology of backward visual masking: Information analysis. Journal of Cognitive Neuroscience, 11(3):300-311, May 1999.
37. K. H. Schindler, M. Palus, M. Vejmelka, and J. Bhattacharya. Causality detection based on information-theoretic approaches in time series analysis. Physics Reports, 441:1-46, 2007.
38. J. Shlens, M. B. Kennel, H. D. I. Abarbanel, and E. J. Chichilnisky. Estimating information rates with confidence intervals in neural spike trains. Neural Computation, 19(7):1683-1719, Jul 2007.
39. R. Strong, S. Koberle, de Ruyter van Steveninck R., and W. Bialek. Entropy and information in neural spike trains. Physical Review Letters, 80:197-202, 1998.
40. Y. Teh. A hierarchical Bayesian language model based on Pitman-Yor processes. Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics, pages 985-992, 2006.
41. A. Treves and S. Panzeri. The upward bias in measures of information derived from limited data samples. Neural Computation, 7:399-407, 1995.
42. V. Q. Vu, B. Yu, and R. E. Kass. Coverage-adjusted entropy estimation. Statistics in Medicine, 26 (21):4039-4060, 2007.
43. D. H. Wolpert and S. DeDeo. Estimating functions of distributions defined over spaces of unknown size. Entropy, 15(11):4668-4699, 2013.
44. D. Wolpert and D. Wolf. Estimating functions of probability distributions from a finite set of samples. Physical Review E, 52(6):6841-6854, 1995.
45. Z. Zhang. Entropy estimation in Turing's perspective. Neural Computation, pages 1-22, 2012.
46. G. K. Zipf. Human Behavior and the Principle of Least Effort. Addison-Wesley Press, 1949.