Bayesian and quasi-Bayesian estimators for mutual information from discrete data

Entropy

Volumn 15, Issue 5, 2013, Pages 1738-1755

  Archer, Evan ^a   Park, Il Memming ^a   Pillow, Jonathan ^a  

^a UNIVERSITY OF TEXAS AT AUSTIN   (United States)

Author keywords

Bayes least squares; Dirichlet distribution; Entropy; Mutual information

Indexed keywords

EID: 84878363315     PISSN: None     EISSN: 10994300     Source Type: Journal    
DOI: 10.3390/e15051738     Document Type: Article

References (23)

1. Schindler, K.H.; Palus, M.; Vejmelka, M.; Bhattacharya, J. Causality detection based on information-theoretic approaches in time series analysis. Phys. Rep. 2007, 441, 1-46.
2. Rényi, A. On measures of dependence. Acta Math. Hung. 1959, 10, 441-451.
3. Chow, C.; Liu, C. Approximating discrete probability distributions with dependence trees. Inf. Theory IEEE Trans. 1968, 14, 462-467.
4. Rieke, F.; Warland, D.; de Ruyter van Steveninck, R.; Bialek, W. Spikes: Exploring the Neural Code; MIT Press: Cambridge, MA, USA, 1996.
5. Ma, S. Calculation of entropy from data of motion. J. Stat. Phys. 1981, 26, 221-240.
6. Bialek, W.; Rieke, F.; de Ruyter van Steveninck, R., R.R.; Warland, D. Reading a neural code. Science 1991, 252, 1854-1857.
7. Strong, S. Koberle, R.; de Ruyter van Steveninck R.; Bialek, W. Entropy and information in neural spike trains. Phys. Rev. Lett. 1998, 80, 197-202.
8. Paninski, L. Estimation of entropy and mutual information. Neural Comput. 2003, 15, 1191-1253.
9. Barbieri, R.; Frank, L.; Nguyen, D.; Quirk, M.; Solo, V.; Wilson, M.; Brown, E. Dynamic Analyses of Information Encoding in Neural Ensembles. Neural Comput. 2004, 16, 277-307.
10. Kennel, M.; Shlens, J.; Abarbanel, H.; Chichilnisky, E. Estimating Entropy Rates with Bayesian Confidence Intervals. Neural Comput. 2005, 17, 1531-1576.
11. Victor, J. Approaches to information-theoretic analysis of neural activity. Biol. Theory 2006, 1, 302-316.
12. Shlens, J.; Kennel, M.B.; Abarbanel, H.D.I.; Chichilnisky, E.J. Estimating information rates with confidence intervals in neural spike trains. Neural Comput. 2007, 19, 1683-1719.
13. Vu, V.Q.; Yu, B.; Kass, R.E. Coverage-adjusted entropy estimation. Stat. Med. 2007, 26, 4039-4060.
14. Montemurro, M.A.; Senatore, R.; Panzeri, S. Tight data-robust bounds to mutual information combining shuffling and model selection techniques. Neural Comput. 2007, 19, 2913-2957.
15. Vu, V.Q.; Yu, B.; Kass, R.E. Information in the Nonstationary Case. Neural Comput. 2009, 21, 688-703.
16. Archer, E.; Park, I.M.; Pillow, J. Bayesian estimation of discrete entropy with mixtures of stick-breaking priors. In Advances in Neural Information Processing Systems 25; Bartlett, P.; Pereira, F.; Burges, C.; Bottou, L.; Weinberger, K., Eds.; MIT Press: Cambridge, MA, 2012; pp. 2024-2032.
17. Nemenman, I.; Shafee, F.; Bialek, W. Entropy and inference, revisited. In Advances in Neural Information Processing Systems 14; MIT Press: Cambridge, MA, 2002; pp. 471-478.
18. Hutter, M. Distribution of mutual information. In Advances in Neural Information Processing Systems 14; MIT Press: Cambridge, MA, 2002; pp. 399-406.
19. Hutter, M.; Zaffalon, M. Distribution of mutual information from complete and incomplete data. Comput. Stat. Data Anal. 2005, 48, 633-657.
20. Treves, A.; Panzeri, S. The upward bias in measures of information derived from limited data samples. Neural Comput. 1995, 7, 399-407.
21. Wolpert, D.; Wolf, D. Estimating functions of probability distributions from a finite set of samples. Phys. Rev. E 1995, 52, 6841-6854.
22. Minka, T. Estimating a Dirichlet Distribution; Technical report, MIT: Cambridge, MA, USA, 2003.
23. Nemenman, I.; Lewen, G.D.; Bialek, W.; de Ruyter van Steveninck, R.R. Neural coding of natural stimuli: information at sub-millisecond resolution. PLoS Comput. Biol. 2008, 4, e1000025.