Minimum mutual information and non-gaussianity through the maximum entropy method: Estimation from finite samples

Entropy

Volumn 15, Issue 3, 2013, Pages 721-752

  Pires, Carlos A L ^a   Perdigão, Rui A P ^b  

^a UNIVERSITY OF LISBON   (Portugal)

^b VIENNA UNIVERSITY OF TECHNOLOGY   (Austria)

Author keywords

Entropy bias; Maximum entropy distributions; Morphism; Mutual information; Mutual information distribution; Non Gaussianity

Indexed keywords

EID: 84875404998     PISSN: None     EISSN: 10994300     Source Type: Journal    
DOI: 10.3390/e15030721     Document Type: Article

References (42)

1. Shannon, C.E. The mathematical theory of communication. Bell Syst. Tech. J. 1948, 27, 379-423.
2. Cover, T.M. Thomas, J.A. Elements of Information Theory John Wiley & Sons, Inc.: New York, NY, USA, 1991.
3. Averbeck, B.B. Latham, P.E. Pouget, A. Neural correlations, population coding and computation. Nat. Rev. Neurosci. 2006, 7, 358-366.
4. Goldie, C.M. Pinch, R.G.E. Communication Theory. In London Mathematical Society Student Texts (No. 20) Cambridge University Press: Cambridge, UK, 1991.
5. Sims, C.A. Rational Inattention: Beyond the Linear-Quadratic Case. Am. Econ. Rev. 2006, 96, 158-163.
6. Sherwin, W.E. Entropy and Information Approaches to Genetic Diversity and its Expression: Genomic Geography. Entropy 2010, 12, 1765-1798.
7. Pothos, E.M. Juola, P. Characterizing linguistic structure with mutual information. Br. J. Psychol. 2007, 98, 291-304.
8. Pires, C.A. Perdigão, R.A.P. Non-Gaussianity and asymmetry of the winter monthly precipitation estimation from the NAO. Mon. Wea. Rev. 2007, 135, 430-448.
9. Globerson, A. Tishby, N. The minimum information principle for discriminative learning. In Proceedings of the 20th conference on Uncertainty in artificial intelligence, Banff, Canada, 7-11 July 2004 pp. 193-200.
10. Globerson, A. Stark, E. Vaadia, E. Tishby, N. The minimum information principle and its application to neural code analysis. Proc. Natl. Accd. Sci. USA 2009, 106, 3490-3495.
11. Foster, D.V. Grassberger, P. Lower bounds on mutual information. Phys. Rev. E 2011, 83, 010101(R):1-010101(R):4.
12. Pires, C.A. Perdigão, R.A.P. Minimum Mutual Information and Non-Gaussianity Through the Maximum Entropy Method: Theory and Properties. Entropy 2012, 14, 1103-1126.
13. Walters-Williams, J. Li, Y. Estimation of mutual information: A survey. Lect. Notes Comput. Sci. 2009, 5589, 389-396.
14. Khan, S. Bandyopadhyay, S. Ganguly, A.R. Saigal, S. Erickson, D.J. Protopopescu, V. Ostrouchov, G. Relative performance of mutual information estimation methods for quantifying the dependence among short and noisy data. Phys. Rev. E 2007, 76, 026209:1-026209:15.
15. Paninski, L. Estimation of entropy and mutual information. Neural Comput. 2003, 15, 1191-1254.
16. Panzeri, S. Treves, A. Analytical estimates of limited sampling biases in different information measures. Comp. Neur. Syst. 1996, 7, 87-107.
17. Victor, J.D. Asymptotic Bias in Information Estimates and the Exponential (Bell) Polynomials. Neur. Comput. 2000, 12, 2797-2804.
18. Panzeri, S. Senatore, R. Montemurro, M.A. Petersen, R.S. Train Information Measures Correcting for the Sampling Bias Problem in Spike Information Measures. J. Neurophysiol. 2007, 98, 1064-1072.
19. Strong, S.P. Koberle, R. de Ruyter van Steveninck, R. Bialek, W. Entropy and information in neural spike trains. Phys. Rev. Lett. 1998, 86, 197-200.
20. Miller, G. Note on the bias of information estimates. In Information Theory in Psycholog Quastler, H., Ed. II-B Free Press: Glencoe, IL, USA, 1955 pp. 95-100.
21. Grassberger, P. Entropy Estimates from Insufficient Samplings. 2008, arXiv:physics/0307138v2.pdf.
22. Bonachela, J.A. Hinrichsen, H. Muñoz, M.A. Entropy estimates of small data sets. J. Phys. A 2008, 41, 202001.
23. Nelsen, R.B. An Introduction to Copulas Springer: New York, NY, USA, 1999 ISBN: 0-387-98623-5.
24. Calsaverini, R.S. Vicente, R. An information-theoretic approach to statistical dependence: Copula information. Europhys. Lett. 2009, 88, 68003.
25. Ma, J. Sun, Z. Mutual information is copula entropy. 2008, arXiv:0808.0845v1.
26. Macke, J.H. Murray, I. Latham, P.E. How biased are maximum entropy models? Adv. Neur. Inf. Proc. Syst. 2011, 24, 2034-2042.
27. Hutter, M. Zaffalon, M. Distribution of mutual information from complete and incomplete data. Comput. Stat. Data An. 2005, 48, 633-657.
28. Jaynes, E.T. On the Rationale of Maximum-entropy methods. P. IEEE 1982, 70, 939-952.
29. Shore, J.E. Johnson, R.W. Axiomatic derivation of the principle of maximum entropy and the principle of the minimum cross-entropy. IEEE Trans. Inform. Theor. 1980, 26, 26-37.
30. Ebrahimi, N. Soofi, E.S. Soyer, R. Information Measures in Perspective. Int. Stat. Rev. 2010, 78, 383-412.
31. Wackernagel, H. Multivariate Geostatistics-An Introduction with Applications Springer Verlag: Berlin, Germany, 1995.
32. Charpentier, A. Fermanian, J.D. Copulas: From Theory to Application in Finance Rank, J., Ed. Risk Publications: London, UK, 2007 Section 2.
33. Tam, S.M. On Covariance in Finite Population Sampling. J. Roy. Stat. Soc. D-Sta. 1985, 34, 429-433.
34. Van det Vaart, A.W. Asymptotic statistics. Cambridge University Press: New York, NY, USA 1998 ISBN 978-0-521-49603-2, LCCN. V22 1998 QA276. V22.
35. Rockinger, M. Jondeau, E. Entropy densities with an application to autoregressive conditional skewness and kurtosis. J. Econometrics 2002, 106, 119-142.
36. Bates, D. Quadratic Forms of Random Variables. STAT 849 lectures. Available online: http://www.stat.wisc.edu/~st849-1/lectures/Ch02.pdf (accessed on 22 February 2013).
37. Goebel, B. Dawy, Z. Hagenauer, J. Mueller, J.C. An approximation to the distribution of finite sample size mutual information estimates. 2005. In Proceedings of IEEE International Conference on Communications (ICC' 05), Seoul, Korea, 16-20 May 2005 pp. 1102-1106.
38. Fisher, R.A. On the "probable error" of a coefficient of correlation deduced from a small sample. Metron 1921, 1, 3-32.
39. Zientek, L.R. Thompson, B. Applying the bootstrap to the multivariate case: bootstrap component/factor analysis. Behav. Res. Methods 2007, 39, 318-325.
40. Mardia, K.V. Algorithm AS 84: Measures of multivariate skewness and kurtosis. Appl. Stat. 1975, 24, 262-265.
41. Hurrell, J.W. Kushnir, Y. Visbeck, M. The North Atlantic Oscillation. Science 2001, 26, 291.
42. The NCEP/NCAR Reanalysis Project. Available online: http://www.esrl.noaa.gov/psd/data/reanalysis/reanalysis.shtml/(accessed on 22 February 2013).