Minimum mutual information and non-gaussianity through the maximum entropy method: Theory and properties

Entropy

Volumn 14, Issue 6, 2012, Pages 1103-1126

  Pires, Carlos A L ^a   Perdigão, Rui A P ^a  

^a UNIVERSITY OF LISBON   (Portugal)

Author keywords

Maximum entropy distributions; Mutual information; Non Gaussian noise; Non Gaussianity

Indexed keywords

EID: 84864594043     PISSN: None     EISSN: 10994300     Source Type: Journal    
DOI: 10.3390/e14061103     Document Type: Article

References (42)

1. Cover, T.M.; Thomas, J.A. Elements of Information Theory; John Wiley and Sons, Inc.: New York, NY, USA, 1991.
2. Ebrahimi, N.; Soofi, E.S.; Soyer R. Information measures in perspective. Int. Stat. Rev. 2010, 78, 383-412.
3. Stögbauer, H.; Kraskov, A.; Astakhov, S.A.; Grassberger, P. Least-dependent-component analysis based on mutual information. Phys. Rev. E 2004, 70, 066123:1-066123:17.
4. Hyvärinen, A.; Oja, E. Independent Component Analysis: Algorithms and application. Neural Network. 2000, 13, 411-430.
5. Fraser, H.M.; Swinney, H.L. Independent coordinates for strange attractors from mutual information. Phys. Rev. A 1986, 33, 1134-1140.
6. DelSole, T. Predictability and information theory. Part I: Measures of predictability. J. Atmos. Sci. 2004, 61, 2425-2440.
7. Majda, A.; Kleeman, R.; Cai, D. A mathematical framework for quantifying predictability through relative entropy. Methods and applications of analysis. Meth. Appl. Anal. 2002, 9, 425-444.
8. Darbellay, G.A.; Vajda, I. Entropy expressions for multivariate continuous distributions. IEEE Trans. Inform. Theor. 2000, 46, 709-712.
9. Nadarajah, S.; Zografos, K. Expressions for Rényi and Shannon entropies for bivariate distributions. Inform. Sci. 2005, 170, 173-189.
10. Khan, S.; Bandyopadhyay, S.; Ganguly, A.R.; Saigal, S.; Erickson, D.J.; Protopopescu, V.; Ostrouchov, G. Relative performance of mutual information estimation methods for quantifying the dependence among short and noisy data. Phys. Rev. E 2007, 76, 026209:1-026209:15.
11. Walters-Williams, J.; Li, Y. Estimation of mutual information: A survey. Lect. Notes Comput. Sci. 2009, 5589, 389-396.
12. Globerson, A.; Tishby, N. The minimum information principle for discriminative learning. In Proceedings of the 20th conference on Uncertainty in artificial intelligence, Banff, Canada, 7-11 July 2004; pp. 193-200.
13. Jaynes, E.T. On the rationale of maximum-entropy methods. Proc. IEEE 1982, 70, 939-952.
14. Wackernagel, H. Multivariate Geostatistics-An Introduction with Applications; Springer-Verlag: Berlin, Germany, 1995.
15. Shams, S.A. Convergent iterative procedure for constructing bivariate distributions. Comm. Stat. Theor. Meth. 2010, 39, 1026-1037.
16. Ebrahimi, N.; Soofi, E.S.; Soyer, R. Multivariate maximum entropy identification, transformation, and dependence. J. Multivariate Anal. 2008, 99, 1217-1231.
17. Abramov, R. An improved algorithm for the multidimensional moment-constrained maximum entropy problem. J. Comput. Phys. 2007, 226, 621-644.
18. Abramov, R. The multidimensional moment-constrained maximum entropy problem: A BFGS algorithm with constraint scaling. J. Comput. Phys. 2009, 228, 96-108.
19. Abramov, R. The multidimensional maximum entropy moment problem: A review on numerical methods. Commun. Math. Sci. 2010, 8, 377-392.
20. Rockinger, M.; Jondeau, E. Entropy densities with an application to autoregressive conditional skewness and kurtosis. J. Econometrics 2002, 106, 119-142.
21. Pires, C.A.; Perdigão, R.A.P. Non-Gaussianity and asymmetry of the winter monthly precipitation estimation from the NAO. Mon. Wea. Rev. 2007, 135, 430-448.
22. Kraskov, A.; Stögbauer, H.; Grassberger, P. Estimating mutual information. Phys. Rev. E 2004, 69, 066138:1-066138:16.
23. Myers, J.L.; Well, A.D. Research Design and Statistical Analysis, 2nd ed.; Lawrence Erlbaum Associates: Mahwah, NJ, USA, 2003.
24. Calsaverini, R.S.; Vicente, R. An information-theoretic approach to statistical dependence: Copula information. Europhys. Lett. 2009, 88, 68003; pp. 1-6.
25. Monahan, A.H.; DelSole, T. Information theoretic measures of dependence, compactness, and non-Gaussianity for multivariate probability distributions. Nonlinear Proc. Geoph. 2009, 16, 57-64.
26. Guo, D.; Shamai, S.; Verdú, S. Mutual information and minimum mean-square error in Gaussian channels. IEEE Trans. Inform. Theor. 2005, 51, 1261-1283.
27. Pires, C.A.; Perdigão, R.A.P. Minimum mutual information and non-Gaussianity through the maximum entropy method: Estimation from finite samples. Entropy 2012, submitted for publication.
28. Shore, J.E.; Johnson, R.W. Axiomatic derivation of the principle of maximum entropy and the principle of the minimum cross-entropy. IEEE Trans. Inform. Theor. 1980, 26, 26-37.
29. Koehler, K.J.; Symmanowski, J.T. Constructing multivariate distributions with specific marginal distributions. J. Multivariate Anal. 1995, 55, 261-282.
30. Cruz-Medina, I.R.; Osorio-Sánchez, M.; García-Páez, F. Generation of multivariate random variables with known marginal distribution and a specified correlation matrix. InterStat 2010, 16, 19-29.
31. van Hulle, M.M. Edgeworth approximation of multivariate differential entropy. Neural Computat. 2005, 17, 1903-1910.
32. Comon, P. Independent component analysis, a new concept? Signal Process. 1994, 36, 287-314.
33. van der Vaart, A.W. Asymptotic Statistics; Cambridge University Press: New York, NY, USA, 1998.
34. Hilbert, D. Über die Darstellung definiter Formen als Summe von Formenquadraten. Math. Ann. 1888, 32, 342-350.
35. Ahmadi, A.A.; Parrilo, P.A. A positive definite polynomial hessian that does not factor. In Proceedings of the Joint 48th IEEE Conference on Decision and Control and 28th Chinese Control Conference, Shanghai, China, 15-18 December 2009; pp. 16-18.
36. Wolfram, S. The Mathematica Book, 3rd ed.; Cambridge University Press: Cambridge, UK, 1996.
37. Bocquet, M.; Pires, C.; Lin, W. Beyond Gaussian statistical modeling in geophysical data assimilation. Mon. Wea. Rev. 2010, 138, 2997-3023.
38. Sura, P.; Newman, M.; Penland, C.; Sardeshmuck, P. Multiplicative noise and non-Gaussianity: A paradigm for atmospheric regimes? J. Atmos. Sci. 2005, 62, 1391-1406.
39. Guo, D.; Shamai, S.; Verdú, S. Mutual information and minimum mean-square error in Gaussian channels. IEEE Trans. Inform. Theor. 2005, 51, 1261-1283.
40. Rioul, O. A simple proof of the entropy-power inequality via properties of mutual information. arXiv 2007, arXiv:cs/0701050v2 [cs.IT].
41. Guo, D.; Wu, Y.; Shamai, S.; Verdú, S. Estimation in gaussian noise: Properties of the minimum mean-square error. IEEE T. Inform. Theory 2011, 57, 2371-2385.
42. Gilbert, J.; Lemaréchal, C. Some numerical experiments with variable-storage quasi-Newton algorithms. Math. Program. 1989, 45, 407-435.