Estimation of Rényi entropy and mutual information based on generalized nearest-neighbor graphs

Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010, NIPS 2010

Volumn , Issue , 2010, Pages

  Pál, Dávid ^a   Póczos, Barnabás ^b   Szepesvári, Csaba ^a  

^a UNIVERSITY OF ALBERTA   (Canada)

^b Carnegie Mellon University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

INFORMATION THEORY; NEAREST NEIGHBOR SEARCH;

COMPUTATIONALLY EFFICIENT; CONTINUOUS DISTRIBUTION; EUCLIDEAN; GENERALISED NEAREST-NEIGHBOUR; MUTUAL INFORMATIONS; NEIGHBOR GRAPH; NONPARAMETRIC ESTIMATORS; POWER; SIMPLE++; UPPER BOUND;

ENTROPY;

EID: 85161981737     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (37)

1. C. Adami. Information theory in molecular biology. Physics of Life Reviews, 1:3-22, 2004.
2. M. Aghagolzadeh, H. Soltanian-Zadeh, B. Araabi, and A. Aghagolzadeh. A hierarchical clustering based on mutual information maximization. In in IEEE ICIP, pages 277-280, 2007.
3. P. A. Alemany and D. H. Zanette. Fractal random walks from a variational formalism for Tsallis entropies. Phys. Rev. E, 49(2):R956-R958, Feb 1994.
4. J. Cardoso. Multidimensional independent component analysis. Proc. ICASSP'98, Seattle, WA., 1998.
5. B. Chai, D. B.Walther, D. M. Beck, and L. Fei-Fei. Exploring functional connectivity of the human brain using multivariate information analysis. In NIPS, 2009.
6. J. A. Costa and A. O. Hero. Entropic graphs for manifold learning. In IEEE Asilomar Conf. on Signals, Systems, and Computers, 2003.
7. J. Dedecker, P. Doukhan, G. Lang, J.R. Leon, S. Louhichi, and C Prieur. Weak Dependence: With Examples and Applications, volume 190 of Lecture notes in Statistics. Springer, 2007.
8. M. N. Goria, N. N. Leonenko, V. V. Mergel, and P. L. Novi Inverardi. A new class of random vector entropy estimators and its applications in testing statistical hypotheses. Journal of Nonparametric Statistics, 17: 277-297, 2005.
9. A. O. Hero and O. J. Michel. Asymptotic theory of greedy approximations to minimal k-point random graphs. IEEE Trans. on Information Theory, 45(6):1921-1938, 1999.
10. A. O. Hero, B. Ma, O. Michel, and J. Gorman. Alpha-divergence for classification, indexing and retrieval, 2002a. Communications and Signal Processing Laboratory Technical Report CSPL-328.
11. A. O. Hero, B. Ma, O. Michel, and J. Gorman. Applications of entropic spanning graphs. IEEE Signal Processing Magazine, 19(5):85-95, 2002b.
12. K. Hlaváckova-Schindler, M. Paluŝb, M. Vejmelkab, and J. Bhattacharya. Causality detection based on information-theoretic approaches in time series analysis. Physics Reports, 441:1-46, 2007.
13. M. M. Van Hulle. Constrained subspace ICA based on mutual information optimization directly. Neural Computation, 20:964-973, 2008.
14. A. Hyvärinen, J. Karhunen, and E. Oja. Independent Component Analysis. John Wiley, New York, 2001.
15. Y. Koo and S. Lee. Rates of convergence of means of Euclidean functionals. Journal of Theoretical Probability, 20(4):821-841, 2007.
16. L. F. Kozachenko and N. N. Leonenko. A statistical estimate for the entropy of a random vector. Problems of Information Transmission, 23:9-16, 1987.
17. A. Kraskov, H. Stögbauer, and P. Grassberger. Estimating mutual information. Phys. Rev. E, 69:066138, 2004.
18. J. Kybic. Incremental updating of nearest neighbor-based high-dimensional entropy estimation. In Proc. Acoustics, Speech and Signal Processing, 2006.
19. E. Learned-Miller and J. W. Fisher. ICA using spacings estimates of entropy. Journal of Machine Learning Research, 4:1271-1295, 2003.
20. N. Leonenko, L. Pronzato, and V. Savani. A class of Rényi information estimators for multidimensional densities. Annals of Statistics, 36(5):2153-2182, 2008.
21. J. Lewi, R. Butera, and L. Paninski. Real-time adaptive information-theoretic optimization of neurophysiology experiments. In Advances in Neural Information Processing Systems, volume 19, 2007.
22. D. Pál, Cs. Szepesvári, and B. Póczos. Estimation of Rényi entropy and mutual information based on generalized nearest-neighbor graphs, 2010. http://arxiv.org/abs/1003.1954.
23. H. Peng and C. Ding. Feature selection based on mutual information: Criteria of max-dependency, maxrelevance, and min-redundancy. IEEE Trans On Pattern Analysis and Machine Intelligence, 27, 2005.
24. B. Póczos and A. Lörincz. Independent subspace analysis using geodesic spanning trees. In ICML, pages 673-680, 2005.
25. B. Póczos and A. Lörincz. Identification of recurrent neural networks by Bayesian interrogation techniques. Journal of Machine Learning Research, 10:515-554, 2009.
26. B. Póczos, S. Kirshner, and Cs. Szepesvári. REGO: Rank-based estimation of Rényi information using Euclidean graph optimization. In AISTATS 2010, 2010.
27. C. Redmond and J. E. Yukich. Asymptotics for Euclidean functionals with power-weighted edges. Stochastic processes and their applications, 61(2):289-304, 1996.
28. D. W. Scott. On optimal and data-based histograms. Biometrika, 66:605-610, 1979.
29. C. Shan, S. Gong, and P. W. Mcowan. Conditional mutual information based boosting for facial expression recognition. In British Machine Vision Conference (BMVC), 2005.
30. J. M. Steele. Probability Theory and Combinatorial Optimization. Society for Industrial and Applied Mathematics, 1997.
31. Z. Szabó, B. Póczos, and A. Lörincz. Undercomplete blind subspace deconvolution. Journal of Machine Learning Research, 8:1063-1095, 2007.
32. A. B. Tsybakov and E. C. van der Meulen. Root-n consistent estimators of entropy for densities with unbounded support. Scandinavian Journal of Statistics, 23:75-83, 1996.
33. O. Vasicek. A test for normality based on sample entropy. Journal of the Royal Statistical Society, Series B, 38:54-59, 1976.
34. Q. Wang, S. R. Kulkarni, and S. Verdú. Universal estimation of information measures for analog sources. Foundations and Trends in Communications and Information Theory, 5(3):265-352, 2009a.
35. Q. Wang, S. R. Kulkarni, and S. Verdú. Divergence estimation for multidimensional densities via κ-nearestneighbor distances. IEEE Transactions on Information Theory, 55(5):2392-2405, 2009b.
36. E. Wolsztynski, E. Thierry, and L. Pronzato. Minimum-entropy estimation in semi-parametric models. Signal Process., 85(5):937-949, 2005.
37. J. E. Yukich. Probability Theory of Classical Euclidean Optimization Problems. Springer, 1998.