Estimation of information theoretic measures for continuous random variables

Advances in Neural Information Processing Systems 21 - Proceedings of the 2008 Conference

Volumn , Issue , 2009, Pages 1257-1264

  Pérez Cruz, Fernando ^a  

^a Princeton University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

MOTION COMPENSATION; NEAREST NEIGHBOR SEARCH;

DIFFERENTIAL ENTROPY; HILBERT-SCHMIDT INDEPENDENCE CRITERIONS; INDEPENDENCE TESTS; INFORMATION THEORETIC MEASURE; K NEAREST NEIGHBOR DENSITY ESTIMATION; KULLBACK LEIBLER DIVERGENCE; MUTUAL INFORMATIONS; NUMBER OF SAMPLES; TWO-SAMPLE PROBLEMS; TWO-SAMPLE TESTS;

RANDOM VARIABLES;

EID: 84858766275     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (26)

1. N. Anderson, P. Hall, and D. Titterington. Two-sample test statistics for measuring discrepancies between two multivariate probability density functions using kernel-based density estimates. Journal of Multivariate Analysis, 50(1):41-54, 7 1994.
2. F. R. Bach and M. I. Jordan. Kernel independent component analysis. JMLR, 3:1-48, 2004.
3. K. Balakrishnan and A. P. Basu. The Exponential Distribution: Theory, Methods and Applications. Gordon and Breach Publishers, Amsterdam, Netherlands, 1996.
4. J. Beirlant, E. Dudewicz, L. Gyorfi, and E. van der Meulen. Nonparametric entropy estimation: An overview. International Journal of the Mathematical Statistics Sciences, pages 17-39, 1997.
5. T. M. Cover and J. A. Thomas. Elements of Information Theory. Wiley, New York, USA, 1991.
6. G. A. Darbellay and I. Vajda. Estimation of the information by an adaptive partitioning of the observation space. IEEE Trans. Information Theory, 45(4):1315-1321, 5 1999.
7. R. L. Dobrushin. A simplified method for experimental estimate of the entropy of a stationary sequence. Theory of Probability and its Applications, (4):428-430, 1958.
8. F. Fleuret. Fast binary feature selection with conditional mutual information. JMLR, 5:1531-1555, 2004.
9. A. Gretton, K. M. Borgwardt, M. Rasch, B. Schölkopf, and A. Smola. A kernel method for the twosample-problem. In B. Schölkopf, J. Platt, and T. Hofmann, editors, Advances in Neural Information Processing Systems 19, Cambridge, MA, 2007. MIT Press.
10. A. Gretton, K. Fukumizu, C. H. Teo, L. Song, B. Schölkopf, and A. Smola. A kernel statistical test of independence. In J.C. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems 20, Cambridge, MA, 2008. MIT Press.
11. G.R. Grimmett and D.R. Stirzaker. Probability and Random Processes. Oxford University Press, Oxford, UK, 3 edition, 2001.
12. Julian Havil. Gamma: Exploring Euler's Constant. Princeton University Press, New York, USA, 2003.
13. S. Mallela I. S. Dhillon and R. Kumar. A divisive information-theoretic feature clustering algorithm for text classification. JMLR, 3:1265-1287, 3 2003.
14. Leonard Kleinrock. Queueing Systems. Volume 1: Theory. Wiley, New York, USA, 1975.
15. L. F. Kozachenko and N. N. Leonenko. Sample estimate of the entropy of a random vector. Problems Inform. Transmission, 23(2):95-101, 4 1987. (Pubitemid 18531046)
16. A. Kraskov, H. Stögbauer, and P. Grassberger. Estimating mutual information. Physical Review E, 69(6):1-16, 6 2004.
17. S. Kullback and R. A. Leibler. On information and sufficiency. Ann. Math. Stats., 22(1):79-86, 3 1951.
18. N. N. Leonenko, L. Pronzato, and V. Savani. A class of renyi information estimators for multidimensional densities. Annals of Statistics, 2007. Submitted.
19. P. J. Moreno, P. P. Ho, and N. Vasconcelos. A kullback-leibler divergence based kernel for svm classification in multimedia applications. Technical Report HPL-2004-4, HP Laboratories, 2004.
20. X. Nguyen, M. J. Wainwright, and M. I. Jordan. Nonparametric estimation of the likelihood ratio and divergence functionals. In IEEE Int. Symp. Information Theory, Nice, France, 6 2007.
21. X. Nguyen, M. J. Wainwright, and M. I. Jordan. Estimating divergence functionals and the likelihood ratio by penalized convex risk minimization. In J.C. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems 20, Cambridge, MA, 2008. MIT Press.
22. L. Paninski. Estimation of entropy and mutual information. Neural Compt, 15(6):1191-1253, 6 2003. (Pubitemid 37049793)
23. C. E. Shannon. A mathematical theory of communication. Bell System Tech. J., pages 379-423, 1948.
24. K. Torkkola. Feature extraction by non parametric mutual information maximization. JMLR, 3:1415-1438, 2003.
25. Q. Wang, S. Kulkarni, and S. Verdú. Divergence estimation of continuous distributions based on datadependent partitions. IEEE Trans. Information Theory, 51(9):3064-3074, 9 2005. (Pubitemid 41430975)
26. Q. Wang, S. Kulkarni, and S. Verdú. A nearest-neighbor approach to estimating divergence between continuous random vectors. In IEEE Int. Symp. Information Theory, Seattle, USA, 7 2006.