Parametric bayesian estimation of differential entropy and relative entropy

Entropy

Volumn 12, Issue 4, 2010, Pages 818-843

  Gupta, Maya ^a   Srivastava, Santosh ^b  

^a University of Washington   (United States)

^b Fred Hutchinson Cancer Research Center   (United States)

Author keywords

Differential entropy; Kullback Leibler; Pareto; Relative entropy; Wishart

Indexed keywords

EID: 77953494524     PISSN: None     EISSN: 10994300     Source Type: Journal    
DOI: 10.3390/e12040818     Document Type: Article

References (30)

1. El Saddik, A.; Orozco, M.; Asfaw, Y.; Shirmohammadi, S.; Adler, A. A novel biometric system for identification and verification of haptic users. IEEE Trans. Instrum. Meas. 2007, 56, 895-906.
2. Choi, H. Adaptive Sampling and Forecasting with Mobile Sensor Networks; PhD Dissertation; MIT: Cambridge, MA, USA, 2009.
3. Moddemeijer, R. On estimation of entropy and mutual information of continuous distributions. Signal Process. 1989, 16, 233-246.
4. Misra, N.; Singh, H.; Demchuk, E. Estimation of the entropy of a multivariate normal distribution. J. Multivariate Anal. 2005, 92, 324-342.
5. Wang, Q.; Kulkarni, S.R.; Verdú, S. Divergence estimation for multi-dimensional densities via k nearest-neighbor distances. IEEE Trans. Inform. Theory 2009, 55, 2392-2405.
6. Ahmed, N.A.; Gokhale, D.V. Entropy expressions and their estimators for multivariate distributions. IEEE Trans. Inform. Theory 1989, pp. 688-692.
7. Beirlant, J.; Dudewicz, E.; Györfi, L.; Meulen, E.V.D. Nonparametric entropy estimation: An overview. Intl. J. Math. Stat. Sci. 1997, 6, 17-39.
8. Nilsson, M.; Kleijn, W.B. On the estimation of differential entropy from data located on embedded manifolds. IEEE Trans. Inform. Theory 2007, 53, 2330-2341.
9. Kozachenko, L.F.; Leonenko, N.N. Sample estimate of entropy of a random vector. Probl. Inform. Transm. 1987, 23, 95-101.
10. Goria, M.N.; Leonenko, N.N.; Mergel, V.V.; Inverardi, P.L. A new class of random vector entropy estimators and its applications in testing statistical hypotheses. J. Nonparametric Stat. 2005, 17, 277-297.
11. n-Nearest neighbor estimators of entropy. Math. Method. Stat. 2008, 17, 261-277.
12. Hero, A.; Michel, O. Asymptotic theory of greedy approximations to minimal k-point random graphs. IEEE Trans. Inform. Theory 1999, 45, 1921-1939.
13. Hero, A.; Ma, B.; Michel, O.; Gorman, J. Applications of entropic spanning graphs. IEEE Signal Process. Mag. 2002, 19, 85-95.
14. Costa, J.; Hero, A. Geodesic entropic graphs for dimension and entropy estimation in manifold learning. IEEE Trans. Signal Process. 2004, 52, 2210-2221.
15. Hulle, M.M.V. Edgeworth approximation of multivariate differential entropy. Neural Comput. 2005, 17, 1903-1910.
16. Wang, Q.; Kulkarni, S.R.; Verdú, S. Divergence estimation of continuous distributions based on data-dependent partitions. IEEE Trans. Inform. Theory 2005, 51, 3064-3074.
17. Nguyen, X.; Wainwright, M.J.; Jordan, M.I. Estimating divergence functional and the likelihood ratio by penalized convex risk minimization. Advances Neural Inform. Process. Syst. 2007.
18. Wang, Q.; Kulkarni, S.R.; Verdú, S. A nearest-neighbor approach to estimating divergence between continuous random vectors. In Proceedings of the 2006 IEEE International Symposium on Information Theory, Seattle, WA, USA, 9-14 July 2006; IEEE: Washington, DC, USA, 2006.
19. Pérez-Cruz, F. Estimation of information-theoretic measures for continuous random variables. Adv. Neural Inform. Process. Syst. (NIPS) 2009.
20. Lehmann, E.L.; Casella, G. Theory of Point Estimation; Springer: New York, NY, USA, 1998; Chapter 4.
21. Bregman, L. The relaxation method of finding the common points of convex sets and its application to the solution of problems in convex programming. USSR Comput. Math. Math. Phys. 1967, 7, 200-217.
22. Banerjee, A.; Guo, X.; Wang, H. On the optimality of conditional expectation as a Bregman predictor. IEEE Trans. Inform. Theory 2005, 51, 2664-2669.
23. Jones, L.K.; Byrne, C.L. General entropy criteria for inverse problems, with applications to data compression, pattern classification, and cluster analysis. IEEE Trans. Inform. Theory 1990, 36, 23-30.
24. Frigyik, B.A.; Srivastava, S.; Gupta, M.R. Functional Bregman divergence and Bayesian estimation of distributions. IEEE Trans. Inform. Theory 2008, 54, 5130-5139.
25. Amari, S.; Nagaoka, H. Methods of Information Geometry; Oxford University Press: New York, NY, USA, 2000.
26. Kass, R.E. The geometry of asymptotic inference. Stat. Sci. 1989, 4, 188-234.
27. Srivastava, S.; Gupta, M.R.; Frigyik, B.A. Bayesian quadratic discriminant analysis. J. Mach. Learn. Res. 2007, 8, 1287-1314.
28. Havil, J. Gamma; Princeton University Press: Princeton, NJ, USA, 2003.
29. Bercher, J.; Vignat, C. Estimating the entropy of a signal with applications. IEEE Trans. Signal Process. 2000, 48, 1687-1694.
30. Bilodeau, M.; Brenner, D. Theory of Multivariate Statistics; Springer Texts in Statistics: New York, NY, USA, 1999.