On the estimation of differential entropy from data located on embedded manifolds

IEEE Transactions on Information Theory

Volumn 53, Issue 7, 2007, Pages 2330-2341

  Nilsson, Mattias ^a   Kleijn, W Bastiaan ^b  

^a MICROSOFT   (United States)

^b ROYAL INSTITUTE OF TECHNOLOGY   (Sweden)

Author keywords

Convergence rate; Manifolds; Nearest neighbor distance; Shannon differential entropy

Indexed keywords

HIGH-RATE QUANTIZATION THEORY; NEAREST-NEIGHBOR ENTROPY ESTIMATOR; RENYI ENTROPY; SHANNON ENTROPY;

ESTIMATION; MEAN SQUARE ERROR; RANDOM VARIABLES;

SIGNAL PROCESSING;

EID: 34447316139     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2007.899533     Document Type: Article

References (41)

1. R. Ahlswede and I. Csiszár, "Hypothesis testing with communication constraints," IEEE Trans. Inf. Theory, vol. IT-32, no. 4, pp. 533-542, Jul. 1986.
2. C. A. Holzmann and M. Avaria, "Entropy-aided diagnosis in genetics," IEEE Engineering in Medicine and Biology Mag., vol. 11, no. 3, pp. 35-40, Sep. 1992.
3. N. Ebrahimi, "Testing exponentiality of the residual life, based on dynamic kullback-leibler information," IEEE Trans. Reliab., vol. 47, no. 2, pp. 197-201, Jun. 1998.
4. A. L. Ruhkin, "Testing randomness: A suite of statistical procedures," SIAM, Theory of Probab. its Applications, vol. 45, no. 1, pp. 111-132, 2001.
5. A. Gersho, "Asymptotically optimal block-quantization," IEEE Trans. Inf. Theory, vol. IT-25, no. 5, pp. 373-380, Sep. 1979.
6. R. M. Gray and D. L. Neuhoff, "Quantization," IEEE Trans. Inf. Theory, vol. 44, no. 6, pp. 2325-2383, Oct. 1998.
7. P. Hedelin and J. Skoglund, "Vector quantization based on Gaussian mixture models," IEEE Trans. Speech Audio Process., vol. 8, pp. 385-401, Aug. 2000.
8. B. V. Bonnlander and A. S. Weigend, "Selecting input variables using mutual information and nonparametric density estimation," in Proc. Int. Symp. Artificial Neural Networks, Tainan, Taiwan, dec. 1994, pp. 42-50.
9. R. Battiti, "Using mutual information for selecting features in supervised neural net learning," IEEE Trans. Neural Netw., vol. 5, no. 4, pp. 537-550, Jul. 1994.
10. A. Al-Ani and M. Deriche, "Feature selection using a mutual information based measure," in IEEE Int. Conf. Pattern Recognition, Quebec City, QC, Canada, Aug. 2002, pp. 82-85.
11. K. E. Hild, D. Erdogmus, and J. Principe, "Blind source separation using Renyi's mutual information," IEEE Signal Process. Lett., vol. 8, no. 6, pp. 174-176, Jun. 2001.
12. D. T. Pham, "Mutual information approach to blind separation of stationary sources," IEEE Trans. Inf. Theory, vol. 48, no. 7, pp. 1935-1946, Jul. 2002.
13. C. E. Shannon, "A mathematical theory of communication," Bell Syst. Tech. J., vol. 27, pp. 379-423, 1948.
14. B. W. Silverman, Density Estimation for Statistics and Data Analysis London, U.K.: Chapman and Hall, 1986.
15. R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification. New York: Wiley, 2000.
16. O. Vasicek, "A test for normality based on sample entropy," J. Roy. Statist. Soc., Ser. B, vol. 38, no. 1, pp. 54-59, 1976.
17. E. Miller, "A new class of entropy estimators for multi-dimensional densities," in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing, Hong Kong, Apr. 2003, pp. 297-300.
18. L. F. Kozachenko and N. N. Leonenko, "Sample estimate of entropy of a random vector," Probl. Inf. Transm., vol. 23, no. 1, pp. 95-101, 1987.
19. A. O. Hero, B. Ma, O. Michel, and J. Gorman, "Applications of entropic spanning graphs," IEEE Signal Process. Mag., vol. 19, no. 5, pp. 85-95, May 2002.
20. J. Beirlant, E. Dudewicz, L. Györfi, and E. v. d. Meulen, "Nonparametric entropy estimation: An overview," Int. J. Math. Statist. Sci., vol. 6, no. 1, pp. 17-39, 1997.
21. J. B. Tenenbaum, V. de Silva, and J. C. Langford, "A global geometric framework for nonlinear dimensionality reduction," Science, vol. 290, no. 1, pp. 2319-2323, 2000.
22. S. T. Roweis and L. K. Saul, "Nonlinear dimensionality reduction by locally linear embeddings," Science, vol. 290, no. 1, pp. 2323-2326, 2000.
23. H. S. Seung and D. D. Lee, "The manifold ways of perception," Science, vol. 290, no. 1, pp. 2268-2269, 2000.
24. W. Klein, R. Plomp, and L. C. W. Pols, "Vowel spectra, vowel spaces, and vowel identification," J. Acoust. Soc. Amer., vol. 48, no. 4, pp. 999-1009, 1970.
25. R. Togneri, M. D. Alder, and Y. Attikiouzel, "Dimension and structure of the speech space," Proc. Inst. Elec. Eng.-I: Commun., Speech and Vision, vol. 139, no. 2, pp. 123-127, 1992.
26. J. Costa and A. Hero, "Manifold learning using Euclidean κ-nearest neighbor graphs," in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing, Montreal, QC, Canada, May. 2004, pp. 988-991.
27. J. Costa and A. Hero, "Geodesic entropic graphs for dimension and entropy estimation in manifold learning," IEEE Trans. Signal Process., vol. 52, no. 8, pp. 2210-2221, Aug. 2004.
28. T. M. Cover and J. A. Thomas, Elements of Information Theory. New York: Wiley, 1991.
29. S. Graf and H. Luschgy, Foundations of Quantization for Probability Distributions. Berlin/Heidelberg, Germany: Springer-Verlag, 2000.
30. P. L. Zador, "Asymptotic quantization error of continuous signals and the quantization dimension," IEEE Trans. Inf. Theory, vol. IT-28, no. 2, pp. 139-149, Mar. 1982.
31. R. M. Gray, Source Coding Theory. Norwell, MA: Kluwer Academic, 1990.
32. A. B. Tsybakov and E. C. van der Meulen, "Root-n Consistent Estimators of Entropy for Densities With Unbounded Support," Institute de Statistique, Universitè Catholique de Louvain, 1992, Tech. Rep. Discussion Paper 9206.
33. A. B. Tsybakov and E. C. van der Meulen, "Root-n consistent estimators of entropy for densities with unbounded support," Scand. J. Statist. vol. 23, no. 1, pp. 75-83, 1996.
34. V. Mergel, "On some properties of Kozachenco-Leonenko estimates and maximum entropy principle in goodness of fit tests construction," in The 21st Int. Workshop on Bayesian Inference and Maximum Entropy Methods in Science and Engineering, Hong Kong, Apr. 2001.
35. A. H. Gray and J. D. Markel, "Distance measures for speech processing," IEEE Trans. Acoust., Speech, Signal Process., vol. ASSP-24, no. 5, pp. 380-389, Oct. 1976.
36. Acoustic-Phonetic Continuous Speech Corpus DARPA-TIMIT, 1990, NIST Speech Disc 1-1.1.
37. K.-F. Lee and H.-W. Hon, "Speaker-independent phone recognition using hidden Markov models," IEEE Trans. Acoust., Speech, Signal Process. vol. 37, no. 11, pp. 1641-1648, Nov. 1989.
38. M. D. Alder, R. Togneri, and Y. Attikiouzel, "Dimension of the speech space," Proc. Inst. Elec. Eng.-I : Commun., Speech and Vision, vol. 138, no. 3, pp. 207-214, 1991.
39. W. B. Kleijn and K. K. Paliwal, Speech Coding and Synthesis. Amsterdam, The Netherlands: Elsevier, 1998.
40. P. Hedelin, "Single stage spectral quantization at 20 bits," in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing, Adelaide, Australia, Apr. 1994, pp. 525-528.
41. H. Lebesgue, "Sur l'intégration des fonctions discontinues," Ann. Ecole Norm., vol. 27, no. 1, pp. 361-450, 1910.