Universal estimation of information measures for analog sources

Foundations and Trends in Communications and Information Theory

Volumn 5, Issue 3, 2008, Pages 265-353

  Wang, Qing ^a   Kulkarni, Sanjeev R ^b   Verd, Sergio ^b  

^a Credit Suisse USA   (United States)

^b Princeton University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

INFORMATION MEASURES; MUTUAL INFORMATIONS; NEAREST-NEIGHBORS; PLUG-INS; SPEED OF CONVERGENCE; UNIVERSAL ESTIMATION;

ESTIMATION;

CONVERGENCE OF NUMERICAL METHODS;

EID: 68349141847     PISSN: 15672190     EISSN: 15672328     Source Type: Journal    
DOI: 10.1561/0100000021     Document Type: Article

References (272)

1. C. Adami, "Information theory in molecular biology," Physics of Life Reviews, vol. 1, pp. 3-22, Online. Available: http://arxiv.org/abs/ q-bio/0405004, 2004.
2. T. M. Adams and A. B. Nobel, "On density estimation from ergodic processes," The Annals of Probability, vol. 26, no. 2, pp. 794-804, April 1998.
3. M. Aghagolzadeh, H. Soltanian-Zadeh, and B. N. Araabi, "New informationbased clustering method using Rényi's entropy and fuzzy C-means clustering," Proceeding of Signal and Image Processing, pp. 411-414, 2005.
4. R. Ahlswede and I. Csiszár, "Common randomness in information theory and cryptography - Part I: Secrete sharing," IEEE Transactions on Information Theory, vol. 39, no. 4, pp. 1121-1132, July 1993.
5. I. Ahmad and P. Lin, "A nonparametric estimation of the entropy for absolutely continuous distributions," IEEE Transactions on Information Theory, vol. 22, no. 3, pp. 372-375, May 1976.
6. N. A. Ahmed and D. V. Gokhale, "Entropy expressions and their estimators for multivariate distributions," IEEE Transactions on Information Theory, vol. 35, no. 3, pp. 688-692, May 1989.
7. H. M. Aktulga, I. Kontoyiannis, L. A. Lyznik, L. Szpankowski, A. Y. Grama, and W. Szpankowski, "Identifying statistical dependence in genomic sequences via mutual information estimates," EURASIP Journal on Bioinformatics and Systems Biology, no. 3, id. 14741, July 2007.
8. H. M. Aktulga, I. Kontoyiannis, L. A. Lyznik, L. Szpankowski, A. Y. Grama, and W. Szpankowski, "Statistical dependence in biological sequences," Proceedings of 2007 International Symposium on Information Theory (ISIT2007), pp. 2676-2680, June 2007.
9. V. R. Algazi, "Useful approximations to optimum quantization," IEEE Transactions on Communications Technology, vol. COM-14, pp. 297-301, 1966.
10. S. M. Ali and S. D. Silvey, "A general class of coefficients of divergence of one distribution from another," Journal of the Royal Statistical Society. Series B (Methodological), vol. 28, no. 1, pp. 131-142, 1966.
11. B. D. O. Anderson, J. B. Moore, and R. M. Hawkes, "Model approximation via prediction error identification," Automatica, vol. 14, pp. 615-622, November 1978.
12. O. Arandjelovic, G. Shakhnarovich, J. Fisher, R. Cipolla, and T. Darrell, "Face recognition with image sets using manifold density divergence," Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), vol. 1, pp. 581-588, 2005.
13. S. Artstein, K. Ball, F. Barthe, and A. Naor, "Solution of Shannon's problem on the monotonicity of entropy," Journal of the American Mathematical Society, vol. 17, pp. 975-982, 2004.
14. S. Arya, "Nearest neighbor searching and applications," Ph.D. Thesis, no. CS-TR-3490, University of Maryland, June, 1995.
15. J. A. Aslam, "Improving algorithms for boosting," Proceedings of the 13th Annual Conference on Computational Learning Theory, pp. 200-207, 2000.
16. S. P. Awate, T. Tasdizen, R. T. Whitaker, and N. L. Foster, "Adaptive nonparametric Markov modeling for unsupervised MRI brain-tissue classification," Medical Image Analysis, vol. 10, no. 5, pp. 726-739, 2006.
17. Y. Bao and H. Krim, "Rényi entropy based divergence measures for ICA," Proceedings of 2003 IEEE Workshop on Statistical Signal Processing, no. 28, pp. 565-568, 2003.
18. A. Bardera, M. Feixas, I. Boada, and M. Sbert, "Compression-based image registration," Proceedings of the 2006 IEEE International Symposium on Information Theory (ISIT2006), pp. 436-440, July 2006.
19. A. R. Barron, "Entropy and the central limit theorem," Annals of Probability, vol. 14, pp. 336-342, 1986.
20. M. Basseville, "Distance measures for signal processing and pattern recognition," Signal Processing, vol. 18, no. 4, pp. 349-369, December 1989.
21. H. Beigi, S. Maes, and J. Sorensen, "A distance measure between collections of distributions and its application to speaker recognition," Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP'98), vol. 2, pp. 753-756, 1998.
22. J. Beirlant, "Limit theory for spacing statistics from general univariate distributions," Publications de l'Institut de Statistique de l'Université de Paris XXXI fasc, vol. 1, pp. 27-57, 1986.
23. J. Beirlant, E. J. Dudewicz, L. Györfi, and E. van der Meulen, "Nonparametric entropy estimation: An overview," The International Journal of Mathematics and Statistical Sciences, vol. 6, no. 1, pp. 17-39, June 1997.
24. J. Beirlant and M. C. A. Zuijlen, "The empirical distribution function and strong laws for functions of order statistics of uniform spacings," Journal of Multivariate Analysis, vol. 16, pp. 300-317, 1985.
25. J. D. Bekenstein, "Information in the holographic universe," Scientific American, pp. 59-65, August 2003.
26. D. Benedetto, E. Caglioti, and V. Loreto, "Language trees and zipping," Physical Review Letters, vol. 88, no. 4, id. 048702, January 2002.
27. C. H. Bennett and P. Shor, "Quantum information theory," IEEE Transactions on Information Theory, vol. 44, no. 6, pp. 2724-2742, October 1998.
28. J. L. Bentley, "Multidimensional binary search trees used for associative searching," Communications of the ACM, vol. 18, no. 9, pp. 509-517, 1975.
29. T. Berger, "Optimum quantizers and permutation codes," IEEE Transactions on Information Theory, vol. 18, pp. 759-765, November 1972.
30. T. Berger, "Minimum entropy quantizers and permutation codes," IEEE Transactions on Information Theory, vol. 28, pp. 149-157, March 1982.
31. P. K. Bhattacharya, "Efficient estimation of a shift parameter from grouped data," The Annals of Mathematical Statistics, vol. 38, no. 6, pp. 1770-1787, December 1967.
32. J. Binia, "On the capacity of certain additive non-Gaussian channels," IEEE Transactions on Information Theory, vol. 25, pp. 448-452, July 1979.
33. J. Binia, "New bounds on the capacity of certain infinite-dimensional additive non-Gaussian channels," IEEE Transactions on Information Theory, vol. 51, pp. 1218-1221, March 2005.
34. J. Binia, "On divergence-power inequalities," IEEE Transactions on Information Theory, vol. 53, no. 3, pp. 1179-1182, March 2007.
35. I. Bocharov and P. Luki, "Application of Kullback-Leibler metric to speech recognition," Nizhny Novgorod Linguistic University, Russia, Technical Report, 2003.
36. A. Borst and F. E. Theunissen, "Information theory and neural coding," Nature Neuroscience, vol. 2, pp. 947-957, 1999.
37. D. R. Brillinger and A. Guha, "Mutual information in the frequency domain," Journal of Statistical Planning and Inference, vol. 137, no. 3, pp. 1076-1084, March 2007.
38. L. G. Brown, "A survey of image registration techniques," ACM Computing Surveys, vol. 24, pp. 325-376, December 1992.
39. P. W. Buchen and M. Kelly, "The maximum entropy eistribution of an asset inferred from option prices," The Journal of Financial and Quantitative Analysis, vol. 31, pp. 143-159, March 1996.
40. J. P. Burg, "Maximum entropy spectral analysis," Ph.D. thesis, Department of Geophysics, Stanford University, Stanford, CA, 1975.
41. A. Butte and I. S. Kohane, "Mutual information relevance networks: Functional genomic clustering using pairwise entropy measurements," Proceedings of the Pacific Symposium on Biocomputing, vol. 5, pp. 415-426, 2000.
42. T. Cacoullus, "Estimation of a multivariate density," The Annals of the Institute of Statistical Mathematics, vol. 18, p. 179, 1966.
43. H. Cai, S. R. Kulkarni, and S. Verdú, "Universal entropy estimation via block sorting," IEEE Transactions on Information Theory, vol. 50, no. 7, pp. 1551- 1561, July 2004.
44. H. Cai, S. R. Kulkarni, and S. Verdú, "Universal divergence estimation for finite-alphabet sources," IEEE Transactions on Information Theory, vol. 52, no. 8, pp. 3456-3475, August 2006.
45. A. Carbonez, L. Györfi, and E. C. van der Meulen, "Nonparametric entropy estimation based on randomly censored data," Problems of Control and Inforrmation Theory, vol. 20, pp. 441-451, 1991.
46. H. Chen, P. K. Varshney, and M. K. Arora, "Performance of mutual information similarity measure for registration of multi-temporal remote sensing images," IEEE Transactions on Geoscience and Remote Sensing, vol. 41, no. 11, part. 1, pp. 2445-2454, November 2003.
47. B. C. Cheng and P. M. Robinson, "Density estimation in strongly dependent non-lineartime series," Statistica Sinica, vol. 1, pp. 335-359, 1991.
48. B. S. Choi and T. M. Cover, "An information-theoretic proof of Burg's maximum entropy spectrum," Proceedings of IEEE, vol. 72, pp. 1094-1095, 1984.
49. R. L. Cilibrasi and P. M. Vitányi, "Clustering by compression," IEEE Transactions on Information Theory, vol. 51, no. 4, pp. 1523-1545, April 2005.
50. P. Comon, "Independent component analysis, a new concept?," Signal Processing, vol. 36, pp. 287-314, 1994.
51. T. Cover and R. King, "A convergent gambling estimate of the entropy of English," IEEE Transactions on Information Theory, vol. 24, no. 4, pp. 413- 421, July 1978.
52. T. M. Cover, "Universal portfolios," Mathematical Finance, vol. 1, no. 1, pp. 1- 29, January 1991.
53. T. M. Cover, "Shannon and investment," IEEE Information Theory Newsletter (Special Golden Jubilee Issue), pp. 10-11, June 1998.
54. T. M. Cover and P. E. Hart, "Nearest neighbor pattern classification," IEEE Transactions on Information Theory, vol. 13, no. 1, pp. 21-27, January 1967.
55. T. M. Cover and J. A. Thomas, Elements of Information Theory. Wiley Series in Telecommunications, 1991.
56. T. M. Cover and J. A. Thomas, Elements of Information Theory. Wiley Series in Telecommunications, Second ed., 2005.
57. H. Cramér, "On some classes of series used in mathematical statistics," Proceedings of the 6th Scandinavian Congress of Mathematicians, pp. 399-425, 1925.
58. N. Cressie, "On the logarithms of higher order spacings," Biometrika, vol. 63, pp. 343-355, 1976.
59. I. Csiszár, "Information-type measures of difference of probability distributions and indirect observations," Studia Scientiarum Mathematicarum Hungarica, vol. 2, pp. 229-318, 1967.
60. I. Csiszár, "On generalized entropy," Studia Scientiarum Mathematicarum Hungarica, vol. 4, pp. 401-419, 1969.
61. I. Csiszár, "Generalized entropy and quantization problems," Transactions of the 6th Praque Conference on Information Theory, Statistical Decision Functions, Random Processes (Academia, Prague), 1973.
62. I. Csiszár, "I-divergence geometry of probability distributions and minimization problems," The Annals of Probability, vol. 3, no. 1, pp. 146-158, February 1975.
63. I. Csiszár and J. Korner, Information Thoery: Coding Theorems for Discrete Memoryless Systems. New York: Academic Press, 1982.
64. I. Csiszár and P. C. Shields, "Information theory and statistics: A tutorial," Foundations and Trends in Communications and Information Theory, vol. 1, no. 4, pp. 417-528, December 2004.
65. G. A. Darbellay and I. Vajda, "Estimation of the information by an adaptive partitioning of the observation space," IEEE Transactions on Information Theory, vol. 45, no. 4, pp. 1315-1321, May 1999.
66. G. A. Darbellay and I. Vajda, "Entropy expressions for multivariate continuous distributions," IEEE Transactions on Information Theory, vol. 46, no. 2, pp. 709-712, March 2000.
67. B. V. Dasarathy, Nearest Neighbor Pattern Classification Techniques. IEEE Computer Society Press, 1991.
68. T. Dasu, S. Krishnan, S. Venkatasubramanian, and K. Yi, "An informationtheoretic approach to detecting changes in multi-dimensional data streams," Proceedings of the 38th Symposium on the Interface of Statistics, Computing Science, and Applications (Interface '06), Pasadena, CA, May 2006.
69. A. Dembo, T. M. Cover, and J. A. Thomas, "Information theoretic inequalities," IEEE Transactions on Information Theory, vol. 37, no. 6, pp. 1501-1518, November 1991.
70. L. Devroye, "A course in density estimation," in Progress in Probability and Statistics, vol. 14, Birkhäuser, 1987.
71. L. Devroye and L. Györfi, Nonparametric Density Estimation, the L1 View. Wiley Series in Probability and Mathematical Statistics, 1985.
72. L. Devroye, L. Györfi, A. Krzyzak, and G. Lugosi, "On the strong universal consistency of nearest neighbor regression function estimates," The Annals of Statistics, vol. 22, no. 3, pp. 1371-1385, September 1994.
73. L. Devroye, L. Györfi, and G. Lugosi, A Probablistic Theory of Pattern Recognition. Springer, 1996.
74. L. P. Devroye and T. J. Wagner, "The strong uniform consistency of nearest neighbor density estimates," The Annals of Statistics, vol. 5, no. 3, pp. 536- 540, May 1977.
75. A. Dionîsio, R. Menezes, and D. A. Mendes, "Entropy-based independence test," Nonlinear Dynamics, vol. 44, no. 1-4, pp. 351-357, June 2006.
76. Y. G. Dmitriev and F. P. Tarasenko, "On the estimation of functionals of the probability density and its derivatives," Thoery of Probability and Its Applications, vol. 18, no. 3, pp. 628-633, 1974.
77. M. N. Do and M. Vetterli, "Wavelet-based texture retrieval using generalized Gaussian density and Kullback-Leibler distance," IEEE Transactions on Image Processing, vol. 11, no. 2, pp. 146-158, February 2002.
78. N. Ebrahimi, M. Habibullah, and E. S. Soofi, "Testing exponentiality based on Kullback-Leibler information," Journal of the Royal Statistical Society. Series B (Methodological), vol. 54, no. 3, pp. 739-748, 1992.
79. K. Eckschlager, Information Theory in Analytical Chemistry. New York: Wiley, 1994.
80. F. Y. Edgeworth, "The generalised law of error, or law of great numbers," Journal of the Royal Statistical Society, vol. 69, no. 3, pp. 497-539, September 1906.
81. I. Eidhammer, I. Jonassen, and W. R. Taylor, Protein Bioinformatics: An Algorithmic Approach to Sequence and Structure. John Wiley and Sons, 2004.
82. Y. Ephraim and N. Merhav, "Hidden Markov process," IEEE Transactions on Information Theory, vol. 48, pp. 1518-1569, June 2002.
83. W. J. Ewens and G. R. Grant, Statistical Methods in Bioinformatics: An Introduction. Springer, 2005.
84. P. Favaro and S. Soatto, "Shape and radiance estimation from the information divergence of blurred images," Proceedings of the 6th European Conference on Computer Vision - Part I, pp. 755-768, 2000.
85. A. A. Fedotov, P. Harremoes, and F. Topsoe, "Refinements of Pinsker's inequality," IEEE Transactions on Information Theory, vol. 49, no. 6, pp. 1491-1498, June 2003.
86. M. Fernandes, "Nonparametric entropy-based tests of independence between stochastic processes," Ph.D. dissertation, the Solvay Business School, Universit é Libre de Bruxelles, 2000.
87. E. Fix and J. L. Hodges, "Discriminatory analysis, nonparametric discrimination: Consistency properties," USAF School of Aviation Medicine, Randolph Field, TX, USA, Technical Report 4, Project Number 21-49-004, 1951.
88. A. M. Fraser and H. L. Swinney, "Independent coordinates or strange attractors from mutual information," Physical Review A, vol. 33, no. 2, pp. 1134- 1140, February 1986.
89. K. Fukunaga and L. D. Hostetler, "Optimization of k-nearest-neighbor density estimates," IEEE Transactions on Information Theory, vol. 20, no. 5, pp. 320- 326, May 1973.
90. K. Fukunaga and P. M. Nerada, "A branch and bound algorithm for computing k-nearest neighbors," IEEE Transactions on Computers, vol. 24, pp. 750- 753, July 1975.
91. A. Gersho, "Asymptotically optimal block quantization," IEEE Transactions on Information Theory, vol. 25, pp. 373-380, July 1979.
92. H. Gish and J. N. Pierce, "Asymptotically efficient quantizing," IEEE Transactions on Information Theory, vol. 14, pp. 676-683, September 1968.
93. M. Godavarti and A. Hero, "Convergence of differential entropies," IEEE Transactions on Information Theory, vol. 50, no. 2, pp. 171-176, January 2004.
94. E. Gokcay and J. C. Principe, "Information theoretic clustering," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 2, pp. 158- 171, February 2002.
95. M. H. Goldwasser, D. S. Johnson, and C. C. McGeoch, Data Structures, Near Neighbor Searches, and Methodology: 5th and 6th DIMACS Implementation Challenge. American Mathematical Society, 2002.
96. M. N. Goria, N. N. Leonenko, V. V. Mergel, and P. L. Novi Inverardi, "A new class of random vector entropy estimators and its applications in testing statistical hypotheses," Journal of Nonparametric Statistics, vol. 17, no. 3, pp. 277-297, April 2005.
97. C. Granger and J. L. Lin, "Using the mutual information coefficient to identify lags in nonlinear models," The Journal of Time Series Analysis, vol. 15, no. 4, pp. 371-384, 1994.
98. R. M. Gray and D. L. Neuhoff, "Quantization," IEEE Transactions on Information Theory, vol. 44, pp. 1-63, October 1998.
99. D. Guo, S. Shamai, and S. Verdú, "Proof of entropy power inequalities via MMSE," in Proceedings of 2006 IEEE International Symposium on Information Theory, pp. 1011-1015, Seattle, WA, July 2006.
100. P. Gurzi, G. Biella, and A. Spalvier, "Estimate of mutual information carried by neuronal responses from small data samples," Network: Computation in Neural Systems, vol. 7, pp. 717-725, 1996.
101. L. Györfi, W. Härdle, P. Sarda, and P. Vieu, Nonparametric Curve Estimation from Time Series. Berlin: Springer-Verlag, 1989.
102. L. Györfi, T. Linder, and E. C. van der Meulen, "On the asymptotic optimality of quantizers," in Proceedings of the 11th Symposium on Information Theory in the Benelux, Noordwijkerhout, pp. 29-35, The Netherlands, October 1990.
103. L. Györfi and G. Lugosi, "Kernel density estimation from ergodic sample is not universally consistent," Computational Statistics and Data Analysis, vol. 14, no. 4, pp. 437-442, November 1992.
104. L. Györfi, G. Morvai, and S. Yakowitz, "Limits to consistent on-line forecasting for ergodic time series," IEEE Transactions on Information Theory, vol. 44, no. 2, pp. 886-892, June 1998.
105. L. Györfi and E. C. van der Meulen, "Density-free convergence properties of various estimators of entropy," Computational Statistics and Data Analysis, vol. 5, no. 4, pp. 425-436, September 1987.
106. P. Hall, "Limit theorems for sums of general functions of m-spacings," Mathematical Proceedings of the Cambridge Philosophical Society, vol. 96, pp. 517-532, 1984.
107. P. Hall, "On Kullback-Leibler loss and density estimation," The Annals of Statistics, vol. 15, pp. 1491-1519, December 1987.
108. P. Hall and S. Morton, "On the estimation of the entropy," The Annals of the Institute of Statistical Mathematics, vol. 45, pp. 69-88, 1993.
109. A. O. Hero, J. Gorman, and O. J. J. Michel, "Image registration with minimum spanning tree algorithm," Proceedings of the 2000 International Conference on Image Processing, vol. 1, pp. 481-484, Vancouver, Canada, September 2000.
110. A. O. Hero, B. Ma, O. J. J. Michel, and J. Gorman, "Applications of entropic spanning graphs," Signal Processing Magazine, vol. 19, no. 5, pp. 85-95, September 2003.
111. A. O. Hero and O. J. J. Michel, "Robust entropy estimation strategies based on edge weighted random graphs," Proceedings of Meeting of the International Society for Optical Engineering (SPIE), pp. 250-261, San Diego, CA, July 1998.
112. A. O. Hero and O. J. J. Michel, "Estimation of Rényi information divergence via pruned minimal spanning trees," Proceedings of 1999 IEEE Workshop on Higher Order Statistics, Caesaria, Israel, June 1999.
113. A. O. Hero and O. J. J. Michel, "Asymptotic theory of greedy approximations to minimal k-point random graphs," IEEE Transactions on Information Theory, vol. 45, no. 6, pp. 1921-1938, September 1999.
114. J. Hidalgo, "Non-parametric estimation with strongly dependent time multivariate time series," Journal of Time Series Analysis, vol. 18, no. 2, pp. 95-122, March 1997.
115. A. Hinneburg, C. C. Aggarwal, and D. A. Keim, "What is the nearest neighbor in high dimensional spaces?," in Proceedings of the 26th VLDB Conference, pp. 506-515, Cairo, Egypt, 2000.
116. H. C. Ho, "On the strong uniform consistency of density estimation for strongly dependent sequences," Statistics and Probability Letters, vol. 22, no. 2, pp. 149-156, February 1995.
117. A. Hyvärinen, "New approximations of differential entropy for independent component analysis and projection pursuit," in Advances in Neural Information Processing System (NIPS'97), vol. 10, pp. 273-279, MIT Press, 1998.
118. A. Hyvärinen, J. Karhunen, and E. Oja, Independent Component Analysis. John Wiley & Sons, 2001.
119. S. Ihara, "On the capacity of channels with additive non-Gaussian noise," Information and Control, vol. 37, pp. 34-39, April 1978.
120. E. T. Jaynes, "Information theory and statistical mechanics," Physical Review, vol. 106, pp. 620-630, May 1957.
121. E. T. Jaynes, "Information theory and statistical mechanics ii," Physical Review, vol. 108, pp. 171-190, October 1957.
122. E. T. Jaynes, "Information theory and statistical mechanics," in Statistical Physics, pp. 181-218, Brandeis Summer Institute 1962, New York, NY: W. A. Benjamin, Inc., 1963.
123. R. Jenssen, K. E. Hild, D. Erdogmus II, J. C. Principe, and T. Eltoft, "Clustering using Rényi's entropy," Proceeding of the 2003 International Joint Conference on Neural Networks, vol. 1, pp. 523-528, July 2003.
124. H. Joe, "On the estimation of entropy and other functionals of a multivariate density," The Annals of the Institute of Statistical Mathematics, vol. 41, no. 4, pp. 683-697, 1989.
125. H. Joe, "Relative entropy measures of multivariate dependence," Journal of the American Statistical Association, vol. 84, no. 405, pp. 171-176, March 1989.
126. D. H. Johnson, C. M. Gruner, K. Baggerly, and C. Seshagiri, "Informationtheoretic analysis of neural coding," Journal of Computational Neuroscience, vol. 10, no. 1, pp. 47-69, January 2001.
127. M. C. Jones and R. Sibson, "What is projection pursuit?," Journal of the Royal Statistical Society. Series A (General), vol. 150, pp. 1-37, 1987.
128. B. Juang and L. R. Rabiner, "A probabilistic distance measure for hidden Markov models," AT&T Technical Journal, vol. 64, pp. 391-408, February 1985.
129. A. Kaltchenko, "Algorithms for estimating information distance with application to bioinformatics and linguistics," Proceedings of Canadian Conference on Electrical and Computer Engineering, vol. 4, pp. 2255-2258, May 2004.
130. D. Kazakos and P. Papantoni-Kazakos, "Spectral distance measures between Gaussian processes," IEEE Transactions on Automatic control, vol. AC-25, no. 5, pp. 950-959, 1980.
131. J. Kim, J. W. Fisher, A. Tsai, C. Wible, A. S. Willsky, and W. M. Wells, "Incorporating spatial priors into an information theoretic approach for fMRI data analysis," Proceedings of the 3rd International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 62-71, October 2000.
132. J. Kivinen and M. K.Warmuth, "Boosting as entropy projection," Proceedings of the 12th Annual Conference on Computational Learning Theory, pp. 133- 144, 1999.
133. I. Kojadinovic, "On the use of mutual information in data analysis: an overview," Proceedings of 11th International Symposium on Applied Stochastic Models and Data Analysis (ASMDA'05), pp. 738-747, May 2005.
134. I. Kontoyiannis, P. H. Algoet, Y. M. Suhov, and A. J. Wyner, "Nonparametric entropy estimation for stationary processes and random fields, with applications to English text," IEEE Transactions on Information Theory, vol. 44, no. 3, pp. 1319-1327, May 1998.
135. L. F. Kozachenko and N. N. Leonenko, "Sample estimate of the entropy of a random vector," Problems of Information Transmission, vol. 23, pp. 95-101, 1987.
136. A. Kraskov, H. Stögbauer, R. G. Andrzejak, and P. Grassberger, "Hierarchical clustering using mutual information," Europhysics Letters, vol. 70, no. 2, pp. 278-284, April 2005.
137. A. Kraskov, H. Stögbauer, and P. Grassberger, "Estimating mutual information," Physical Review E, vol. 69, id. 066138, 2004.
138. C. J. Krebs, Ecological Methodology. Addison Wesley Longman, 1999.
139. R. E. Krichevsky and V. K. Trofimov, "The performance of universal encoding," IEEE Transactions on Information Theory, vol. 27, no. 2, pp. 199-207, March 1981.
140. B. Krishnamurthy, H. V. Madhyastha, and S. Venkatasubramanian, "On stationarity in internet measurements through an information-theoretic lens," Proceedings of the 1st IEEE Workshop on Networking and Database, 2000.
141. S. R. Kulkarni, S. E. Posner, and S. Sandilya, "Data-dependent kn-nn and kernel estimators consistent for arbitrary processes," IEEE Transactions on Information Theory, vol. 48, no. 10, pp. 2785-2788, October 2002.
142. S. Kullback, Information Theory and Statistics. New York: John Wiley and Sons, 1959.
143. S. Kullback, "A lower bound for discrimination information in terms of variation (corresp.)," IEEE Transactions on Information Theory, vol. 13, no. 1, pp. 126-127, January 1967.
144. S. Kullback and R. A. Leibler, "On information and sufficiency," The Annals of Mathematical Statistics, vol. 22, no. 1, pp. 79-86, March 1951.
145. N. Kwak and C. H. Choi, "Input feature selection by mutual information based on Parzen window," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 12, 2002.
146. J. Kybic, "High-dimensional mutual information estimation for image registration," Proceedings of the 2004 International Conference on Image Processing (ICIP '04), vol. 3, pp. 1779-1982, October 2004.
147. R. Landauer, "Information is physical," Physics Today, vol. 44, no. 5, pp. 23-29, May 1991.
148. A. V. Lazo and P. Rathie, "On the entropy of continuous probability distributions (corresp.)," IEEE Transactions on Information Theory, vol. 24, no. 1, pp. 120-122, January 1978.
149. E. Learned-Miller, "Hyperspacings and the estimation of information theoretic quantities," UMass Amherst, Technical Report, 04-104, 2004.
150. H. S. Leff and A. F. Rex, Maxwells Demon: Entropy, Information, Computing. Princeton University Press, 2002.
151. J.-J. Lin, N. Saito, and R. A. Levine, "Edgeworth expansions of the Kullback- Leibler information," Division of Statistics, University of California, Davis, US, Technical Report, 1999.
152. C. Liu and H.-Y. Shum, "Kullback-Leibler boosting," Proceedings of the 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, no. 18-20, pp. I-587-I-594, June 2003.
153. J. Liu and P. Moulin, "Information-theoretic analysis of interscale and intrascale dependencies between image wavelet coefficients," IEEE Transactions on Image Processing, vol. 10, no. 11, pp. 1647-1658, November 2001.
154. D. O. Loftsgaarden and C. P. Quesenberry, "A nonparametric estimate of a multivariate density function," The Annals of Mathematical Statistics, vol. 36, no. 3, pp. 1049-1051, June 1965.
155. B. Logan and A. Salomon, "A music similarity function based on signal analysis," IEEE International Conference on Multimedia and Expo, pp. 745-748, August 2001.
156. S. Lyu, "Infomax boosting," Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 533-538, June 2005.
157. B. Ma, "Parametric and non-parametric approaches for multisensor data fusion," Ph.D. dissertation, EECS Department, University of Michigan, Ann Arbor, MI, 2001.
158. M. Madiman and A. Barron, "Generalized entropy power inequalities and monotonicity properties of information," IEEE Transactions on Information Theory, vol. 53, no. 7, pp. 2317-2329, July 2007.
159. M. Madiman and A. R. Barron, "Generalized entropy power inequalities and monotonicity properties of information," IEEE Transactions on Information Theory, vol. 53, no. 7, pp. 2317-2329, July 2007.
160. F. Maes, D. Vandermeulen, and P. Suetens, "Medical image registration using mutual information," Proceedings of the IEEE, vol. 91, no. 10, pp. 1699-1722, October 2003.
161. N. Mars and G. van Arragon, "Time delay estimation in non-linear systems," IEEE Transactions on Acoustics, Speech and Signal Processing, vol. 20, no. 3, pp. 619-621, June 1981.
162. N. Mars and G. van Arragon, "Time delay estimation in non-linear systems using average amount of mutual information analysis," Signal Processing, vol. 4, no. 2-3, pp. 139-153, 1982.
163. J. R. Mathiassen, A. Skavhaug, and K. Bø, "Texture similarity measure using Kullback-Leibler divergence between Gamma distributions," Proceedings of the 7th European Conference on Computer Vision - Part III, pp. 133-147, 2002.
164. C. R. Maurer and J. M. Fitzpatrick, "A review of medical image registration," in Interactive Image-Guided Neurosurgery, (R. J. Maciunas, ed.), pp. 17-44, Park Ridge, IL: American Association of Neurological Surgeons, 1993.
165. U. Maurer, "Secret key agreement by public discussion," IEEE Transactions on Information Theory, vol. 39, no. 3, pp. 733-742, May 1993.
166. M. Menendez, D. Morales, L. Pardo, and I. Vajda, "Divergence-based estimation and testing of statistical models of classification," Journal of Multivariate Analysis, vol. 54, no. 2, pp. 329-354, 1995.
167. M. E. Meyer and D. V. Gokhale, "Kullback-Leibler information measure for studying convergence rates of densities and distributions," IEEE Transactions on Information Theory, vol. 39, pp. 1401-1404, July 1993.
168. G. Michaels, D. Carr, M. Askenazi, S. Fuhrman, X. Wen, and R. Somogyi, "Cluster analysis and data visualization," Proceedings of Pacific Symposium Biocomputing, vol. 3, pp. 42-53, 1989.
169. E. G. Miller, "A new class of entropy estimators for multi-dimensional densities," Proceedings of 2003 IEEE International Conference on Acoustics, Speech and Signal (ICASSP'03), vol. 3, pp. 297-300, April 2003.
170. R. Moddemeijer, "On estimation of entropy and mutual information of continuous distributions," Signal Processing, vol. 16, no. 3, pp. 233-246, March 1989.
171. R. Moddemeijer, "A statistic to estimate the variance of the histogram based mutual information estimator based on dependent pairs of observations," Signal Processing, vol. 75, no. 1, pp. 51-63, March 1999.
172. A. Mokkadem, "Estimation of the entropy and information of absolutely continuous random variables," IEEE Transactions on Information Theory, vol. 35, no. 1, pp. 193-196, January 1989.
173. Y. Moon, B. Rajagopalan, and U. Lall, "Estimation of mutual information using kernel density estimators," Physical Review E, vol. 52, no. 3, pp. 2318- 2321, September 1995.
174. P. J. Moreno, P. P. Ho, and N. Vasconcelos, "A Kullback-Leibler divergence based kernel for SVM classification in multimedia applications," HPL-2004-4, Cambridge, MA, Technical Report, 2004.
175. P. J. Moreno and R. Rifkin, "Using the Fisher kernel method for web audio classification," HP Lab, Cambridge, MA, Technical Report, 2000.
176. I. Nemenman, W. Bialek, and R. de Ruyter van Steveninck, "Entropy and information in neural spike trains: Progress on the sampling problem," Physical Review E, vol. 69, id. 056111, 2004.
177. I. Nemenman, F. Shafee, and W. Bialek, "Entropy and inference: Revisited," in Advances in Neural Information Processing Systems, (T. G. Dietterich, S. Becker, and Z. Ghahramani, eds.), Cambridge, MA: MIT Press, 2002.
178. D. N. Neuhoff, "On the asymptotic distribution of the errors in vector quantization," IEEE Transactions on Information Theory, vol. 42, no. 2, pp. 461-468, March 1996.
179. X. Nguyen, M. J. Wainwright, and M. I. Jordan, "Divergence Measures, Surrogate Loss Functions and Experimental Design," in NIPS, Cambridge, MA: MIT Press, 2005.
180. M. Nilsson and W. B. Kleijn, "On the estimation of differential entropy from data located on embedded manifolds," IEEE Transactions on Information Theory, vol. 53, no. 7, pp. 2330-2341, July 1996.
181. A. B. Nobel, "Limits to classification and regression estimation from ergodic processes," The Annals of Statistics, vol. 27, no. 1, pp. 262-273, February 1999.
182. A. Okabe, B. Boots, K. Sugihara, and S. N. Chiu, Spatial Tessellations: Concepts and Applications of Voronoi Diagrams. Wiley Series in Probability and Statistics, Second ed., 1992.
183. F. Osterreicher, "Csiszár's f-divergences - basic properties," Austria, 2002.
184. L. Paninski, "Estimation of entropy and mutual information," Neural Computation, vol. 15, no. 6, pp. 1191-1253, June 2003.
185. P. F. Panter and W. Dite, "Quantization in pulse-count modulation with nonuniform spacing of levels," Proceedings of IRE, vol. 39, pp. 44-48, 1951.
186. S. Panzeri and A. Treves, "Analytical estimates of limited sampling biases in different information measures," Network: Computation in Neural Systems, vol. 7, pp. 87-107, 1996.
187. E. Parzen, "On estimation of a probability density function and mode," The Annals of Mathematical Statistics, vol. 33, no. 3, pp. 1065-1076, September 1962.
188. W. D. Penny, "KL-divergences of Normal, Gamma, Dirichlet and Wishart densities," Wellcome department of cognitive neurology, University College London, Technical Report, March 2001. [Online]. Available: www.fil.ion.ucl.ac. uk/wpenny/publications/densities.ps.
189. M. Pinsker, V. V. Prelov, and S. Verdú, "Sensitivity of channel capacity," IEEE Transactions on Information Theory, vol. 33, pp. 405-422, November 1989.
190. M. S. Pinsker, Information and Information Stability of Random Variables and Processes. Izd. Akad. Nauk, 1960, translated by A. Feinstein, 1964.
191. J. P. Pluim, J. Maintz, and M. A. Viergever, "f-information measures in medical image registration," Proceedings of the Meeting of the International Society Optical Engineering (SPIE), vol. 4322, pp. 579-587, 2001.
192. J. P. W. Pluim, J. B. A. Maintz, and M. A. Viergever, "Mutual information based registration of medical images: A survey," IEEE Transactions on Medical Imaging, vol. 22, no. 8, pp. 986-1004, August 2003.
193. H. V. Poor and J. B. Thomas, "Applications of Ali-Silvey distance measures in the design of generalized quantizers for binary decision systems," IEEE Transactions on Communications, vol. 25, no. 9, pp. 893-900, September 1977.
194. V. V. Prelov, "Asymptotic behavior of a continuous channel with small additive noise," Problems of Information and Transmission, vol. 4, pp. 31-37, 1968.
195. V. V. Prelov, "Asymptotic behavior of the capacity of a continuous channel with a large amount of noise," Problemy Peredachi Informastsii, vol. 6, pp. 40- 57, April-June 1970.
196. V. V. Prelov, "Communication channel capacity with almost Gaussian noise," Theory of Probability and its Applications, vol. 33, pp. 405-422, 1989.
197. R. Q. Quiroga, J. Arnhold, K. Lehnertz, and P. Grassberger, "Kullback- Leibler and renormalized entropies: Applications to electroencephalograms of epilepsy patients," Physical Review E, vol. 62, no. 6, pp. 8380-8386, December 2000.
198. B. L. S. P. Rao, Nonparametric Functional Estimation. Academic Press, 1983.
199. A. Rényi, "On the dimension and entropy of probability distributions," Acta Mathematica Academiac Scientiarum Hungaricae, vol. 10, no. 1-2, pp. 193-215, March 1959.
200. A. Rényi, "On measures of entropy and information," Proceedings of the 4th Berkeley Symposium on Mathematics, Statistics and Probability, vol. 1, pp. 547-561, 1961.
201. F. Rieke, D. Warland, R. de Ruyter van Steveninck, and W. Bialek, Spikes: Exploring the Neural Code. Cambridge, MA, USA: MIT Press, 1999.
202. P. M. Robinson, "Consistent nonparametric entropy-based testing," The Review of Economic Studies, Special Issue: The Econometrics of Financial Markets, vol. 58, no. 3, pp. 437-453, May 1991.
203. C. C. Rodrigez and J. Van Ryzin, "Large sample properties of maximum entropy histograms," IEEE Transactions on Information Theory, vol. 32, pp. 751-759, November 1986.
204. M. Rosenblatt, "A central limit theorem and a strong mixing condition," Proceedings of the National Academy of Sciences of the United States of America, vol. 42, pp. 43-47, 1956.
205. M. Rosenblatt, "Remarks on some nonparametric estimates of a density function," The Annals of Mathematical Statistics, vol. 27, no. 3, pp. 832-837, September 1956.
206. M. Rosenblatt, "Density estimates and Markov sequences," in Nonparametric Techniques in Statistical Inference, (M. Puri, ed.), pp. 199-210, Cambridge University Press, London, 1970.
207. G. Roussas, "Nonparametric estimation in Markov processes," Annals of the Institute of Statistical Mathematics, vol. 21, no. 1, pp. 73-87, December 1969.
208. G. Roussas, "Nonparametric estimation of the transition distribution function of a Markov process," Annals of Mathematical Statistics, vol. 40, no. 4, pp. 1386-1400, 1969.
209. M. R. Sabuncu and P. J. Ramadge, "Spatial information in entropy based image registration," Proceedings of the 2003 International Workshop on Biomedical Image Registration, 2003.
210. M. R. Sabuncu and P. J. Ramadge, "Using spanning graphs for efficient image registration," IEEE Transactions on Image Processing, vol. 17, no. 5, pp. 788-797, May 2008.
211. P. K. Sahoo and G. Arora, "A thresholding method based on two dimensional Rényi's entropy," Pattern Recognition, vol. 37, no. 6, pp. 1149-1161, July 2004.
212. T. D. Schneider, "Use of information theory in molecular biology," Proceedings of 1992 Workshop on Physics and Computation (PhysComp '92), pp. 102-110, October 1992.
213. E. Schneidman, W. Bialek, and M. J. Berry II, "An information theoretic approach to the functional classification of neurons," in Advances in Neural Information Processing, (S. Becker, S. Thrun, and K. OberMayer, eds.), pp. 197-204, Cambridge, US: MIT Press, 2003.
214. J. Seckbach and E. Rubin, The New Avenues in Bioinformatics. Springer, 2004.
215. G. Shakhnarovich, J. W. Fisher, and T. Darrell, "Face recognition from longterm observations," Proceedings of the 7th European Conference on Computer Vision - Part III, pp. 851-868, 2002.
216. C. E. Shannon, "A mathematical theory of communication," Bell System Technical Journal, pp. 379-423, 623-656, July, October 1948.
217. C. E. Shannon, "Prediction and entropy of printed English," Bell System Technical Journal, vol. 30, no. 1, pp. 50-64, 1951.
218. P. C. Shields, "The interactions between ergodic theory and information theory," IEEE Transactions on Information Theory, vol. 44, no. 6, pp. 2079-2093, October 1998.
219. J. Shore and R. Johnson, "Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy," IEEE Transactions on Information Theory, vol. 26, pp. 26-37, January 1980.
220. R. H. Shumway and A. N. Unger, "Linear discriminant functions for stationary time series," Journal of the American Statistical Association, vol. 69, no. 348, pp. 948-956, December 1974.
221. J. Silva and S. S. Narayanan, "Average divergence distance as a statistical discrimination measure for hidden Markov models," IEEE Transactions on Audio, Speech, and Language Processing, vol. 14, pp. 890-906, May 2006.
222. J. Silva and S. S. Narayanan, "Upper bound Kullback-Leibler divergence for hidden Markov models with application as discrimination measure for speech recognition," Proceedings of 2006 International Symposium on Information Theory (ISIT 2006), pp. 2299-2303, 2006.
223. J. Silva and S. S. Narayanan, "Universal consistency of data-driven partitions for divergence estimation," Proceedings of 2007 IEEE International Symposium on Information Theory (ISIT2007), June 2007.
224. B. W. Silverman, Density Estimation for Statistics and Data Analysis. Chapman and Hall, 1986.
225. H. Skaug and D. Tjøstheim, "Nonparametric tests of serial independence," in Developments in Time Series Analysis, (S. Rao, ed.), pp. 207-229, Chapman and Hill, 1993.
226. H. Skaug and D. Tjøstheim, "Testing for serial independence using measures of distance between densities," in Athens Conference on Applied Probability and Time Series, (P. R. M. Rosenblatt, ed.), Springer Lecture Notes in Statistics, Springer, 1996.
227. M. B. Skouson, Q. J. Guo, and Z. P. Liang, "A bound on mutual information for image registration," IEEE Transactions on Medical Imaging, vol. 20, pp. 843-846, August 1987.
228. N. Slonim, G. S. Atwal, G. Tkacik, and W. Bialek, "Estimating mutual information and multi-information in large networks," [Online]. Available: http://arxiv.org/abs/cs.IT/0502017, 2005.
229. A. Stam, "Some inequalities satisfied by the quantities of information of Fisher and Shannon," Information and Control, vol. 2, pp. 101-112, June 1959.
230. R. Steuer, J. Kurths, C. O. Daub, J. Weise, and J. Selbig, "The mutual information: Detecting and evaluating dependencies between variables," Bioinformatics, vol. 18, pp. S231-S240, October 2002.
231. H. Stögbauer, A. Kraskov, S. A. Astakhov, and P. Grassberger, "Leastdependent- component analysis based on mutual information," Physical Review E, vol. 70, id. 066123, 2004.
232. S. P. Strong, R. Koberle, R. R. de Ruyter van Steveninck, and W. Bialek, "Entropy and information in neural spike trains," Physical Review Letter, vol. 80, no. 1, pp. 197-200, January 1998.
233. F. P. Tarasenko, "On the evaluation of an unknown probability density function, the direct estimation of the entropy from independent observations of a continuous random variable and the distribution-free entropy test of goodnessof- fit," Proccedings of IEEE, vol. 56, pp. 2052-2053, 1968.
234. T. Tasdizen, S. P. Awate, R. T. Whitaker, and N. L. Foster, "MRI tissue classification with neighborhood statistics: A nonparametric, entropy-minimizing approach," Proceedings of the 2005 International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI2005), vol. 3750, pp. 517-525, 2005.
235. H. Theil, "The entropy of the maximum entropy distribution," Economic Letters, pp. 145-148, 1980.
236. H. Theil and D. G. Fiebig, Exploiting Continuity: Maximum Entropy Estimation of Continuous Distributions. Cambridge, MA: Ballinger Publishing Company, 1984.
237. D. Tjøstheim, "Measures and tests of independence: A survey," Statistics, vol. 28, pp. 249-284, 1996.
238. F. Topsøe, "Some inequalities for information divergence and related measures of discrimination," IEEE Transactions on Information Theory, vol. 46, no. 4, pp. 1602-1609, July 2000.
239. K. Torkkola, "Feature extraction by nonparametric mutual information maximization," Journal of Machine Learning Research, vol. 3, pp. 1415-1438, March 2003.
240. A. Tsai, J. W. Fisher III, C. Wible, W. M. Wells III, J. Kim, and A. S. Willsky, "Analysis of functional MRI data using mutual information," Proceedings of the 2nd International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 473-480, October 1999.
241. A. Tsai, W.Wells, C. Tempany, E. Grimson, and A. Willsky, "Mutual information in coupled multi-shape model for medical image segmentation," Medical Image Analysis, vol. 8, no. 4, pp. 429-445, December 2004.
242. A. B. Tsybakov and E. C. van der Meulen, "Root-n consistent estimators of entropy for densities with unbounded support," Technical Report, 1992.
243. A. B. Tsybakov and E. C. van der Meulen, "Root-n consistent estimators of entropy for densities with unbounded support," Scandinavian Journal of Statistics, vol. 23, pp. 75-83, 1996.
244. A. M. Tulino and S. Verdú, "Monotonic decrease of the non-Gaussianness of the sum of independent random variables: A simple proof," IEEE Transactions on Information Theory, vol. 52, no. 9, pp. 4295-4297, September 2006.
245. I. Vajda, "Note on discrimination information and variation," IEEE Transactions on Information Theory, vol. 16, no. 6, pp. 771-773, November 1970.
246. I. Vajda, "On the f-divergence and singularity of probability measures," Periudicu Mathematica Hiingarrricu, vol. 2, pp. 223-234, 1972.
247. I. Vajda, Theory of Statistical Inference and Information. Dordrecht-Boston: Kluwer, 1989.
248. P. A. van den Elsen, E.-J. D. Pol, and M. A. Viergever, "Medical image matching - A review with classification," IEEE Engineering in Medicine and Biology Magazine, pp. 26-38, March 1993.
249. B. Van Es, "Estimating functionals related to a density by a class of statistics based on spacings," Scandinavian Journal of Statistics, vol. 19, pp. 61-72, 1992.
250. M. M. Van Hulle, "Edgeworth approximation of multivariate differential entropy," Neural Computation, vol. 17, pp. 1903-1910, September 2005.
251. V. Vapnik, The Nature of Statistical Learning Theory. Springer-Verlag, 1999.
252. N. Vasconcelos, P. Ho, and P. Moreno, "The Kullback-Leibler kernel as a framework for discriminant and localized representations for visual recognition," Proceedings of 2004 European Conference on Computer Vision (ECCV 2004), pp. 430-441, May 2004.
253. O. Vasicek, "A test for normality based on sample entropy," Journal of the Royal Statistical Society, Series B ( Methodological), vol. 38, no. 1, pp. 54-59, 1976.
254. S. Verdú, "Lectures notes on information theory," unpublished.
255. S. Verdú, "Fifty years of Shannon theory," IEEE Transactions on Information Theory, vol. 44, pp. 2057-2078, October 1998.
256. S. Verdú, Multiuser Detection. Cambridge University Press, 1998.
257. J. D. Victor, "Binless strategies for estimation of information from neural data," Physical Review E, vol. 66, id. 051903, 2002.
258. A. Wald, "Sequential tests of statistical hypotheses," Annals of Mathematical Statistics, vol. 16, no. 2, pp. 117-186, 1945.
259. D. L. Wallace, "Asymptotic approximations to distributions," The Annals of Mathematical Statistics, vol. 29, no. 3, pp. 635-654, September 1958.
260. Q. Wang, S. R. Kulkarni, and S. Verdú, "Divergence estimation of continuous distributions based on data-dependent partitions," IEEE Transactions on Information Theory, vol. 51, no. 9, pp. 3064-3074, September 2005.
261. Q. Wang, S. R. Kulkarni, and S. Verdú, "A nearest-neighbor approach to estimating divergence between continuous random vectors," Proceedings of 2006 IEEE International Symposium on Information Theory (ISIT2006), July 2006.
262. Q. Wang, S. R. Kulkarni, and S. Verdú, "Divergence estimation for multi-dimensional distributions via k-nearest-neighbor distances," IEEE Transactions on Information Theory, vol. 55, no. 5, pp. 2392-2405, May 2009.
263. N. Wiener, Cybernetics: Control and Communication in the Animal and the Machine. Cambridge, MA: MIT Press, 1948.
264. R. Wilson, D. Tse, and R. A. Scholtz, "Channel identification: Secret sharing using reciprocity in ultrawideband channels," IEEE Transactions on Information Forensics and Security, vol. 2, pp. 364-375, September 2007.
265. D. Wolpert and D. Wolf, "Estimating functions of probability distributions from a finite set of samples," Physical Review E, vol. 52, pp. 6841-6854, 1995.
266. Y.Wu, K. L. Chan, and Y. Huang, "Image texture classification based on finite Gaussian mixture models," Proceedings of The 3rd International Workshop on Texture Analysis and Synthesis (In Conjunction with ICCV2003), pp. 107- 112, October 2003.
267. A. D. Wyner and J. Ziv, "Some asymptotic properties of the entropy of a stationary ergodic data source with applications to data-compression," IEEE Transactions on Information Theory, vol. 35, no. 6, pp. 1250-1258, November 1989.
268. H. Yang, S. van Vuuren, and H. Hermansky, "Relevancy of time-frequency features for phonetic classification measured by mutual information," Proceedings of the International Conference on Acoustics, Speech, and Signal Processing (ICASSP'99), vol. 1, pp. 225-228, 1999.
269. C. Ye, A. Reznik, and Y. Shah, "Extracting secrecy from jointly Gaussian random variables," Proceedings of 2006 IEEE International Symposium on Information Theory (ISIT2006), pp. 2593-2597, July 2006.
270. H. P. Yockey, Information theory and molecular biology. New York: Cambridge University Press, 1992.
271. J. Ziv and N. Merhav, "A measure of relative entropy between individual sequences with application to universal classification," IEEE Transactions on Information Theory, vol. 39, no. 4, pp. 1270-1279, July 1993.
272. K. ZŻyczkowski, "Rényi extrapolation of Shannon entropy," Open Systems and Information Dynamics, vol. 10, pp. 297-310, 2003.