Estimation of entropy and mutual information

Neural Computation

Volumn 15, Issue 6, 2003, Pages 1191-1253

  Paninski, Liam ^a  

^a NEW YORK UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0041877169     PISSN: 08997667     EISSN: None     Source Type: Journal    
DOI: 10.1162/089976603321780272     Document Type: Article

References (53)

1. Antos, A., & Kontoyiannis, I. (2001). Convergence properties of functional estimates for discrete distributions. Random Structures and Algorithms, 19, 163-193.
2. Azuma, K. (1967). Weighted sums of certain dependent variables. Tohoku Mathematical Journal, 3, 357-367.
3. Basharin, G. (1959). On a statistical estimate for the entropy of a sequence of independent random variables. Theory of Probability and Its Applications, 4, 333-336.
4. Beirlant, J., Dudewicz, E., Gyorfi, L., & van der Meulen, E. (1997). Nonparametric entropy estimation: An overview. International Journal of the Mathematical Statistics Sciences, 6, 17-39.
5. Bialek, W., Rieke, F., de Ruyter van Steveninck, R., & Warland, D. (1991). Reading a neural code. Science, 252, 1854-1857.
6. Billingsley, P. (1965). Ergodic theory and information. New York: Wiley.
7. Buracas, G., Zador, A., DeWeese, M., & Albright, T. (1998). Efficient discrimination of temporal patterns by motion-sensitive neurons in primate visual cortex. Neuron, 5, 959-969.
8. Carlton, A. (1969). On the bias of information estimates. Psychological Bulletin, 71, 108-109.
9. Chernoff, H. (1952). A measure of asymptotic efficiency for tests of hypothesis based on the sum of observations. Annals of Mathematical Statistics, 23, 493-509.
10. Chow, Y., & Teicher, H. (1997). Probability theory. New York: Springer-Verlag.
11. Cover, T., & Thomas, J. (1991). Elements of information theory. New York: Wiley.
12. Cucker, F., & Smale, S. (2002). On the mathematical foundations of learning. Bulletins of the American Mathematical Society, 39, 1-49.
13. Darbellay, G., & Vajda, I. (1999). Estimation of the information by an adaptive partitioning of the observation space. IEEE Transactions on Information Theory, 45, 1315-1321.
14. Darling, D. (1953). On a class of problems related to the random division of an interval. Annals of Mathematical Statistics, 24, 239-253.
15. Dembo, A., & Zeitouni, O. (1993). Large deviations techniques and applications. New York: Springer-Verlag.
16. Devore, R., & Lorentz, G. (1993). Constructive approximation. New York: Springer-Verlag.
17. Devroye, L., Gyorfi, L., & Lugosi, G. (1996). A probabilistic theory of pattern recognition. New York: Springer-Verlag.
18. Ditzian, Z., & Totik, V. (1987). Moduli of smoothness. Berlin: Springer-Verlag.
19. Donoho, D., & Liu, R. (1991). Geometrizing rates of convergence. Annals of Statistics, 19, 633-701.
20. Efron, B., & Stein, C. (1981). The jackknife estimate of variance. Annals of Statistics, 9, 586-596.
21. Gedeon, T., Parker, A., & Dimitrov, A. (2003). Information distortion and neural coding. Manuscript submitted for publication.
22. Gibbs, A., & Su, F. (2002). On choosing and bounding probability metrics. International Statistical Review, 70, 419-436.
23. Grenander, U. (1981). Abstract inference. New York: Wiley.
24. Jongbloed, G. (2000). Minimax lower bounds and moduli of continuity. Statistics and Probability Letters, 50, 279-284.
25. Kolmogorov, A. (1993). Information theory and the theory of algorithms. Boston: Kluwer.
26. Kontoyiannis, I. (1997). Second-order noiseless source coding theorems. IEEE Transactions Information Theory, 43, 1339-1341.
27. Lugosi, A., & Nobel, A. B. (1996). Consistency of data-driven histogram methods for density estimation and classification. Annals of Statistics, 24, 687-706.
28. McDiarmid, C. (1989). On the method of bounded differences. In J. Siemons (Ed.), Surveys in combinatorics (pp. 148-188). Cambridge: Cambridge University Press.
29. Miller, G. (1955). Note on the bias of information estimates. In H. Quastler (Ed.), Information theory in psychology II-B (pp. 95-100). Glencoe, IL: Free Press.
30. Nemenman, I., Shafee, F., & Bialek, W. (2002). Entropy and inference, revisited. In T. G. Dietterich, S. Becker, & Z. Ghahramani (Eds.), Advances in neural information processing, 14. Cambridge, MA: MIT Press.
31. Paninski, L., Fellows, M., Hatsopoulos, N., & Donoghue, J. (1999). Coding dynamic variables in populations of motor cortex neurons. Society for Neuroscience Abstracts, 25, 665.9.
32. Paninski, L., Fellows, M., Hatsopoulos, N., & Donoghue, J. (2003). Temporal tuning properties for hand position and velocity in motor cortical neurons. Manuscript submitted for publication.
33. Paninski, L., Lau, B., & Reyes, A. (in press). Noise-driven adaptation: In vitro and mathematical analysis. Neurocomputing. Available on-line: http://www. cns.nyu.edu/̃liam/adapt.html.
34. Panzeri, S., & Treves, A. (1996). Analytical estimates of limited sampling biases in different information measures. Network: Computation in Neural Systems, 7, 87-107.
35. Prakasa Rao, B. (2001). Cramer-Rao type integral inequalities for general loss functions. TEST, 10, 105-120.
36. Press, W., Teukolsky, S., Vetterling, W., & Flannery, B. (1992). Numerical recipes in C. Cambridge: Cambridge University Press.
37. Rieke, F., Warland, D., de Ruyter van Steveninck, R., & Bialek, W. (1997). Spikes: Exploring the neural code. Cambridge, MA: MIT Press.
38. Ritov, Y., & Bickel, P. (1990). Achieving information bounds in non- and semiparametric models. Annals of Statistics, 18, 925-938.
39. Schervish, M. (1995). Theory of statistics. New York: Springer-Verlag.
40. Serfling, R. (1980). Approximation theorems of mathematical statistics. New York: Wiley.
41. Shao, Y., & Hahn, M. (1995). Limit theorems for the logarithm of sample spacings. Statistics and Probability Letters, 24, 121-132.
42. Steele, J. (1986). An Efron-Stein inequality for nonsymmetric statistics. Annals of Statistics, 14, 753-758.
43. Strong, S. Koberle, R., de Ruyter van Steveninck R., & Bialek, W. (1998). Entropy and information in neural spike trains. Physical Review Letters, 80, 197-202.
44. Tishby, N., Pereira, F., & Bialek, W. (1999). The information bottleneck method. In B. Hajek & R. S. Sreenivas (Eds.), Proceedings of the 37th Allerton Conference on Communication, Control, and Computing. Urbana, IL: University of Illinois Press.
45. Treves, A., & Panzeri, S. (1995). The upward bias in measures of information derived from limited data samples. Neural Computation, 7, 399-407.
46. van der Vaart, A., & Wellner, J. (1996). Weak convergence and empirical processes. New York: Springer-Verlag.
47. Vapnik, V. N., & Chervonenkis, A. J. (1971). On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and Its Applications, 16, 264-280.
48. Victor, J. (2000a). Asymptotic bias in information estimates and the exponential (Bell) polynomials. Neural Computation, 12, 2797-2804.
49. Victor, J. (2000b). How the brain uses time to represent and process visual information. Brain Research, 886, 33-46.
50. Victor, J. (2002). Binless strategies for estimation of information from neural data. Physical Review E, 66, 51903-51918.
51. Watson, G. (1980). Approximation theory and numerical methods. New York: Wiley.
52. Weaver, W., & Shannon, C. E. (1949). The mathematical theory of communication. Urbana: University of Illinois Press.
53. Wolpert, D., & Wolf, D. (1995). Estimating functions of probability distributions from a finite set of samples. Physical Review E, 52, 6841-6854.