Estimating entropy rates with bayesian confidence intervals

Neural Computation

Volumn 17, Issue 7, 2005, Pages 1531-1576

  Kennel, Matthew B ^a   Shlens, Jonathon ^a,b   Abarbanel, Henry D I ^c   Chichilnisky, E J ^b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b SALK INSTITUTE FOR BIOLOGICAL STUDIES   (United States)

^c UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; BAYES THEOREM; CONFIDENCE INTERVAL; ENTROPY; STATISTICS;

BAYES THEOREM; CONFIDENCE INTERVALS; ENTROPY; STATISTICS;

EID: 18744410140     PISSN: 08997667     EISSN: None     Source Type: Journal    
DOI: 10.1162/0899766053723050     Document Type: Article

References (70)

1. Amigo, J., Szczepanski, J., Wajnryb, E., & Sanchez-Vives, M. (2004). Estimating the entropy rate of spike trains via Lempel-Ziv complexity. Neural Computation, 16, 717-736.
2. Balasubramanian, V. (1997). Statistical inference, Occam's razor, and statistical mechanics on the space of probability distributions. Neural Computation, 9, 349-368.
3. Bialek, W., Nemenman, I., & Tishby, N. (2001). Predictability, complexity, and learning. Neural Computation, 13, 2409-2463.
4. Bialek, W., Rieke, F., de Ruyter van Steveninck, R., & Warland, D. (1991). Reading a neural code. Science, 252, 1854-1857.
5. Borst, A., & Theunissen, F. (1999). Information theory and neural coding. Nature Neuroscience, 2, 947-957.
6. Brenner, N., Strong, S., Koberle, R., Bialek, W., & de Ruyter van Steveninck, R. (2000). Synergy in a neural code. Neural Computation, 12, 1531-1552.
7. Buracas, G., Zador, A., DeWeese, M., & Albright, T. (1998). Efficient discrimination of temporal patterns by motion-sensitive neurons in primate visual cortex. Neuron, 20, 959-969.
8. Butts, D. (2003). How much information is associated with a particular stimulus? Network, 14, 177-187.
9. Costa, J., & Hero, A. (2004). Geodesic entropic graphs for dimension and entropy estimation in manifold learning. IEEE Transactions on Signal Processing, 25, 2210-2221.
10. Cover, T., & Thomas, J. (1991). Elements of information theory. New York: Wiley.
11. de Ruyter van Steveninck, R., Lewen, G., Strong, S., Koberle, R., & Bialek, W. (1997). Reproducibility and variability in neural spike trains. Science, 275, 1805-1808.
12. DeWeese, M., & Meister, M. (1999). How to measure the information gained from one symbol. Network, 10, 325-340.
13. Duda, R., Hart, P., & Stork, D. (2000). Pattern classification (2nd ed.). New York: Wiley.
14. Field, D. (1987). Relations between the statistics of natural images and the response properties of cortical cells. Journal of the Optical Society of America, A4, 2379-2394.
15. Gamerman, D. (1997). Markov chain Monte Carlo: Stochastic simulation of Bayesian inference. New York: CRC Press.
16. Gilmore, R., & Lefranc, M. (2002). The topology of chaos: Alice in stretch and squeeze land. New York: Wiley.
17. Grassberger, P. (2003). Entropy estimates from insufficient samplings, e-print, physics/0307138.
18. Hastings, W. (1970). Monte Carlo sampling using Markov chains and their applications. Biometrica, 57, 97-109.
19. Hilborn, R. (2000). Chaos and nonlinear dynamics: An introduction for scientists and engineers. New York: Oxford University Press.
20. Johnson, D., Gruner, C., Baggerly, K., & Seshagiri, C. (2001). Information-theoretic analysis of neural coding. Journal of Computational Neuroscience, 10, 47-69.
21. Kennel, M., & Buhl, M. (2003). Estimating good discrete partitions from observed data: Symbolic false nearest neighbors. Phys. Rev. Lett., 91, 084102.
22. Kennel, M., & Mees, A. (2002). Context-tree modeling of observed symbolic dynamics. Physical Review E, 66, 056209.
23. Kontoyiannis, I., Algoet, P., Suhov, Y., & Wyner, A. (1998). Nonparametric entropy estimation for stationary processes and random fields with applications to English text. IEEE Transactions in Information Theory, 44, 1319-1327.
24. Kraskov, A., Stogbauer, H., & Grassberger, P. (2004). Estimating mutual information. Physical Review E, 69, 006138.
25. Krichevsky, R. E., & Trofimov, V. K. (1981). The performance of universal coding. IEEE Transactions in Information Theory, 28, 199-207.
26. Lehmann, E., & Casella, G. (1998). Theory of point estimation. New York: Springer.
27. Lem̀pel, A., & Ziv, J. (1976). On the complexity of finite sequences. IEEE Transactions in Information Theory, 22, 75-78.
28. Lewen, G., Bialek, W., & de Ruyter van Steveninck, R. (2001). Neural coding of naturalistic motion stimuli. Network: Computation in Neural Systems, 12, 317-329.
29. Lind, D., & Marcus, B. (1996). Symbolic dynamics and coding. Cambridge: Cambridge University Press.
30. Liu, R., Tzonev, S., Rebrik, S., & Miller, K. (2001). Variability and information in a neural code of the cat lateral geniculate nucleus. Journal of Neurophysiology, 86, 2789-2806.
31. London, M., Schreibman, A., Hausser, M., Larkum, M., & Segev, I. (2002). The information efficacy of a synapse. Nature Neuroscience, 5, 332-340.
32. Ma, S. (1981). Calculation of entropy from data of motion. Journal of Statistical Physics, 26, 221-240.
33. MacKay, D. (2003). Information theory, inference and learning algorithms. Cambridge: Cambridge University Press.
34. MacKay, D., & McCulloch, W. (1952). The limiting information capacity of a neuronal link. Bulletin of Mathematical Biophysics, 14, 127-135.
35. Martin, A., Seroussi, G., & Weinberger, M. (2004). Linear time universal coding and time reversal of tree sources via FSM closure. IEEE Transactions on Information Theory, 50, 1442-1468.
36. Metropolis, N., Rosenbluth, M., Rosenbluth, A., Teller, M., & Teller, A. (1953). Equations of state calculations by fast computing machines. Journal of Chemical Physics, 21, 1087-1091.
37. Miller, G., & Madow, W. (1954). On the maximum likelihood estimate of the Shannon-Wiener measure of information. Air Force Cambridge Research Center Technical Report, 75, 54.
38. Nemenman, I. (2002). Inference of entropies of discrete random variables with unknown cardinalities, physics/02070009. Available online: www.arxiv.org.
39. Nemenman, I., Bialek, W., & de Ruyter van Steveninck, R. (2004). Entropy and information in neural spike trains: Progress on the sampling problem. Physical Review E, 69.
40. Nemenman, I., Shafee, F., & Bialek, W. (2002). Entropy and inference, revisited. Advances in neural information processsing systems, 14, In T. G. Dietterich, S. Becker, & Z. Ghahraman: (Eds.), Cambridge, MA: MIT Press.
41. Orlitsky, A., Santhanam, N., & Zhang, J. (2004). Universal compression of memoryless sources over unknown alphabets. IEEE Trans. Inf. Theo., 50, 1469-1481.
42. Ott, E. (2002). Chaos in dynamical systems. Cambridge: Cambridge University Press.
43. Paninski, L. (2003). Estmation of entropy and mutual information. Neural Computation, 15, 1191-1254.
44. Perkel, D., & Bullock, T. (1968). Neural coding. Neurosci Res. Prog. Sum., 3, 405-527.
45. Rapp, P., Vining, E., Cohen, N., Albano, A., & Jimenez-Montano, M. (1994). The algorithmic complexity of neural spike trains increases during focal seizures. Journal of Neuroscience, 14, 4731-4739.
46. Reinagel, P., & Reid, R. (2000). Temporal coding of visual information in the thalamus. Journal of Neuroscience, 20, 5392-5400.
47. Rieke, F., Bodnar, D., & Bialek, W. (1995). Naturalistic stimuli increase the rate and efficiency of information transmission by primary auditory afferents. Proceedings of the Royal Society of London B: Biological Sciences, 262, 259-265.
48. Rieke, F., Warland, D., de Ruyter van Steveninck, R., & Bialek, W. (1997). Spikes: Exploring the neural code. Cambridge, MA: MIT Press.
49. Rissanen, J. (1989). Stochastic complexity in statistical inquiry. Singapore: World Scientific.
50. Roulston, M. (1999). Estimating the errors on measured entropy and mutual information. Physica D, 125, 285-294.
51. Ruderman, D., & Bialek, W. (1994). Statistics of natural images: Scaling in the woods. Physical Review Letters, 73, 814-817.
52. Schneidman, E., Bialek, W., & Berry, M. (2003). Synergy, redundancy, and independence in population codes. Journal of Neuroscience, 23, 11539-11553.
53. Schurmann, T., & Grassberger, P. (1996). Entropy estimation of symbol sequences. Chaos, 6, 414-427.
54. Shannon, C. (1948). A mathematical theory of communication. Bell Systems Technology Journal, 27, 379-423.
55. Shlens, J., Kennel, M., Abarbanel, H., & Chichilnisky, E. (In preparation). Estimating information rates with Bayesian confidence intervals in neural spike trains. University of California, San Diego, and Salk Institute.
56. Shtarkov, Y., Tjalkens, T., & Willems, F. (1995). Multialphabet weighting: Universal coding of context tree sources. Problems of Info. Trans., 33, 17-28.
57. Simonicelli, E., & Olhausen, B. (2001). Natural image statistics and neural representation. Annual Review of Neuroscience, 24, 1193-1215.
58. Solomonoff, R. (1964). A formal theory of inductive inference. Part I. Information and Control, 7, 1-22.
59. Strong, S., Koberle, R., de Ruyter van Steveninck, R., & Bialek, W. (1998). Entropy and information in neural spike trains. Physical Review Letters, 80, 197-200.
60. Treves, A., & Panzeri, S. (1995). The upward bias in measures of information derived from limited data samples. Neural Computation, 7, 399-407.
61. Victor, J. (2000). Asymptotic bias in information estimates and the exponential (Bell) polynomials. Neural Computation, 12, 2797-2804.
62. Victor, J. (2002). Binless strategies for estimation of information from neural data. Physical Review E, 66, 051903.
63. Victor, J., & Purpura, K. (1997). Metric-space analysis of spike trains: Theory, algorithms, and application. Network, 8, 127-164.
64. Warland, D., Reinagel, P., & Meister, M. (1997). Decoding visual information from a population of retinal ganglion cells. Journal of Neurophysiology, 78, 2336-2350.
65. Willems, F. (1998). Context tree weighting: Extensions. IEEE Transactions on Information Theory, 44, 792-798.
66. Willems, F., Shtarkov, Y., & Tjalkens, T. (1995). The context tree weighting method: Basic properties. IEEE Transactions in Information Theory, IT-41, 653-664.
67. Wolpert, D., & Wolf, D. (1995). Estimating functions of probability distributions from a finite set of samples. Physical Review E, 52, 6841-6854.
68. Wyner, A. D., Ziv, J., & Wyner, A. J. (1998). On the role of pattern matching in information theory. IEEE Transactions on Information Theory, 44, 2045-2056.
69. Ziv, J., & Lempel, A. (1977). A universal algorithm for sequential data compression. IEEE Transactions in Information Theory, 23, 337-343.
70. Ziv, J., & Lempel, A. (1978). Compression of individual sequences via variable-rate coding. IEEE Transactions in Information Theory, 24, 530-536.