Predictability, complexity, and learning

Neural Computation

Volumn 13, Issue 11, 2001, Pages 2409-2463

  Bialek, William ^a   Nemenman, Ilya ^a,b   Tishby, Naftali ^a,c  

^a NEC LABORATORIES AMERICA   (United States)

^b PRINCETON UNIVERSITY   (United States)

^c HEBREW UNIVERSITY OF JERUSALEM   (Israel)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; FORECASTING; HUMAN; INFORMATION SCIENCE; LEARNING; PHYSIOLOGY; PSYCHOLOGICAL MODEL;

FORECASTING; HUMANS; INFORMATION THEORY; LEARNING; MODELS, PSYCHOLOGICAL;

EID: 0035514587     PISSN: 08997667     EISSN: None     Source Type: Journal    
DOI: 10.1162/089976601753195969     Document Type: Article

References (103)

1. Abarbanel, H. D. I., Brown, R., Sidorowich, J. J., & Tsimring, L. S. (1993). The analysis of observed chaotic data in physical systems. Revs. Mod. Phys., 65, 1331-1392.
2. Adami, C., & Cerf, N. J. (2000). Physical complexity of symbolic sequences. Physica D, 137, 62-69. See also adap-org/9605002.
3. Aida, T. (1999). Field theoretical analysis of on-line learning of probability distributions. Phys. Rev. Lett., 83, 3554-3557. See also cond-mat/9911474.
4. Akaike, H. (1974a). Information theory and an extension of the maximum likelihood principle. In B. Petrov & F. Csaki (Eds.), Second International Symposium of Information Theory Budapest: Akademia Kiedo.
5. Akaike, H. (1974b). A new look at the statistical model identification. IEEE Trans. Automatic Control, 19, 716-723.
6. Atick, J. J. (1992). Could information theory provide an ecological theory of sensory processing? In W. Bialek (Ed.), Princeton lectures on biophysics (pp. 223-289). Singapore: World Scientific.
7. Attneave, F. (1954). Some informational aspects of visual perception. Psych. Rev., 61, 183-193.
8. Balasubramanian, V. (1997). Statistical inference, Occam's razor, and statistical mechanics on the space of probability distributions. Neural Comp., 9, 349-368. See also cond-mat/9601030.
9. Barlow, H. B. (1959). Sensory mechanisms, the reduction of redundancy and intelligence. In D. V. Blake & A. M. Uttley (Eds.), Proceedings of the Symposium on the Mechanization of Thought Processes, (Vol. 2, pp. 537-574), London: H. M. Stationery Office.
10. Barlow, H. B. (1961). Possible principles underlying the transformation of sensory messages. In W. Rosenblith (Ed.), Sensory Communication (pp. 217-234), Cambridge, MA: MIT Press.
11. Barlow, H. B. (1983). Intelligence, guesswork, language. Nature, 304, 207-209.
12. Barron, A., & Cover, T. (1991). Minimum complexity density estimation. IEEE Trans. Inf. Thy., 37, 1034-1054.
13. Bennett, C. H. (1985). Dissipation, information, computational complexity and the definition of organization. In D. Pines (Ed.),Emerging syntheses in science (pp. 215-233). Reading, MA: Addison-Wesley.
14. Bennett, C. H. (1990). How to define complexity in physics, and why. In W. H. Zurek (Ed.), Complexity, entropy and the physics of information (pp. 137-148) Redwood City, CA: Addison-Wesley.
15. Berry II, M. J., Warland, D. K., & Meister, M. (1997). The structure and precision of retinal spike trains. Proc. Natl. Acad. Scii. (USA), 94, 5411-5416.
16. Bialek, W. (1995). Predictive information and the complexity of time series (Tech. Note). Princeton, NJ: NEC Research Institute.
17. Bialek, W., Callan, C. G., & Strong, S. P. (1996). Field theories for learning probability distributions. Phys. Rev. Lett., 77, 4693-4697. See also cond-mat/9607180.
18. Bialek, W., & Tishby, N. (1999). Predictive information. Preprint. Available at cond-mat/9902341.
19. Bialek, W., &Tishby, N. (in preparation). Extracting relevant information.
20. Brenner, N., Bialek, W., & de Ruyter van Steveninck, R. (2000). Adaptive rescaling optimizes information transmission. Neuron, 26, 695-702.
21. Bruder, S. D. (1998). Predictive information in spike trains from the blowfly and monkey visual systems. Unpublished doctoral dissertation, Princeton University.
22. Brunel, N., & Nadal, P. (1998). Mutual information, Fisher information, and population coding. Neural Comp., 10, 1731-1757.
23. Capaldi, E. A., Robinson, G. E., & Fahrbach, S. E. (1999). Neuroethology of spatial learning: The birds and the bees. Annu. Rev. Psychol., 50, 651-682.
24. Carpenter, R. H. S., & Williams, M. L. L. (1995). Neural computation of log likelihood in control of saccadic eye movements. Nature, 377, 59-62.
25. Cesa-Bianchi, N., & Lugosi, G. (in press). Worst-case bounds for the logarithmic loss of predictors. Machine Learning.
26. Chaitin, G. J. (1975). A theory of program size formally identical to information theory. J. Assoc. Comp. Mach., 22, 329-340.
27. Clarke, B. S., & Barron, A. R. (1990). Information-theoretic asymptotics of Bayes methods. IEEE Trans. Inf. Thy., 36, 453-471.
28. Cover, T. M., & King, R. C. (1978). A convergent gambling estimate of the entropy of English. IEEE Trans. Inf. Thy., 24, 413-421.
29. Cover, T. M., & Thomas, J. A. (1991). Elements of information theory. New York: Wiley.
30. Crutchfield, J. P., & Feldman, D. P. (1997). Statistical complexity of simple 1-D spin systems. Phys. Rev. E, 55, 1239-1243R. See also cond-mat/9702191.
31. Crutchfield, J. P., Feldman, D. P., & Shalizi, C. R. (2000). Comment I on "Simple measure for complexity." Phys. Rev E, 62, 2996-2997. See also chao-dyn/9907001.
32. Crutchfield, J. P., & Shalizi, C. R. (1999). Thermodynamic depth of causal states: Objective complexity via minimal representation. Phys. Rev. E, 59, 275-283. See also cond-mat/9808147.
33. Crutchfield, J. P., & Young, K. (1989). Inferring statistical complexity. Phys. Rev. Lett., 63, 105-108.
34. Eagleman, D. M., & Sejnowski, T. J. (2000). Motion integration and postdiction in visual awareness. Science, 287, 2036-2038.
35. Ebeling, W., & Pöschel, T. (1994). Entropy and long-range correlations in literary English. Europhys. Lett., 26, 241-246.
36. Feldman, D. P., & Crutchfield, J. P. (1998). Measures of statistical complexity: Why? Phys. Lett. A, 238, 244-252. See also cond-mat/9708186.
37. Gell-Mann, M., & Lloyd, S. (1996). Information measures, effective complexity, and total information. Complexity, 2, 44-52.
38. de Gennes, P.-G. (1979). Scaling concepts in polymer physics. Ithaca, NY: Cornell University Press.
39. Grassberger, P. (1986). Toward a quantitative theory of self-generated complexity. Int. J. Theor. Phys., 25, 907-938.
40. Grassberger, P. (1991). Information and complexity measures in dynamical systems. In H. Atmanspacher & H. Schreingraber (Eds.), Information dynamics (pp. 15-33). New York: Plenum Press.
41. Hall, P., & Hannan, E. (1988). On stochastic complexity and nonparametric density estimation. Biometrica, 75, 705-714.
42. Haussler, D., Kearns, M., & Schapire, R. (1994). Bounds on the sample complexity of Bayesian learning using information theory and the VC dimension. Machine Learning, 14, 83-114.
43. Haussler, D., Kearns, M., Seung, S., & Tishby N. (1996). Rigorous learning curve bounds from statistical mechanics. Machine Learning, 25, 195-236.
44. Haussler, D., & Opper, M. (1995). General bounds on the mutual information between a parameter and n conditionally independent events. In Proceedings of the Eighth Annual Conference on Computational Learning Theory (pp. 402-411). New York: ACM Press.
45. Haussler, D., & Opper, M. (1997). Mutual information, metric entropy and cumulative relative entropy risk. Ann. Statist., 25, 2451-2492.
46. Haussler, D., & Opper, M. (1998). Worst case prediction over sequences under log loss. In G. Cybenko, D. O'Leary, & J. Rissanen (Eds.), The mathematics of information coding, extraction and distribution. New York: Springer-Verlag.
47. Hawken, M. J., & Parker, A. J. (1991). Spatial receptive field organization in monkey V1 and its relationship to the cone mosaic. In M. S. Landy & J. A. Movshon (Eds.), Computational models of visual processing (pp. 83-93). Cambridge, MA: MIT Press.
48. Herschkowitz, D., & Nadal, J.-P. (1999). Unsupervised and supervised learning: Mutual information between parameters and observations. Phys. Rev. E, 59, 3344-3360.
49. Hilberg, W. (1990). The well-known lower bound of information in written language: Is it a misinterpretation of Shannon experiments? Frequenz, 44, 243-248. (in German)
50. Holy, T. E. (1997). Analysis of data from continuous probability distributions. Phys. Rev. Lett., 79, 3545-3548. See also physics/9706015.
51. Ibragimov, I., & Hasminskii, R. (1972). On an information in a sample about a parameter. In Second International Symposium on Information Theory (pp. 295-309). New York: IEEE Press.
52. Kang, K., & Sompolinsky, H. (2001). Mutual information of population codes and distance measures in probability space. Preprint. Available at cond-mat/0101161.
53. Kemeney, J. G. (1953). The use of simplicity in induction. Philos. Rev., 62, 391-315.
54. Kolmogoroff, A. N. (1939). Sur l'interpolation et extrapolations des suites stationnaires. C. R. Acad. Sci. Paris, 208, 2043-2045.
55. Kolmogorov, A. N. (1941). Interpolation and extrapolation of stationary random sequences. Izv. Akad. Nauk. SSSR Ser. Mat., 5, 3-14. (in Russian) Translation in A. N. Shiryagev (Ed.), Selected works of A. N. Kolmogorov (Vol. 23, pp. 272-280). Norwell, MA: Kluwer.
56. Kolmogorov, A. N. (1941). Interpolation and extrapolation of stationary random sequences. Izv. Akad. Nauk. SSSR Ser. Mat., 5, 3-14. (in Russian) Translation in A. N. Shiryagev (Ed.), Selected works of A. N. Kolmogorov (Vol. 23, pp. 272-280). Norwell, MA: Kluwer.
57. Kolmogorov, A. N. (1965). Three approaches to the quantitative definition of information. Prob. Inf. Trans., 1, 4-7.
58. Li, M., & Vitányi, P. (1993). An Introduction to Kolmogorov complexity and its applications. New York: Springer-Verlag.
59. Lloyd, S., & Pagels, H. (1988). Complexity as thermodynamic depth. Ann. Phys., 188, 186-213.
60. Logothetis, N. K., & Sheinberg, D. L. (1996). Visual object recognition. Annu. Rev. Neurosci., 19, 577-621.
61. Lopes, L. L., & Oden, G. C. (1987). Distinguishing between random and nonrandom events. J. Exp. Psych.: Learning, Memory, and Cognition, 13, 392-400.
62. Lopez-Ruiz, R., Mancini, H. L., & Calbet, X. (1995). A statistical measure of complexity. Phys. Lett. A, 209, 321-326.
63. MacKay, D. J. C. (1992). Bayesian interpolation. Neural Comp., 4, 415-447.
64. Nemenman, I. (2000). Information theory and learning: A physical approach. Unpublished doctoral dissertation. Princeton University. See also physics/0009032.
65. Nemenman, I., & Bialek, W. (2001). Learning continuous distributions: Simulations with field theoretic priors. In T. K. Leen, T. G. Dietterich, & V. Tresp (Eds.), Advances in neural information processing systems, 13 (pp. 287-295). MIT Press, Cambridge, MA: MIT Press. See also cond-mat/0009165.
66. Opper, M. (1994). Learning and generalization in a two-layer neural network: The role of the Vapnik-Chervonenkis dimension. Phys. Rev. Lett., 72, 2113-2116.
67. Opper, M., & Haussler, D. (1995). Bounds for predictive errors in the statistical mechanics of supervised learning. Phys. Rev. Lett., 75, 3772-3775.
68. Periwal, V. (1997). Reparametrization invariant statistical inference and gravity. Phys. Rev. Lett., 78, 4671-4674. See also hep-th/9703135.
69. Periwal, V. (1998). Geometrical statistical inference. Nucl. Phys. B, 554[FS], 719-730. See also adap-org/9801001.
70. Pöschel, T., Ebeling, W., & Rosé, H. (1995). Guessing probability distributions from small samples. J. Stat. Phys., 80, 1443-1452.
71. Reinagel, P., & Reid, R. C. (2000). Temporal coding of visual information in the thalamus. J. Neurosci., 20, 5392-5400.
72. Renyi, A. (1964). On the amount of information concerning an unknown parameter in a sequence of observations. Publ. Math. Inst. Hungar. Acad. Sci., 9, 617-625.
73. Rieke, F., Warland, D., de Ruyter van Steveninck, R., & Bialek, W. (1997). Spikes: Exploring the Neural Code (MIT Press, Cambridge).
74. Rissanen, J. (1978). Modeling by shortest data description. Automation, 14, 465-471.
75. Rissanen, J. (1984). Universal coding, information, prediction, and estimation. IEEE Trans. Inf. Thy., 30, 629-636.
76. Rissanen, J. (1986). Stochastic complexity and modeling. Ann. Statist., 14, 1080-1100.
77. Rissanen, J. (1987). Stochastic complexity. J. Roy. Stat. Soc. B, 49, 223-239, 253-265.
78. Rissanen, J. (1989). Stochastic complexity and statistical inquiry. Singapore: World Scientific.
79. Rissanen, J. (1996). Fisher information and stochastic complexity. IEEE Trans. Inf. Thy., 42, 40-47.
80. Rissanen, J., Speed, T., & Yu, B. (1992). Density estimation by stochastic complexity. IEEE Trans. Inf. Thy., 38, 315-323.
81. Saffran, J. R., Aslin, R. N., & Newport, E. L. (1996). Statistical learning by 8-month-old infants. Science, 274, 1926-1928.
82. Saffran, J. R., Johnson, E. K., Aslin, R. H., & Newport, E. L. (1999). Statistical learning of tone sequences by human infants and adults. Cognition, 70, 27-52.
83. Schmidt, D. M. (2000). Continuous probability distributions from finite data. Phys. Rev. E, 61, 1052-1055.
84. Seung, H. S., Sompolinsky, H., & Tishby, N. (1992). Statistical mechanics of learning from examples. Phys. Rev. A, 45, 6056-6091.
85. Shalizi, C. R., & Crutchfield, J. P. (1999). Computational mechanics: Pattern and prediction, structure and simplicity. Preprint. Available at cond-mat/9907176.
86. Shalizi, C. R., & Crutchfield, J. P. (2000). Information bottleneck, causal states, and statistical relevance bases: How to represent relevant information in memoryless transduction. Available at nlin.AO/0006025.
87. Shannon, C. E. (1948). A mathematical theory of communication. Bell Sys. Tech. J., 27, 379-423, 623-656. Reprinted in C. E. Shannon & W. Weaver, The mathematical theory of communication. Urbana: University of Illinois Press, 1949.
88. Shannon, C. E. (1948). A mathematical theory of communication. Bell Sys. Tech. J., 27, 379-423, 623-656. Reprinted in C. E. Shannon & W. Weaver, The mathematical theory of communication. Urbana: University of Illinois Press, 1949.
89. Shannon, C. E. (1951). Prediction and entropy of printed English. Bell Sys. Tech. J., 30, 50-64. Reprinted in N. J. A. Sloane & A. D. Wyner, (Eds.), Claude Elwood Shannon: Collected papers. New York: IEEE Press, 1993.
90. Shannon, C. E. (1951). Prediction and entropy of printed English. Bell Sys. Tech. J., 30, 50-64. Reprinted in N. J. A. Sloane & A. D. Wyner, (Eds.), Claude Elwood Shannon: Collected papers. New York: IEEE Press, 1993.
91. Shiner, J., Davison, M., & Landsberger, P. (1999). Simple measure for complexity. Phys. Rev. E, 59, 1459-1464.
92. Smirnakis, S., Berry III, M. J., Warland, D. K., Bialek, W., & Meister, M. (1997). Adaptation of retinal processing to image contrast and spatial scale. Nature, 386, 69-73.
93. Sole, R. V., & Luque, B. (1999). Statistical measures of complexity for strongly interacting systems. Preprint. Available at adap-org/9909002.
94. Solomonoff, R. J. (1964). A formal theory of inductive inference. Inform, and Control, 7, 1-22, 224-254.
95. Strong, S. P., Koberle, R., de Ruyter van Steveninck, R., & Bialek, W. (1998). Entropy and information in neural spike trains. Phys. Rev. Lett., 80, 197-200.
96. Tishby, N., Pereira, F., & Bialek, W. (1999). The information bottleneck method. In B. Hajek & R. S. Sreenivas (Eds.), Proceedings of the 37th Annual Allerton Conference on Communication, Control and Computing (pp. 368-377). Urbana: University of Illinois. See also physics/0004057.
97. Vapnik, V. (1998). Statistical learning theory. New York: Wiley.
98. Vitányi, P., & Li, M. (2000). Minimum description length induction, Bayesianism, and Kolmogorov complexity. IEEE Trans. Inf. Thy., 46, 446-464. See also cs.LG/9901014.
99. Wallace, C. S., & Boulton, D. M. (1968). An information measure for classification. Comp. J., 11, 185-195.
100. Weigend, A. S., & Gershenfeld, N. A., eds. (1994). Time series prediction: Forecasting the future and understanding the past. Reading, MA: Addison-Wesley.
101. Wiener, N. (1949). Extrapolation, interpolation and smoothing of time series. New York: Wiley.
102. Wolfram, S. (1984). Computation theory of cellular automata. Commun. Math. Physics, 96, 15-57.
103. Wong, W., & Shen, X. (1995). Probability inequalities for likelihood ratios and convergence rates of sieve MLE's. Ann. Statist., 23, 339-362.