Computational mechanics: Pattern and prediction, structure and simplicity

Journal of Statistical Physics

Volumn 104, Issue 3-4, 2001, Pages 817-879

  Shalizi, Cosma Rohilla ^a,b   Crutchfield, James P ^a  

^a SANTA FE INSTITUTE   (United States)

^b University of Wisconsin Madison   (United States)

Author keywords

machine; Causal state; Complexity; Computation; Entropy; Information; Pattern; Statistical mechanics

Indexed keywords

EID: 0035429965     PISSN: 00224715     EISSN: None     Source Type: Journal    
DOI: 10.1023/A:1010388907793     Document Type: Article

References (144)

1. J. M. Yeomans, Statistical Mechanics of Phase Transitions (Clarendon Press, Oxford, 1992).
2. P. Manneville, Dissipative Structures and Weak Turbulence (Academic Press, Boston, Massachusetts, 1990).
3. P. M. Chaikin and T. C. Lubensky, Principles of Condensed Matter Physics (Cambridge University Press, Cambridge, England, 1995).
4. M. C. Cross and P. Hohenberg, Pattern Formation Out of Equilibrium, Reviews of Modern Physics 65:851-1112 (1993).
5. J. P. Crutchfield, The calculi of emergence: Computation, dynamics, and induction, Physica D 75:11-54 (1994).
6. J. P. Crutchfield and K. Young, Inferring statistical complexity, Physical Review Letters 63:105-108 (1989).
7. (371) pages 223-269.
8. N. Perry and P.-M. Binder, Finite statistical complexity for sofic systems, Physical Review E 60:459-463 (1999).
9. J. E. Hanson and J. P. Crutchfield, Computational mechanics of cellular automata: An example, Physica D 103:169-189 (1997).
10. D. R. Upper, Theory and Algorithms for Hidden Markov Models and Generalized Hidden Markov Models, PhD thesis (University of California, Berkeley, 1997). Online from http://www.santafe.edu/projects/CompMech/.
11. J. P. Crutchfield and M. Mitchell, The evolution of emergent computation, Proceedings of the National Academy of Sciences 92:10742-10746 (1995).
12. A. Witt, A. Neiman, and J. Kurths, Characterizing the dynamics of stochastic bistable systems by measures of complexity, Physical Review E 55:5050-5059 (1997).
13. J. Delgado and R. V. Solé, Collective-induced computation, Physical Review E 55:2338-2344 (1997).
14. W. M. Gonçalves, R. D. Pinto, J. C. Sartorelli, and M. J. de Oliveira, Inferring statistical complexity in the dripping faucet experiment, Physica A 257:385-389 (1998).
15. A. J. Palmer, C. W. Fairall, and W. A. Brewer, Complexity in the atmosphere, IEEE Transactions on Geoscience and Remote Sensing 38:2056-2063 (2000).
16. J. P. Crutchfield and C. R. Shalizi, Thermodynamic depth of causal states: Objective complexity via minimal representations, Physical Review E 59:275-283 (1999). E-print, arxiv.org, cond-mat/9808147.
17. J. L. Borges, Other Inquisitions, 1937-1952 (University of Texas Press, Austin, 1964). Trans. Ruth L. C. Simms.
18. J. P. Crutchfield, Semantics and thermodynamics, In Nonlinear Modeling and Forecasting, M. Casdagli and S. Eubank, eds., volume 12 of Santa Fe Institute Studies in the Sciences of Complexity, pages 317-359 (Addison-Wesley, Reading, Massachusetts, 1992).
19. Plato, Phaedrus.
20. A. R. Luria, The Working Brain: An Introduction to Neuropsychology (Basic Books, New York, 1973).
21. N. V. S. Graham, Visual Pattern Analyzers, volume 16 of Oxford Psychology Series (Oxford University Press, Oxford, 1989).
22. S. J. Shettleworth, Cognition, Evolution and Behavior (Oxford University Press, Oxford, 1998).
23. J. T. Tou and R. C. Gonzalez, Pattern Recognition Principles (Addison-Wesley, Reading, Massachusetts, 1974).
24. S. P. Banks, Signal Processing, Image Processing, and Pattern Recognition (Prentice Hall, New York, 1990).
25. J. S. Lim, Two-Dimensional Signal and Image Processing (Prentice Hall, New York, 1990).
26. Plato, Meno, In 8OD Meno says: "How will you look for it, Socrates, when you do not know at all what it is? How will you aim to search for something you do not know at all? If you should meet it, how will you know that this is the thing that you did not know?" The same difficulty is raised in Theaetetus, 197 et seq.
27. A. N. Whitehead and B. Russell, Principia Mathematica, 2nd ed. (Cambridge University Press, Cambridge, England), pp. 1925-27.
28. B. Russell, Introduction to Mathematical Philosophy, The Muirhead Library of Philosophy, revised ed. (George Allen and Unwin, London, 1920). First edition, 1919. Reprinted New York: Dover Books, 1993.
29. J. P. Crutchfield, Information and its metric, In Nonlinear Structures in Physical Systems-Pattern Formation, Chaos and Waves, L. Lam and H. C. Morris, eds. (Springer-Verlag, New York, 1990), pp. 119.
30. B. Russell, Human Knowledge: Its Scope and Limits (Simon and Schuster, New York, 1948).
31. J. Rhodes, Applications of Automata Theory and Algebra via the Mathematical Theory of Complexity to Biology, Physics, Psychology, Philosophy, Games, and Codes (University of California, Berkeley, California, 1971).
32. C. L. Nehaniv and J. L. Rhodes, Krohn-Rhodes theory, hierarchies, and evolution, In Mathematical Hierarchies and Biology: DIMACS workshop, November 13-15, 1996, B. Mirkin, F. R. McMorris, F. S. Roberts, and A. Rzhetsky, eds., volume 37 of DIMACS Series in Discrete Mathematics and Theoretical Computer Science (American Mathematical Society, Providence, Rhode Island, 1997).
33. U. Grenander, Elements of Pattern Theory, Johns Hopkins Studies in the Mathematical Sciences (Johns Hopkins University Press, Baltimore, Maryland, 1996).
34. H. Jaeger, Observable operator models for discrete stochastic time series, Neural Computation 12:1371-1398 (2000).
35. U. Grenander, Y. Chow, and D. M. Keenan, Hands: A Pattern Theoretic Study of Biological Shapes, volume 2 of Research Notes in Neural Computing (Springer-Verlag, New York, 1991).
36. U. Grenander and K. Manbeck, A stochastic shape and color model for defect detection in potatoes, American Statistical Association 12:131-151 (1993).
37. A. N. Kolmogorov, Three approaches to the quantitative definition of information, Problems of Information Transmission 1:1-7 (1965).
38. G. Chaitin, On the length of programs for computing finite binary sequences, Journal of the Association for Computing Machinery 13:547-569 (1966).
39. A. N. Kolmogorov, Combinatorial foundations of information theory and the calculus of probabilities, Russian Mathematical Surveys 38:29 (1983).
40. M. Li and P. M. B. Vitanyi, An Introduction to Kolmogorov Complexity and its Applications (Springer-Verlag, New York, 1993).
41. M. Minsky, Computation: Finite and Infinite Machines (Prentice-Hall, Englewood Cliffs, New Jersey, 1967).
42. P. Martin-Löf, The definition of random sequences, Information and Control 9:602-619 (1966).
43. L. A. Levin, Laws of information conservation (nongrowth) and aspects of the foundation of probability theory, Problemy Peredachi Informatsii 10:30-35 (1974). Translation: Problems of Information Transmission 10:206-210 (1974).
44. L. A. Levin, Laws of information conservation (nongrowth) and aspects of the foundation of probability theory, Problemy Peredachi Informatsii 10:30-35 (1974). Translation: Problems of Information Transmission 10:206-210 (1974).
45. V. G. Gurzadyan, Kolmogorov complexity as a descriptor of cosmic microwave background maps, Europhysics Letters 46:114-117 (1999).
46. R. J. Solomonoff, A formal theory of inductive inference, Information and Control 7:1-22 and 224-254 (1964).
47. P. M. B. Vitányi and M. Li, Minimum description length induction, Bayesianism, and Kolmogorov complexity, E-print, arxiv.org, cs.LG/9901014, 1999.
48. G. W. Flake, The Computational Beauty of Nature: Computer Explorations of Fractals, Chaos, Complex Systems and Adaptation (MIT Press, Cambridge, Massachusetts, 1998).
49. J. Rissanen, Stochastic Complexity in Statistical Inquiry (World Scientific, Singapore, 1989).
50. (137) pages 137-148.
51. M. Koppel, Complexity, depth, and sophistication, Complex Systems 1:1087-1091 (1987).
52. M. Koppel and H. Atlan, An almost machine-independent theory of program-length complexity, sophistication and induction, Information Sciences 56:23-44 (1991).
53. D. C. Dennett, Real patterns, Journal of Philosophy 88:27-51 (1991). Reprinted in ref. 138.
54. J. P. Crutchfield, Is anything ever new? Considering emergence, In Complexity: Metaphors, Models, and Reality, G. Cowan, D. Pines, and D. Melzner, eds., volume 19 of Santa Fe Institute Studies in the Sciences of Complexity (Addison-Wesley, Reading, Massachusetts, 1994), pp. 479-497.
55. J. H. Holland, Emergence: From Chaos to Order (Addison-Wesley, Reading, Massachusetts, 1998).
56. L. Boltzmann, Lectures on Gas Theory (University of California Press, Berkeley, 1964).
57. H. Cramér, Mathematical Methods of Statistics (Almqvist and Wiksells, Uppsala, 1945). Republished by Princeton University Press, 1946, as vol. 9 in the Princeton Mathematics Series, and as a paperback, in the Princeton Landmarks in Mathematics and Physics series, 1999.
58. C. E. Shannon, A mathematical theory of communication, Bell System Technical Journal 27:379-423 (1948). Reprinted in ref. 139.
59. D. Hume, A Treatise of Human Nature: Being an Attempt to Introduce the Experimental Method of Reasoning into Moral Subjects (John Noon, London, 1739). Reprint (Oxford: Clarendon Press, 1951) of original edition, with notes and analytical index.
60. M. Bunge, Causality: The Place of the Causal Princple in Modern Science (Harvard University Press, Cambridge, Massachusetts, 1959). Reprinted as Causality and Modern Science, NY: Dover Books, 1979.
61. M. Bunge, Causality: The Place of the Causal Princple in Modern Science (Harvard University Press, Cambridge, Massachusetts, 1959). Reprinted as Causality and Modern Science, NY: Dover Books, 1979.
62. W. C. Salmon, Scientific Explanation and the Causal Structure of the World (Princeton University Press, Princeton, 1984).
63. P. Billingsley, Ergodic Theory and Information, Tracts on Probablity and Mathematical Statistics (Wiley, New York, 1965).
64. R. M. Gray, Entropy and Information Theory (Springer-Verlag, New York, 1990).
65. T. M. Cover and J. A. Thomas, Elements of Information Theory (Wiley, New York, 1991).
66. W. O. Ockham, Philosophical Writings: A Selection, Translated, with an Introduction, by Philotheus Boehner, O.F.M., Late Professor of Philosophy, The Franciscan Institute (Bobbs-Merrill, Indianapolis, 1964). First pub. various European cities, early 1300s.
67. Anonymous, Kuan Yin Tzu, T'ang Dynasty, Written in China during the T'ang dynasty. Partial translation in Joseph Needham, Science and Civilisation in China, vol. II (Cambridge University Press, 1956), p. 73.
68. D. P. Feldman and J. P. Crutchfield, Discovering non-critical organization: Statistical mechanical, information theoretic, and computational views of patterns in simple one-dimensional spin systems, Journal of Statistical Physics submitted (1998), Santa Fe Institute Working Paper 98-04-026, http://www.santafe.edu/projects/CompMech/papers/ DNCO.html.
69. J. E. Hopcroft and J. D. Ullman, Introduction to Automata Theory, Languages, and Computation (Addison-Wesley, Reading, 1979), 2nd edition of Formal Languages and Their Relation to Automata, 1969.
70. H. R. Lewis and C. H. Papadimitriou, Elements of the Theory of Computation, 2nd ed. (Prentice-Hall, Upper Saddle River, New Jersey, 1998).
71. J. G. Kemeny and J. L. Snell, Finite Markov Chains (Springer-Verlag, New York, 1976).
72. J. G. Kemeny, J. L. Snell, and A. W. Knapp, Denumerable Markov Chains, 2nd ed. (Springer-Verlag, New York, 1976).
73. J. E. Hanson, Computational Mechanics of Cellular Automata, PhD thesis (University of California, Berkeley, 1993).
74. G. Bateson, Mind and Nature: A Necessary Unity (E. P. Dutton, New York, 1979).
75. S. Kullback, Information Theory and Statistics, 2nd ed. (Dover Books, New York, 1968). First edition New York: Wiley, 1959.
76. K. L. Klinkner, C. R. Shalizi, and J. P. Crutchfield, Extensive state estimation: Reconstructing causal states by splitting, Manuscript in preparation, 2001.
77. J. P. Crutchfield and N. H. Packard, Symbolic dynamics of noisy chaos, Physica D 7:201-223 (1983).
78. R. Shaw, The Dripping Faucet as a Model Chaotic System (Aerial Press, Santa Cruz, California, 1984).
79. P. Grassberger, Toward a quantitative theory of self-generated complexity, International Journal of Theoretical Physics 25:907-938 (1986).
80. K. Lindgren and M. G. Nordahl, Complexity measures and cellular automata, Complex Systems 2:409-440 (1988).
81. W. Li, On the relationship between complexity and entropy for Markov chains and regular languages, Complex Systems 5:381-399 (1991).
82. D. Arnold, Information-theoretic analysis of phase transitions, Complex Systems 10:143-155 (1996).
83. W. Bialek and N. Tishby, Predictive information, E-print, arxiv.org, cond-mat/ 9902341, 1999.
84. J. P. Crutchfield and D. P. Feldman, Statistical complexity of simple one-dimensional spin systems, Physical Review E 55:1239R-1243R (1997).
85. W. R. Ashby, An Introduction to Cybernetics (Chapman and Hall, London, 1956).
86. H. Touchette and S. Lloyd, Information-theoretic limits of control, Physical Review Letters 84:1156-1159 (1999).
87. A. Lempel and J. Ziv, Compression of two-dimensional data, IEEE Transactions in Information Theory IT-32:2-8 (1986).
88. D. P. Feldman, Computational Mechanics of Classical Spin Systems, PhD thesis (University of California, Davis, 1998). Online at http://hornacek.coa.edu/dave/ Thesis/thesis.html.
89. D. G. Mayo, Error and the Growth of Experimental Knowledge (Science and Its Conceptual Foundations, University of Chicago Press, Chicago, 1996).
90. J. P. Crutchfield and C. Douglas, Imagined complexity: Learning a random process. Manuscript in preparation, 1999.
91. R. Lidl and G. Pilz, Applied Abstract Algebra (Springer, New York, 1984).
92. E. S. Ljapin, Semigroups, volume 3 of Translations of Mathematical Monographs (American Mathematical Society, Providence, Rhode Island, 1963).
93. K. Young, The Grammar and Statistical Mechanics of Complex Physical Systems, PhD thesis (University of California, Santa Cruz, 1991).
94. P. Billingsley, Probability and Measure, Wiley Series in Probability and Mathematical Statistics (Wiley, New York, 1979).
95. J. L. Doob, Stochastic Processes, Wiley Publications in Statistics (Wiley, New York, 1953).
96. M. Loéve, Probability Theory, 1st ed. (D. Van Nostrand Company, New York, 1955).
97. M. M. Rao, Conditional Measures and Applications, volume 177 of Monographs and Textbooks in Pure and Applied Mathematics (Marcel Dekker, New York, 1993).
98. N. H. Packard, J. P. Crutchfield, J. D. Farmer, and R. S. Shaw, Geometry from a time series, Physical Review Letters 45:712-716 (1980).
99. F. Takens, Detecting strange attractors in fluid turbulence, In D. A. Rand and L. S. Young, editors, Symposium on Dynamical Systems and Turbulence, volume 898 of Lecture Notes in Mathematics (Springer-Verlag, Berlin, 1981), pp. 366.
100. J. P. Crutchfield and B. S. McNamara, Equations of motion from a data series, Complex Systems 1:417-452 (1987).
101. J. Neyman, First Course in Probability and Statistics (Henry Holt, New York, 1950).
102. D. Blackwell and M. A. Girshick, Theory of Games and Statistical Decisions (Wiley, New York, 1954). Reprinted New York: Dover Books, 1979.
103. R. D. Luce and H. Raiffa, Games and Decisions: Introduction and Critical Survey (Wiley, New York, 1957).
104. I. M. Gel'fand and A. M. Yaglom, Calculation of the amount of information about a random function contained in another such function, Uspekhi Matematicheski Nauk 12:3-52 (1956). Trans. in American Mathematical Society Translations 12(2):199-246 (1959).
105. I. M. Gel'fand and A. M. Yaglom, Calculation of the amount of information about a random function contained in another such function, Uspekhi Matematicheski Nauk 12:3-52 (1956). Trans. in American Mathematical Society Translations 12(2):199-246 (1959).
106. P. E. Caines, Linear Stochastic Systems (Wiley, New York, 1988).
107. D. Blackwell and L. Koopmans, On the identifiability problem for functions of finite Markov chains, Annals of Mathematical Statistics 28:1011-1015 (1957).
108. H. Ito, S.-I. Amari, and K. Kobayashi, Identifiability of hidden Markov information sources and their minimum degrees of freedom, IEEE Transactions on Information Theory 38:324-333 (1992).
109. P. Algoet, Universal schemes for prediction, gambling and portfolio selection, The Annals of Probability 20:901-941 (1992). See also an important Correction, The Annals of Probability 23:474-478 (1995).
110. P. Algoet, Universal schemes for prediction, gambling and portfolio selection, The Annals of Probability 20:901-941 (1992). See also an important Correction, The Annals of Probability 23:474-478 (1995).
111. A. N. Kolmogorov, Interpolation und extrapolation von stationären zufälligen folgen, Bull. Acad. Sci. U.S.S.R., Math. 3:3-14 (1941). In Russian with German summary.
112. 2, National Defense Research Council".
113. N. Wiener, Nonlinear Problems in Random Theory (The Technology Press of the Massachusetts Institute of Technology, Cambridge, Massachusetts, 1958).
114. M. Schetzen, The Volterra and Wiener Theories of Nonlinear Systems, 2nd ed. (Robert E. Krieger Publishing Company, Malabar, Florida, 1989). Reprint, with additions, of the first edition, New York: John Wiley, 1980.
115. N. Wiener, Cybernetics: Or, Control and Communication in the Animal and the Machine, 2nd ed. (MIT Press, Cambridge, Massachusetts, 1961). First edition New York: Wiley, 1948.
116. N. Chomsky, Three models for the description of language, IRE Transactions on Information Theory 2:113 (1956).
117. N. Chomsky, Syntactic Structures, volume 4 of Janua linguarum, series minor (Mouton, The Hauge, 1957).
118. B. A. Trakhtenbrot and Y. M. Barzdin, Finite Automata (North-Holland, Amsterdam, 1973).
119. E. Charniak, Statistical Language Learning, Language, Speech and Communication (MIT Press, Cambridge, Massachusetts, 1993).
120. M. J. Kearns and U. V. Vazirani, An Introduction to Computational Learning Theory (MIT Press, Cambridge, Massachusetts, 1994).
121. V. N. Vapnik, The Nature of Statistical Learning Theory, 2nd ed. (Springer-Verlag, Berlin, 2000).
122. L. G. Valiant, A theory of the learnable, Communications of the Association for Computing Machinery 27:1134-1142 (1984).
123. M. A. Boden, Precis of "The Creative Mind: Myths and Mechanisms," Behaviorial and Brain Sciences 17:519-531 (1994).
124. C. Thornton, Truth from Trash: How Learning Makes Sense, Complex Adaptive Systems (MIT Press, Cambridge, Massachusetts, 2000).
125. J. Pearl, Causality: Models, Reasoning, and Inference (Cambridge University Press, Cambridge, England, 2000).
126. M. I. Jordan, ed., Learning in Graphical Models, volume 89 of NATO Science Series D: Behavioral and Social Sciences (Dordrecht, 1998).
127. P. Spirtes, C. Glymour, and R. Scheines, Causation, Prediction, and Search, Adaptive Computation and Machine Learning (MIT Press, Cambridge, Massachusetts, 2000).
128. D. V. Lindley, Bayesian Statistics, a Review (Society for Industrial and Applied Mathematics, Philadelphia, 1972).
129. J. Rissanen, A universal data compression system, IEEE Transactions in Information Theory IT-29:656-664 (1983).
130. P. Bühlmann and A. J. Wyner, Variable length Markov chains, Technical Report 497 (UC Berkeley Statistics Department, 1997). Online from http://www.stat. berkeley.edu/tech-reports/.
131. A. Lempel and J. Ziv, On the complexity of finite sequences, IEEE Transactions in Information Theory IT-22:75-81 (1976).
132. J. Ziv and A. Lempel, A universal algorithm for sequential data compression, IEEE Transactions in Information Theory 11-23:337-343 (1977).
133. R. Badii and A. Politi, Complexity: Hierarchical Structures and Scaling in Physics, volume 6 of Cambridge Nonlinear Science Series (Cambridge University Press, Cambridge, 1997).
134. B. Weiss, Subshifts of finite type and sofic systems, Monatshefte für Mathematik 77:462-474 (1973).
135. C. Moore, Recursion theory on the reals and continuous-time computation, Theoretical Computer Science 162:23-44 (1996).
136. C. Moore, Dynamical recognizers: Real-time language recognition by analog computers, Theoretical Computer Science 201:99-136 (1998).
137. P. Orponen, A survey of continuous-time computation theory, In D.-Z. Du and K.-I. Ko, editors, Advances in Algorithms, Languages, and Complexity (Kluwer Academic, Dordrecht, 1997), pp. 209-224.
138. L. Blum, M. Shub, and S. Smale, On a theory of computation and complexity over the real numbers: NP-completeness, recursive functions and universal machines, Bulletin of the American Mathematical Society 21:1-46 (1989).
139. C. Moore, Unpredictability and undecidability in dynamical systems, Physical Review Letters 64:2354-2357 (1990).
140. S. Sinha and W. L. Ditto, Dynamics based computation, Physical Review Letters 81:2156-2159 (1998).
141. W. H. Zurek, editor, Complexity, Entropy, and the Physics of Information, volume 8 of Santa Fe Institute Studies in the Sciences of Complexity (Addison-Wesley, Reading, Massachusetts, 1990).
142. D. C. Dennett, Brainchildren: Essays on Designing Minds, Representation and Mind (MIT Press, Cambridge, Massachusetts, 1997).
143. C. E. Shannon and W. Weaver, The Mathematical Theory of Communication (University of Illinois Press, Urbana, Illinois, 1963).
144. B. F. Schutz, Geometrical Methods of Mathematical Physics (Cambridge University Press, Cambridge, England, 1980).