An information-theoretic primer on complexity, self-organization, and emergence

Complexity

Volumn 15, Issue 1, 2009, Pages 11-28

  Prokopenko, Mikhail ^a   Boschetti, Fabio ^a   Ryan, Alex J ^b  

^a CSIRO   (Australia)

^b DEFENCE SCIENCE AND TECHNOLOGY ORGANISATION   (Australia)

Author keywords

Adaptation; Assortativeness; Complexity; Emergence; Entropy rate; Excess entropy; Information theory; Predictive efficiency; Predictive information; Self organization

Indexed keywords

EID: 70349629569     PISSN: 10762787     EISSN: 10990526     Source Type: Journal    
DOI: 10.1002/cplx.20249     Document Type: Article

References (73)

1. Shannon, C.E. A mathematical theory of communication.The Bell Syst Tech J 1948, 27, 379-423, 623-656.
2. David, J.C.M. Information Theory, Inference, and Learning Algorithms; Cambridge University Press: Cambridge, UK, 2003.
3. Newman, M.E.J.Assortative mixing in networks. Phys Rev Lett 2002, 89, 208701.
4. Sole, R.V.; Valverde, S. Information theory of complex networks:on evolution and architectural constraints, In Complex Networks, Vol.650: Lecture Notes in Physics; Ben-Naim, E.; Frauenfelder, H.; Toroczkai, Z., Eds.; Springer: Berlin, 2004.
5. Schreiber, T. Measuring information transfer. Phys Rev Lett 2000, 85, 461.
6. Lizier, J.T.; Prokopenko, M.; Zomaya,A.Y. Local information transfer as a spatiotemporal filter for complex systems. Phys Rev E 2008, 77, 026110.
7. Lungarella, M.; Sporns, O. Mapping information flow in sensorimotor networks. PLoS Comput Biol 2006, e144.
8. Ay, N.; Polani, D. Information flows in causal networks.Adv Complex Syst 2008, 11, 17-41.
9. Polani, D.; Nehaniv, C.; Martinetz, T.; Kim, J.T. Relevant information in optimized persistence vs. progeny strategies, In: Artificial Life X: Proceedings of the 10th International Conference on the Simulation and Synthesis of Living Systems; Rocha, L.M.; Yaeger, L.S.; Bedau, M.A.; Floreano, D.; Goldstone, R.L.; Vespignani,A., Eds.; Bloomington IN, USA, 2006.
10. Foreman, M.; Prokopenko, M.; Wang, P. Phase transitions in self-organising sensor networks, In: Advances in Artificial Life-Proceedings of the 7th European Conference on Artificial Life (ECAL), Vol. 2801: Lecture Notes in Artificial Intelligence; Banzhaf, W.; Christaller, T.; Dittrich, P.; Kim, J.T.; Ziegler, J., Eds.; Springer Verlag: Berlin, 2003, pp. 781-791.
11. Klyubin, A.S.; Polani, D.; Nehaniv, C.L. Organization of the information flow in the perception-action loop of evolved agents, In: Proceedings of 2004 NASA/DoD Conference on Evolvable Hardware; IEEE Computer Society, 2004, pp. 177-180.
12. Klyubin, A.S.; Polani, D.; Nehaniv, C.L. All else being equal be empowered. In: Advances in Artificial Life, 8th European Conference, ECAL 2005, Vol. 3630: LNCS; Capcarr'ere, M.S.; Freitas,A.A.; Bentley, P.J.; Johnson, C.G.; Timmis, J., Eds.; Springer: Berlin, 2005, pp. 744-753.
13. Nehaniv, C.L.; Polani, D.; Olsson, L.A.; Klyubin,A.S. Evolutionary information-theoretic foundations of sensory ecology: Channels of organism-specific meaningful information,In:Modeling Biology:Structures,Behaviour,Evolution,Vienna Series in Theoretical Biology; Fontoura Costa,L.; Müller,G.B.,Eds.; MIT Press:Cambridge, MA, 2005.
14. Prokopenko, M.; Gerasimov, V.; Tanev, I. Evolving spatiotemporal coordination in a modular robotic system, In: From Animals to Animats 9: 9th International Conference on the Simulation of Adaptive Behavior(SAB2006), Rome, Italy, September 25-29 2006, Vol. 4095: Lecture Notes in Computer Science, Nolfi, S., Baldassarre, G., Calabretta, R., Hallam, J., Marocco, D., Meyer, J.-A., Parisi, D., Eds.; Springer Verlag: Berlin, 2006, pp. 558-569.
15. Prokopenko, M.; Gerasimov, V.; Tanev, I. Measuring spatiotemporal coordination in a modular robotic system, In: Artificial Life X: Proceedings of The 10th International Conference on the Simulation and Synthesis of Living Systems; Rocha, L.M.; Yaeger, L.S.; Bedau, M.A.; Floreano, D.; Goldstone, R.L.; Vespignani,A., Eds.; Bloomington IN, USA, 2006, pp. 185-191.
16. Piraveenan, M.; Polani, D.; Prokopenko, M. Emergence of genetic coding: an information-theoretic model, In: Advances in Artificial Life: 9th European Conference on Artificial Life (ECAL-2007), Lisbon, Portugal, September 10-14, Vol. 4648: Lecture Notes in Artificial Intelligence; Almeida e Costa, F.; Rocha, L.M.; Costa, E.; Harvey, I.; Coutinho,A., Eds.; Springer: Berlin, 2007, pp. 42-52.
17. Adami, C. Introduction to Artificial Life; Springer: Berlin, 1998.
18. Wojciech H. Zurek, Ed. Valuable Information, Santa Fe Studies in the Sciences of Complexity; Addison-Wesley: Reading, Mass.: 1990.
19. Polani, D.; Prokopenko, M.; Chadwick, M. Modelling stigmergic gene transfer, In:Artificial Life XI: Proceedings of The 11th International Conference on the Simulation and Synthesis of Living Systems; Bullock, S.; Noble, J.; Watson, R.; Bedau, M., Eds.; MIT Press:Winchester, UK, 2008.
20. Boschetti, F. Mapping the complexity of ecological models. Ecol Complexity 2008, 5, 37-47.
21. Li, M.; Vitanyi, P.M.B.An Introduction to Kolmogorov Complexity and its Applications, 2nd ed.; Springer-Verlag: New York, 1997.
22. Chaitin, G.J.Algorithmic Information Theory; Cambridge University Press: Cambridge, UK, 1987.
23. Chaitin, G.J. Information-theoretic limitations of formal systems. J ACM 1974, 21, 403-424.
24. Crutchfield, J.P.; Young, K. Inferring statistical complexity. Phys Rev Lett 1989, 63,105-108.
25. Grassberger, P.Toward a quantitative theory of selfgenerated complexity. Int J Theor Phys 1986, 25, 907-938.
26. Boffetta, G.; Cencini, M.; Falcioni, M.; Vulpiani,A. Predictability:A way to characterize complexity. Phys Rep 2002, 356, 367-474.
27. Bialek, W.; Nemenman, I.; Tishby, N. Complexity through nonextensivity. Physica A 2001,302,89-99.
28. Crutchfield, J.P.; Packard, N.H. Symbolic dynamics of noisy chaos. Physica D 1983, 7, 201-223.
29. Crutchfield, J.P.; Feldman, D.P. Statistical complexity of simple one-dimensional spin systems. Phys Rev E 1997, 55, R1239-R1243.
30. Shaw, R.The Dripping Faucet as a Model Chaotic System; Aerial Press: Santa Cruz, California, 1984.
31. Lindgren, K.; Norhdal, M.G. Complexity measures and cellular automata. Complex Syst 1988, 2, 409-440.
32. Eriksson, K-E.; Lindgren, K. Structural information in self-organizing systems. Phys Scripta 1987, 35, 388-397.
33. Li, W. On the relationship between complexity and entropy for Markov chains and regular languages. Complex Syst 1991, 5, 381-399.
34. Arnold, D. Information-theoretic analysis of phase transitions. Complex Systems 1996, 10, 143-155.
35. Bialek, W.; Nemenman, I.; Tishby, N. Predictability, complexity, and learning. Neur Comp 2001, 13, 2409-2463.
36. James P. Crutchfield; Cosma Rohilla Shalizi Thermodynamic depth of causal states: Objective complexity via minimal representations. Phys Rev E 1999, 59, 275-283.
37. Shalizi, C.R.; Crutchfield, J.P. Computational mechanics: Pattern and prediction, structure and simplicity. J Stat Phys 2001, 104, 819-881.
38. Shalizi, C.R.; Shalizi, K.L. Optimal nonlinear prediction of random fields on networks. Discrete Math Theor Comput Sci 2003,AB(DMCS), 11-30.
39. Shalizi, C. Causal architecture, complexity and self-organization in time series and cellular automata. PhD thesis, University of Michigan, 2001.
40. Varn, D.P. Language extraction from ZnS. Phd thesis, University of Tennessee, 2001.
41. Shalizi, C.R.; Shalizi, K.L. Blind construction of optimal nonlinear recursive predictors for discrete sequences, In: Uncertainty in Artificial Intelligence: Proceedings of the Twentieth Conference; Chickering, M.; Joseph Halpern, J., Eds.; AUAI Press:Arlington,Virginia, 2004, pp. 504-511.
42. Feldman, D.P.; Crutchfield, J.P. Discovering noncritical organization: Statistical mechanical, information theoretic, and computational views of patterns in one-dimensional spin systems,Technical Report 98-04-026, SFI Working Paper 1998.
43. Crutchfield, J.P.; Feldman, D.P. Regularities unseen, randomness observed: Levelsof entropy convergence. Chaos 2003, 13, 25-54.
44. Kolmogorov, A.N. Entropy per unit time as a metric invariant of automorphisms. Doklady Akademii Nauk SSSR 1959, 124, 754-755.
45. Ebeling, W. Prediction and entropy of nonlinear dynamical systems and symbolic sequences with lro. Physica D 1997, 109, 42-52.
46. Feldman, D.P.; Crutchfield, J.P. Structural information in two-dimensional patterns: Entropy convergence and excess entropy. Phys Rev E 2003, 67, 051104.
47. Cover, T.M.; Thomas, J.A. Elements of Information Theory; Wiley: New York, 1991.
48. Erdos, P.; Renyi,A. On the strength of connectedness of random graphs. Acta Math Sci Hungary 1961, 12, 261-267.
49. Callaway, D.S.; Hopcroft, J.E.; Kleinberg, J.M.; Newman, M.E.; Strogatz, S.H.Are randomly grown graphs really random? Phys Rev E October 2001, 64(4 Pt 1), 041902.
50. Ash, R.B. Information theory; Dover: London, 1965.
51. T De Wolf; Holvoet, T. Emergence versus self-organisation: Different concepts but promising when combined. In: Engineering Self-Organising Systems; Brueckner, S.; Serugendo, G.D.M.; Karageorgos,A.; Nagpal, R., Eds.; Springer: Berlin, 2005, pp. 1-15.
52. Shalizi, C.R.; Shalizi, K.L.; Haslinger, R. Quantifying self-organization with optimal predictors. Phys Rev Lett 2004, 93, 11870114.
53. Halley, J.; Winkler, D. Consistent concepts of self-organization and self-assembly.Complexity, in press.
54. Kauffman, S.A. Investigations; Oxford University Press: Oxford, 2000.
55. Correia, L. Self-organisation: a case for embodiment, In: Proceedings of The Evolution of Complexity Workshop at Artificial Life X:The 10th International Conference on the Simulation and Synthesis of Living Systems, 2006, pp. 111-116.
56. Wagner,A. Robustness and Evolvability in Living Systems; Princeton University Press: Princeton, NJ, 2005.
57. Ay, N.; Flack, J.; Krakauer, D. Robustness and complexity co-constructed in multimodal signalling networks. Philos Trans R Soc B 2007, 362, 441-447.
58. Hubbell, S.P.; Johnson, L.K.; Stanislav, E.; Wilson, B.; Fowler, H. Foraging by bucket-brigade in leafcutter ants. Biotropica 1980, 12, 210-213.
59. Rényi,A. Probability theory; North-Holland:Amsterdam, 1970.
60. Corning, P.A. The re-emergence of "emergence":A venerable concept in search of a theory. Complexity 2002, 7, 18-30.
61. Boschetti, F.; Prokopenko, M.; Macreadie, I.; Grisogono, A.-M. Defining and detecting emergence in complex networks, In: Knowledge-Based Intelligent Information and Engineering Systems, 9th International Conference, KES 2005, Melbourne, Australia, September 14-16, 2005; Proceedings, Part IV; Khosla, R.; Howlett, R.J.; Jain, L.C., Eds.; Vol. 3684: Lecture Notes in Computer Science; Springer: Berlin, 2005, pp. 573-580.
62. Crutchfield, J.The calculi of emergence: Computation, dynamics, and induction. Physica D 1994, 75, 11-54.
63. Grisogono, A.-M. Co-adaptation, In: SPIE Symposium on Microelectronics, MEMS and Nanotechnology, Vol. Paper 6039-1; Brisbane, Australia, 2005.
64. Sutton, R.S.; Barto,A.G. Reinforcement Learning:An Introduction,A Bradford Book; The MIT Press: Cambridge, MA, 1998.
65. Dawkins, R. Universal Darwinism, In: Evolution from Molecules to Men, Bendall, D.S., Ed.; Cambridge University Press, 1983.
66. Adami, C.What is complexity? Bioessays 2002, 24, 1085-1094.
67. Prokopenko, M.; Wang, P.; Price, D.C.; Valencia, P.; Foreman, M.; Farmer, A.J. Self-organizing hierarchies in sensor and communication networks. Artificial Life, Special Issue on Dynamic Hierarchies 2005, 11, 407-426.
68. Prokopenko, M.; Wang, P.; Foreman, M.; Valencia, P.; Price, D.C.; Poulton, G.T. On connectivity of reconfigurable impact networks in ageless aerospace vehicles. Journal of Robotics and Autonomous Systems 2005, 53, 36-58.
69. Mathews, G.; Durrant-Whyte, H.; Prokopenko, M. Measuring global behaviour of multi-agent systems from pair-wise mutual information. In: Knowledge-Based Intelligent Information and Engineering Systems, 9th International Conference, KES 2005, Melbourne, Australia, September 14-16, 2005; Proceedings, Part IV; Khosla, R.; Howlett, R.J.; Jain, L.C., Eds.; Vol. 3684: Lecture Notes in Computer Science; Springer: Berlin, 2005, pp. 587-594.
70. Ridley, M. Nature Via Nurture: Genes, Experience and What Makes Us Human; Fourth Estate, 2003.
71. Wolfram, S. Universality and complexity in cellular automata. Physica D 1984, 10, 1-35.
72. Casti, J.L. Chaos, godel and truth, In: Beyond Belief: Randomness, Prediction, and Explanation in Science; Casti, J.L.; Karlqvist,A., Eds.; CRC Press: London, 1991.
73. Langton, C. Computation at the edge of chaos: Phase transitions and emergent computation, In Emergent Computation; Forest, S. Ed.; MIT: Cambridge, MA, 1991.