Information processing in echo state networks at the edge of chaos

Theory in Biosciences

Volumn 131, Issue 3, 2012, Pages 205-213

  Boedecker, Joschka ^a,b   Obst, Oliver ^c,d   Lizier, Joseph T ^c,d   Mayer, N Michael ^e   Asada, Minoru ^a,b  

^a OSAKA UNIVERSITY   (Japan)

^b JAPAN SCIENCE AND TECHNOLOGY AGENCY   (Japan)

^c CSIRO   (Australia)

^d UNIVERSITY OF SYDNEY   (Australia)

^e NATIONAL CHUNG CHENG UNIVERSITY   (Taiwan)

Author keywords

Active information storage; Information transfer; Phase transition; Recurrent neural networks; Reservoir computing

Indexed keywords

ARTICLE; ARTIFICIAL NEURAL NETWORK; BRAIN CORTEX; HUMAN; INFORMATION SCIENCE; PHYSIOLOGY;

CEREBRAL CORTEX; HUMANS; INFORMATION THEORY; NEURAL NETWORKS (COMPUTER);

EID: 84865120516     PISSN: 14317613     EISSN: 16117530     Source Type: Journal    
DOI: 10.1007/s12064-011-0146-8     Document Type: Article

References (43)

1. N Ay N Bertschinger R Der F Güttler E Olbrich 2008 Predictive information and explorative behavior of autonomous robots Eur Phys J B 63 329 339 10.1140/epjb/e2008-00175-0 1:CAS:528:DC%2BD1cXpt1Whsbk%3D
2. JM Beggs 2008 The criticality hypothesis: how local cortical networks might optimize information processing Phil Trans R Soc A 366 1864 329 343 17673410 10.1098/rsta.2007.2092
3. JM Beggs D Plenz 2003 Neuronal avalanches in neocortical circuits J Neurosci 23 35 11,167 11,177 1:CAS:528:DC%2BD3sXpvV2gtbg%3D
4. Bell AJ, Sejnowski TJ (1995) An information-maximization approach to blind separation and blind deconvolution. Neural Comput, MIT Press 7(6):1129-1159
5. Bertschinger N, Natschläger T (2004) Real-time computation at the edge of chaos in recurrent neural networks. Neural Comput, MIT Press 16(7):1413-1436
6. J Boedecker O Obst NM Mayer M Asada 2009 Initialization and self-organized optimization of recurrent neural network connectivity HFSP J 3 5 340 349 20357891 10.2976/1.3240502
7. A Borst FE Theunissen 1999 Information theory and neural coding Nat Neurosci 2 947 957 10526332 10.1038/14731 1:CAS:528:DyaK1MXmvFGlsr4%3D (Pubitemid 30599556)
8. Büsing L, Schrauwen B, Legenstein R (2010) Connectivity, dynamics, and memory in reservoir computing with binary and analog neurons. Neural Comput, MIT Press 22(5):1272-1311
9. DR Chialvo 2004 Critical brain networks Physica A 340 4 756 765 10.1016/j.physa.2004.05.064
10. Cover TM, Thomas JA (2006) Elements of information theory. 2nd edn edn. Wiley, New York, NY
11. B Derrida Y Pomeau 1986 Random networks of automata: a simple annealed approximation Europhys Lett 1 2 45 49 10.1209/0295-5075/1/2/001
12. Jaeger H (2001a) The "echo state" approach to analysing and training recurrent neural networks. Tech Rep 148, GMD Report-German National Research Institute for Computer Science
13. Jaeger H (2001b) Short term memory in echo state networks. Tech Rep 152, GMD-German National Research Institute for Computer Science
14. H Jaeger H Haas 2004 Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication Science 304 5667 78 80 15064413 10.1126/science.1091277 1:CAS:528:DC%2BD2cXis1ansbc%3D (Pubitemid 38455427)
15. Klyubin AS, Polani D, Nehaniv CL (2004) Tracking information flow through the environment: simple cases of stigmergy. In: Pollack J, Bedau M, Husbands P, Ikegami T, Watson RA (eds) Proceedings of the 9th international conference on the simulation and synthesis of living systems. MIT Press, Cambridge, MA, pp 563-568
16. Klyubin AS, Polani D, Nehaniv CL (2005) All else being equal be empowered. In: Capcarrère MS, Freitas AA, Bentley PJ, Johnson CG, Timmis J (eds) Proceedings of the 8th European conference on artificial life, vol 3630. Lecture Notes in Artificial Intelligence. Springer, Heidelberg, pp 744-753
17. CG Langton 1990 Computation at the edge of chaos: phase transitions and emergent computation Physica D 42 1-3 12 37 10.1016/0167-2789(90)90064-V
18. Lazar A, Pipa G, Triesch J (2009) Sorn: a self-organizing recurrent neural network. Front Comput Neurosci 3(23). doi: 10.3389/neuro.10.023.2009
19. R Legenstein W Maass 2007 Edge of chaos and prediction of computational performance for neural circuit models Neural Networks 20 3 323 334 17517489 10.1016/j.neunet.2007.04.017 (Pubitemid 46856111)
20. Legenstein R, Maass W (2007b) What makes a dynamical system computationally powerful? In: Haykin S, Principe JC, Sejnowski T, McWhirter J (eds) New directions in statistical signal processing: from systems to brains, MIT Press, Cambridge, MA, pp 127-154
21. A Levina JM Herrmann T Geisel 2007 Dynamical synapses causing self-organized criticality in neural networks Nat Phys 3 12 857 860 10.1038/nphys758 1:CAS:528:DC%2BD2sXhtlOls7jE
22. Lizier JT, Prokopenko M, Zomaya AY (2007) Detecting non-trivial computation in complex dynamics. In: Almeida e Costa F, Rocha LM, Costa E, Harvey I, Coutinho A (eds) Proceedings of the 9th European conference on artificial life (ECAL 2007), Lisbon, Portugal, vol 4648. Springer, Lecture Notes in Artificial Intelligence, Berlin, Heidelberg, pp 895-904
23. Lizier JT, Prokopenko M, Zomaya AY (2008a) A framework for the local information dynamics of distributed computation in complex systems. http://arxiv.org/abs/0811.2690. Accessed 1 Nov 2010
24. Lizier JT, Prokopenko M, Zomaya AY (2008b) The information dynamics of phase transitions in random boolean networks. In: Bullock S, Noble J, Watson R, Bedau MA (eds) Proceedings of the 11th international conference on the simulation and synthesis of living systems (ALife XI), Winchester, UK. MIT Press, Cambridge, MA, pp 374-381
25. Lizier JT, Prokopenko M, Zomaya AY (2008c) Local information transfer as a spatiotemporal filter for complex systems. Phys Rev E 77(2):026,110
26. Lizier JT, Prokopenko M, Zomaya AY (2010) Coherent information structure in complex computation. Theory Biosci. (to appear)
27. M Lukosevicius H Jaeger 2009 Reservoir computing approaches to recurrent neural network training Comput Sci Rev 3 3 127 149 10.1016/j.cosrev.2009.03.005
28. M Lungarella O Sporns 2006 Mapping information flow in sensorimotor networks PLoS Comput Biol 2 10 e144 17069456 10.1371/journal.pcbi.0020144 (Pubitemid 44656523)
29. Maass W, Natschläger T, Markram H (2002) Real-time computing without stable states: a new framework for neural computation based on perturbations. Neural Comput, MIT Press 14(11):2531-2560
30. M Mitchell PT Hraber JP Crutchfield 1993 Revisiting the edge of chaos: evolving cellular automata to perform computations Complex Syst 7 89 130
31. Obst O, Boedecker J, Asada M (2010) Improving recurrent neural network performance using transfer entropy. In: Wong KW, Mendis BSU, Bouzerdoum A (eds) Neural information processing. Models and applications, vol 6444. Lecture Notes in Computer Science, Springer, Heidelberg pp 193-200
32. LA Olsson CL Nehaniv D Polani 2006 From unknown sensors and actuators to actions grounded in sensorimotor perceptions Connect Sci 18 2 121 144 10.1080/09540090600768542 (Pubitemid 43982718)
33. Prokopenko M, Gerasimov V, Tanev I (2006) Evolving spatiotemporal coordination in a modular robotic system. In: Nolfi S, Baldassarre G, Calabretta R, Hallam JCT, Marocco D, Meyer JA, Miglino O, Parisi D (eds) From animals to animats 9, 9th international conference on simulation of adaptive behavior, SAB 2006, vol 4095. Springer, Lecture Notes in Computer Science, Heidelberg, pp 558-569 (Pubitemid 44553386)
34. T Schreiber 2000 Measuring information transfer Phys Rev Lett 85 2 461 464 10991308 10.1103/PhysRevLett.85.461 1:CAS:528:DC%2BD3cXks1ersL4%3D
35. Shannon CE, Weaver W (1949) The mathematical theory of communication. University of Illinois Press, Urbana, IL
36. Sporns O, Lungarella M (2006) Evolving coordinated behavior by maximizing information structure. In: Rocha LM, Yaeger LS, Bedau MA, Floreano D, Goldstone RL, Vespignani A (eds) Proceedings of the 10th international conference on the simulation and synthesis of living systems, MIT Press, Cambridge, pp 323-329
37. Sprott JC (2003) Chaos and time-series analysis. Oxford University Press, Oxford. Accessed 1 Nov 2010
38. Sprott JC (2004) Numerical calculation of largest Lyapunov exponent. http://sprott.physics.wisc.edu/chaos/lyapexp.htm
39. S Strong R Koberle R van Steveninck W Bialek 1998 Entropy and information in neural spike trains Phys Rev Lett 80 197 200 10.1103/PhysRevLett.80.197 1:CAS:528:DyaK1cXislOjsQ%3D%3D (Pubitemid 128621483)
40. A Tang D Jackson 2008 A maximum entropy model applied to spatial and temporal correlations from cortical networks in vitro J Neurosci 28 505 518 18184793 10.1523/JNEUROSCI.3359-07.2008 1:CAS:528:DC%2BD1cXovVGhsQ%3D%3D
41. Tang A, Honey C, Hobbs J, Sher A, Litke A, Sporns O, Beggs J (2008) Information flow in local cortical networks is not democratic. BMC Neurosci 9(Suppl 1):O3. http://www.biomedcentral.com/1471-2202/9/S1/O3. doi: 10.1186/1471-2202-9-S1-O3
42. Triesch J (2005) A gradient rule for the plasticity of a neuron's intrinsic excitability. In: Duch W, Kacprzyk J, Oja E, Zadrozny S (eds) Proceedings of the international conference on artificial neural networks (ICANN 2005). Springer, Lecture Notes in Computer Science, Heidelberg, pp 65-70
43. D Zhou Y Sun AV Rangan D Cai 2010 Spectrum of Lyapunov exponents of non-smooth dynamical systems of integrate-and-fire type J Comput Neurosci 28 229 245 20012178 10.1007/s10827-009-0201-3 1:CAS:528:DC%2BC3MXks1GqsLY%3D