Democratic population decisions result in robust policy-gradient learning: A parametric study with GPU simulations

PLoS ONE

Volumn 6, Issue 5, 2011, Pages

  Richmond, Paul ^a   Buesing, Lars ^b   Giugliano, Michele ^c   Vasilaki, Eleni ^a  

^a UNIVERSITY OF SHEFFIELD   (United Kingdom)

^b UNIVERSITY COLLEGE LONDON   (United Kingdom)

^c UNIVERSITY OF ANTWERP   (Belgium)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; ARTIFICIAL NEURAL NETWORK; BIOINFORMATICS; COMPUTER; COMPUTER PROGRAM; COMPUTER SIMULATION; DECISION MAKING; GRAPHICS PROCESSING UNIT; MATHEMATICAL COMPUTING; MATHEMATICAL MODEL; NETWORK LEARNING; NEUROSCIENCE; PARAMETRIC TEST; REINFORCEMENT; SIGNAL NOISE RATIO; ALGORITHM; COMPUTER GRAPHICS;

ALGORITHMS; COMPUTER GRAPHICS;

EID: 79955761534     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0018539     Document Type: Article

References (83)

1. Gummaraju J, Rosenblum M, (2005) Stream programming on general-purpose processors. In: MICRO 38: Proceedings of the 38th annual IEEE ACM International Symposium on Microarchitecture. IEEE Computer Society Washington, DC, USA pp. 343-354.
2. Buck I, Foley T, Horn D, Sugerman J, Fatahalian K, et al. (2004) Brook for gpus: stream computing on graphics hardware. In: SIGGRAPH ′04: ACM SIGGRAPH 2004 Papers. ACM New York, NY, USA pp. 777-786 doi:http://doi.acm.org/10.1145/1186562.1015800.
3. Amari SI, (1975) Homogeneous nets of neuron-like elements. Biological Cybernetics 17: 211-220.
4. Amari S, (1977) Dynamics of pattern formation in lateral-inhibition type neural fields. Biol Cybern 27: 77-87.
5. Hamaguchi K, Okada M, Yamana M, Aihara K, (2005) Correlated firing in a feedforward network with mexican-hat-type connectivity. Neural Comput 17: 2034-2059.
6. Coultrip R, Granger R, Lynch G, (1992) A cortical model of winner-take-all competition via lateral inhibition. Neural Networks 5: 47-54.
7. Spiridon M, Gerstner W, (2001) Effect of lateral connections on the accuracy of the population code for a network of spiking neurons. Network: Computation in Neural Systems 12: 409-421257-272.
8. Piekniewski F, (2010) Persistent activation blobs in spiking neural networks with mexican hat connectivity. In: Rutkowski L, Scherer R, Tadeusiewicz R, Zadeh L, Zurada J, editors. Artifical Intelligence and Soft Computing, Springer Berlin/Heidelberg, volume 6114 of Lecture Notes in Computer Science pp. 64-71.
9. Georgopoulos AP, Schwartz A, Kettner RE, (1986) Neuronal population coding of movement direction. Science 233: 1416-1419.
10. Pfister JP, Toyoizumi T, Barber D, Gerstner W, (2006) Optimal spike-timing dependent plasticity for precise action potential firing in supervised learning. Neural Computation 18: 1309-1339.
11. Florian RV, (2007) Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural Computation 19: 1468-1502.
12. Vasilaki E, Frémaux N, Urbanczik R, Senn W, Gerstner W, (2009) Spike-based reinforcement learning in continuous state and action space: When policy gradient methods fail. PLoS Comput Biol 5: e1000586.
13. Williams R, (1992) Simple statistical gradient-following methods for connectionist reinforcement learning. Machine Learning 8: 229-256.
14. Baxter J, Bartlett P, Weaver L, (2001) Experiments with infinite-horizon, policy- gradient estimation. Journal of Artificial Intelligence Research 15: 351-381.
15. Wang X, (2006) A microcircuit model of prefrontal functions: ying and yang of reverberatory neurodynamics in cognition. The frontal lobes development, function, and pathology.
16. Vasilaki E, Fusi S, Wang XJ, Senn W, (2009) Learning exible sensori-motor mappings in a complex network. Biol Cybern 100: 147-158.
17. Beierholm U, Dayan P, (2010) Pavlovian-Instrumental Interaction in "Observing Behavior". PLoS Computational Biology 6.
18. Dayan P, (2009) Goal-directed control and its antipodes. Neural Networks 22: 213-219.
19. Talmi D, Dayan P, Kiebel S, Frith C, Dolan R, (2009) How humans integrate the prospects of pain and reward during choice. Journal of Neuroscience 29: 14617.
20. Dayan P, Daw N, (2008) Decision theory, reinforcement learning, and the brain. Cognitive, Affective, & Behavioral Neuroscience 8: 429.
21. Dayan P, (2008) The role of value systems in decision making. Better than conscious pp. 51-70.
22. Sutton RS, Barto AG, (1981) Towards a modern theory of adaptive networks: expectation and prediction. Psychol Review 88: 135-171.
23. Barto A, Sutton R, Anderson C, (1983) Neuronlike adaptive elements that can solve difficult learning and control problems. IEEE transactions on systems, man, and cybernetics 13: 835-846.
24. Sutton R, Barto A, (1990) Time-derivative models of pavlovian reinforcement. In: Gabriel M, Moore J, editors. Learning and Computational Neuroscience: Foundations of Adaptive Networks Cambridge MIT-Press pp. 497-537.
25. Morris R, Garrard P, Rawlins J, O'Keefe J, (1982) Place navigation impaired in rats with hippocampal lesions. Nature 297: 681-683.
26. Foster D, Morris R, Dayan P, (2000) Models of hippocampally dependent navigation using the temporal difference learning rule. Hippocampus 10: 1-16.
27. Arleo A, Gerstner W, (2000) Spatial cognition and neuro-mimetic navigation: a model of hippocampal place cell activity. Biological Cybernetics 83: 287-299.
28. Stroesslin T, Sheynikhovich D, Chavarriaga R, Gerstner W, (2005) Robust self-localisation and navigation based on hippocampal place cells. Neural Networks 18: 1125-1140.
29. Sheynikhovich D, Chavarriaga R, Strösslin T, Gerstner W, (2005) Spatial representation and navigation in a bio-inspired robot. Biomimetic Neural Learning for Intelligent Robots: Intelligent Systems, Cognitive Robotics, and Neuroscience pp. 245-264.
30. Legenstein R, Wilbert N, Wiskott L, (2010) Reinforcement learning on slow features of highdimensional input streams. PLoS Comput Biol 6: e1000894.
31. Seung HS, (2003) Learning in spiking neural networks by reinforcement of stochastic synaptic transmission. Neuron 40: 1063-1073.
32. Xie X, Seung H, (2004) Learning in neural networks by reinforcement of irregular spiking. Physical Review E 69: 41909.
33. Fiete I, Seung H, (2006) Gradient Learning in Spiking Neural Networks by Dynamic Perturbation of Conductances. Physical Review Letters 97: 48104.
34. Baras D, Meir R, (2007) Reinforcement learning, spike-time-dependent plasticity, and the bcm rule. Neural Computation 19: 2245-2279.
35. Friedrich J, Urbanczik R, Senn W, (2010) Learning spike-based population codes by reward and population feedback. Neural computation 22: 1698-1717.
36. Watkins C, (1989) Learning from delayed rewards. Cambridge PhD-thesis, Cambridge University.
37. Sutton R, Barto A, (1998) Reinforcement learning. MIT Press, Cambridge.
38. Potjans W, Morrison A, Diesmann M, (2009) A spiking neural network model of an actor-critic learning agent. Neural Comp 21: 301-339.
39. Di Castro D, Volkinshtein S, Meir R, (2009) Temporal difference based actor critic learning- convergence and neural implementation. NIPS 22: 385-392.
40. Suri R, Schultz W, (2001) Temporal difference model reproduces anticipatory neural activity. Neural Computation 13: 841-862.
41. Gerstner W, Kempter R, van Hemmen JL, Wagner H, (1996) A neuronal learning rule for submillisecond temporal coding. Nature 383: 76-78.
42. Kempter R, Gerstner W, van Hemmen JL, (1999) Hebbian learning and spiking neurons. Phys Rev E 59: 4498-4514.
43. Abbott LF, Nelson SB, (2000) Synaptic plasticity: Taming the beast. Nature Neuroscience 3: 1178-1183.
44. Izhikevich E, (2007) Solving the distal reward problem through linkage of stdp and dopamine signaling. Cerebral Cortex 17: 2443-2452.
45. Farries MA, Fairhall AL, (2007) Reinforcement Learning With Modulated Spike Timing Dependent Synaptic Plasticity. J Neurophysiol 98: 3648-3665.
46. Legenstein R, Pecevski D, Maass W, (2008) A learning theory for reward-modulated spiketiming-dependent plasticity with application to biofeedback. PLoS Computational Biology 4 (10): e1000180.
47. O'Keefe J, Dostrovsky J, (1971) The hippocampus as a spatial map. preliminary evidence from unit activity in the freely-moving rat. Brain Res 34: 171-175.
48. Jensen O, Lisman JE, (2000) Position Reconstruction From an Ensemble of Hippocampal Place Cells: Contribution of Theta Phase Coding. J Neurophysiol 83: 2602-2609.
49. Stein RB, (1965) A theoretical analysis of neuronal variability. Biophys J 5: 173-194.
50. Gerstner W, Kistler WK, (2002) Spiking Neuron Models. Cambridge UK Cambridge University Press.
51. Barto A, (1985) Learning by statistical cooperation of self-interested neuron-like neuron elements. Human Neurobiology 4: 229-256.
52. Van Rossum M, (2001) The transient precision of integrate and fire neurons: Effect of background activity and noise. Journal of Computational Neuroscience 10: 303-311.
53. Van Rossum M, Turrigiano G, Nelson S, (2002) Fast propagation of firing rates through layered networks of noisy neurons. Journal of Neuroscience 22: 1956.
54. Van Rossum M, Renart A, (2004) Computation with populations codes in layered networks of integrate-and-fire neurons. Neurocomputing 58: 265-270.
55. Sanders J, Kandrot E, (2010) CUDA by Example: An Introduction to General-Purpose GPU Programming. Addison-Wesley.
56. NVIDIA Corporation (2008) NVIDIA CUDA Programming Guide.
57. Ly DL, Paprotski V, Yen D, (2008) Neural networks on gpus: Restricted boltzmann machines. Technical report, University of Toronto.
58. Martnez-Zarzuela M, Daz Pernas F, Dez Higuera J, Rodrguez M, (2007) Fuzzy art neural network parallel computing on the gpu. In: Sandoval F, Prieto A, Cabestany J, Graa M, editors. Computational and Ambient Intelligence, Springer Berlin, Heidelberg, volume 4507 of Lecture Notes in Computer Science pp. 463-470.
59. Bernhard F, Keriven R, (2006) Spiking neurons on GPUs. International Conference on Computational Science (4) pp. 236-243.
60. Goodman D, Brette R, (2008) Brian: a simulator for spiking neural networks in Python. Frontiers in neuroinformatics 2.
61. Goodman D, Brette R, (2009) The Brian simulator. Frontiers in neuroscience 3: 192.
62. Cyrille R, M GDF, Jonathan P, Romain B, (2010) Automatic fitting of spiking neuron models to electrophysiological recordings. Frontiers in Neuroinformatics 4: 12.
63. Goodman D, (2010) Code generation: A strategy for neural network simulators. Neuroinformatics 8: 183-196.
64. Nageswaran J, Dutt N, Krichmar J, Nicolau A, Veidenbaum A, (2009) A configurable simulation environment bfor the efficient simulation of large-scale spiking neural networks on graphics processors. Neural Networks 22: 791-800.
65. Bhuiyan M, Pallipuram V, Smith M, (2010) Acceleration of spiking neural networks in emerging multi-core and gpu architectures. Parallel Distributed Processing, Workshops and Phd Forum (IPDPSW), 2010 IEEE International Symposium on pp. 1-8 doi:10.1109/IPDPSW.2010.5470899.
66. Markram H, (2006) The blue brain project. Nature Reviews Neuroscience 7: 153-160.
67. Ananthanarayanan R, Esser SK, Simon HD, Modha DS, (2009) The cat is out of the bag: cortical simulations with 109 neurons, 1013 synapses. In: SC ′09: Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis. 1-12 New York, NY, USA ACM doi:http://doi.acm.org/10.1145/1654059.1654124.
68. Furber S, Temple S, (2008) Neural systems engineering. In: Fulcher J, Jain L, editors. Computational Intelligence: A Compendium, Springer Berlin Heidelberg, volume 115 of Studies in Computational Intelligence pp. 763-796.
69. Renaud S, Tomas J, Bornat Y, Daouzli A, Saighi S, (2007) Neuromimetic ICs with analog cores: an alternative for simulating spiking neural networks. In: Circuits and Systems, 2007. ISCAS 2007. IEEE International Symposium on. IEEE pp. 3355-3358.
70. Bruederle D, Bill J, Kaplan B, Kremkow J, Meier K, et al. (2010) Simulator-Like Exploration of Cortical Network Architectures with a Mixed-Signal VLSI System. Proceedings of the 2010 IEEE International Symposium on Circuits and Systems (ISCAS 10) pp. 2784-8787.
71. Bill J, Schuch K, Bruederle D, Schemmel J, Maass W, et al. (2010) Compensating inhomogeneities of neuromorphic vlsi devices via short-term synaptic plasticity. Frontiers in Computational Neuroscience 4: 12.
72. Davison A, Yger P, Kremkow J, Perrinet L, Muller E, (2007) PyNN: towards a universal neural simulator API in Python. BMC Neuroscience 8: P2.
73. Davison A, Bruederle D, Eppler J, Kremkow J, Muller E, et al. (2008) PyNN: a common interface for neuronal network simulators. Frontiers in neuroinformatics 2.
74. Eppler J, Helias M, Muller E, Diesmann M, Gewaltig M, (2008) PyNEST: a convenient interface to the NEST simulator. Frontiers in neuroinformatics 2.
75. Bruederle D, Mueller E, Davison A, Muller E, Schemmel J, et al. (2009) Establishing a novel modeling tool: a python-based interface for a neuromorphic hardware system. Frontiers in neuroinformatics 3.
76. Davison A, Hines M, Muller E, (2009) Trends in programming languages for neuroscience simulations. Frontiers in neuroscience 3: 374.
77. Davison A, Muller E, Bruederle D, Kremkow J, (2010) A common language for neuronal networks in software and hardware. The Neuromorphic Engineer.
78. van Meel J, Arnold A, Frenkel D, Portegies Zwart S, Belleman R, (2008) Harvesting graphics power for MD simulations. Molecular Simulation 34: 259-266.
79. Bell N, Garland M, (2008) Efficient sparse matrix-vector multiplication on CUDA. NVIDIA Technical Report NVR-2008-004, NVIDIA Corporation.
80. Sengupta S, Harris M, Zhang Y, Owens JD, (2007) Scan primitives for gpu computing. In: GH ′07: Proceedings of the 22nd ACM SIGGRAPH/EUROGRAPHICS symposium on Graphics hardware. Aire-la-Ville, Switzerland, Switzerland: Eurographics Association, 97-106.
81. Hines M, Morse T, Migliore M, Carnevale N, Shepherd G, (2004) ModelDB: a database to support computational neuroscience. Journal of Computational Neuroscience 17: 7-11.
82. Richmond P, Romano D, (2008) A high performance framework for agent based pedestrian dynamics on gpu hardware. Proceedings of EUROSIS ESM 2008.
83. Richmond P, Walker D, Coakley S, Romano D, (2010) High performance cellular level agent-based simulation with FLAME for the GPU. Briefings in bioinformatics 11: 334.