A biologically plausible embodied model of action discovery

Frontiers in Neurorobotics

Volumn 7, Issue MAR, 2013, Pages

  Bolado Gomez, Rufino ^a   Gurney, Kevin ^a  

^a UNIVERSITY OF SHEFFIELD   (United Kingdom)

Author keywords

Action selection; Basal ganglia; Intrinsic motivation; Operant behavior; Phasic dopamine; Reinforcement learning; Synaptic plasticity

Indexed keywords

DOPAMINE;

ANIMAL EXPERIMENT; ARTICLE; BASAL GANGLION; BIOLOGICALLY PLAUSIBLE EMBODIED MODEL; CONTROLLED STUDY; HYPOTHESIS; IN VIVO STUDY; LEARNING; MODEL; NERVE CELL PLASTICITY; NONHUMAN; RAT; REINFORCEMENT; SENSORY CORTEX; THALAMUS;

EID: 84897945362     PISSN: None     EISSN: 16625218     Source Type: Journal    
DOI: 10.3389/fnbot.2013.00004     Document Type: Article

References (100)

1. Alexander, W. H., and Sporns, O. (2002). An embodied model of learning, plasticity, and reward. Adaptive Behav. 10, 143-159.
2. Baldassarre, G., Mannella, F., Fiore, V. G., Redgrave, P., Gurney, K., and Mirolli, M. (2013). Intrinsically motivated action-outcome learning and goal-based action recall: a system-level bio-constrained computational model. Neural Netw. doi: 10.1016/j.neunet.2012.09.015. [Epub ahead of print].
3. Bar, M. (2007). The proactive brain: using analogies and associations to generate predictions. Trends Cogn. Sci. 11, 280-289.
4. Barto, A. G. (1995). "Adaptive critics and the basal ganglia," in Models of Information Processing in the Basal Ganglia, eds J. C. Houk, J. Davis, and D. Beiser (Cambridge, MA: MIT Press), 215-232.
5. Barto, A. G., Singh, S., and Chantanez, N. (2004). "Intrinsically motivated reinforcement learning," in 18th Annual Conference on Neural Information Processing Systems (NIPS) (Vancouver, BC).
6. Beckstead, R. M., Domesick, V. B., and Nauta, W. J. (1979). Efferent connections of the substantia nigra and ventral tegmental area in the rat. Brain Res. 175, 191-217.
7. Bienenstock, E. L., Cooper, L. N., and Munro, P. W. (1982). Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex. J. Neurosci. 2, 32-48.
8. Bromberg-Martin, E. S., Matsumoto, M., and Hikosaka, O. (2010). Dopamine in motivational control: rewarding, aversive, and alerting. Neuron 68, 815-834. (PMID: 21144997 PMCID: PMC3032992).
9. Brown, J. W., Bullock, D., and Grossberg, S. (2004). How laminar frontal cortex and basal ganglia circuits interact to control planned and reactive saccades. Neural Netw. 17, 471-510.
10. Bubic, A., von Cramon, D. Y., and Schubotz, R. I. (2010). Prediction, cognition and the brain. Front. Hum. Neurosci. 4:25. doi: 10.3389/fnhum.2010.00025
11. Bunzeck, N., and Düzel, E. (2006). Absolute coding of stimulus novelty in the human substantia nigra/VTA. Neuron 51, 369-379.
12. Calabresi, P., Picconi, B., Tozzi, A., and Di Filippo, M. (2007). Dopamine-mediated regulation of corticostriatal synaptic plasticity. Trends Neurosci. 30, 211-219.
13. Cisek, P. (2007). Cortical mechanisms of action selection: the affordance competition hypothesis. Philos. Trans. R. Soc. Lond. B Biol. Sci. 362, 1585-1599.
14. Coizet, V., Comoli, E., Westby, G. W. M., and Redgrave, P. (2003). Phasic activation of substantia nigra and the ventral tegmental area by chemical stimulation of the superior colliculus: an electrophysiological investigation in the rat. Eur. J. Neurosci. 17, 28-40.
15. Comoli, E., Coizet, V., Boyes, J., Bolam, J. P., Canteras, N. S., Quirk, R. H., et al. (2003). A direct projection from superior colliculus to substantia nigra for detecting salient visual events. Nat. Neurosci. 6, 974-980.
16. Cooper, L. N., Intrator, N., Blais, S. B., and Shouval, Z. H. (2004). Theory of Cortical Plasticity. Hackensack, NJ: World Scientific Publishing.
17. Cragg, S., and Rice, M. (2004). DAncing past the DAT at a DA synapse. Trends Neurosci. 27, 270-277.
18. Cyberbotics. (2010a). Webots Reference Manual. Retrieved from http://www.cyberbotics.com/reference.pdf
19. Cyberbotics. (2010b). Webots User Guide. Retrieved from http://www.cyberbotics.com/guide.pdf
20. Dean, P., Redgrave, P., and Westby, G. (1989). Event or emergency - 2 response systems in the mammalian superior colliculus. Trends Neurosci. 12, 137-147.
21. Deniau, J., and Chevalier, G. (1985). Disinhibition as a basic process in the expression of striatal functions II. The striato-nigral influence on thalamocortical cells of the ventromedial thalamic nucleus. Brain Res. 334, 227-233.
22. Dommett, E., Coizet, V., Blaha, C. D., Martindale, J., Lefebvre, V., Walton, N., et al. (2005). How visual stimuli activate dopaminergic neurons at short latency. Science 307, 1476-1479.
23. Doya, K. (1999). What are the computations of the cerebellum, the basal ganglia and the cerebral cortex? Neural Netw. 12, 961-974.
24. Drager, U. C., and Hubel, D. H. (1975). Responses to visual stimulation and relationship between visual, auditory, and somatosensory inputs in mouse superior colliculus. J. Neurophysiol. 38, 690-713.
25. Fino, E., Glowinski, J., and Venance, L. (2005). Bidirectional activity-dependent plasticity at corticostriatal synapses. J. Neurosci. 25, 11279-11287.
26. Fiorillo, C. D., Tobler, P. N., and Schultz, W. (2003). Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299, 1898.
27. Frank, M. J. (2005). Dynamic dopamine modulation in the basal ganglia: a neurocomputational account of cognitive deficits in medicated and nonmedicated parkinsonism. J. Cogn. Neurosci. 17, 51-72.
28. Frank, M. J. (2006). Hold your horses: a dynamic computational role for the subthalamic nucleus in decision making. Neural Netw. 19, 1120-1136.
29. Frank, M. J. (2011). Computational models of motivated action selection in corticostriatal circuits. Curr. Opin. Neurobiol. 21, 381-386.
30. Frank, M. J., Seeberger, L. C., and O'Reilly, R. C. (2004). By carrot or by stick: cognitive reinforcement learning in parkinsonism. Science 306, 1940-1943.
31. Friston, K. (2010). The free-energy principle: a unified brain theory? Nat. Rev. Neurosci. 11, 127-138.
32. Gancarz, A. M., San George, M. A., Ashrafioun, L., and Richards, J. B. (2011). Locomotor activity in a novel environment predicts both responding for a visual stimulus and self-administration of a low dose of methamphetamine in rats. Behav. Processes 86, 295-304.
33. Gauthier, J., Parent, M., Lvesque, M., and Parent, A. (1999). The axonal arborization of single nigrostriatal neurons in rats. Brain Res. 83, 228-232.
34. Gibson, J. J. (1986). The Ecological Approach to Visual Perception. Oxford, UK: Lawrence Erlbaum Associates.
35. Groves, P. M., and Thompson, R. F. (1970). Habituation: a dual-process theory. Psychol. Rev. 77, 419-450.
36. Gurney, K., and Humphries, M. (2012). "Methodological issues in modelling at multiple levels of description," in Computational Systems Neurobiology. (Netherlands: Springer), 259-281.
37. Gurney, K., Prescott, T. J., and Redgrave, P. (2001a). A computational model of action selection in the basal ganglia. I. A new functional anatomy. Biol. Cybern. 84, 401-410.
38. Gurney, K., Prescott, T. J., and Redgrave, P. (2001b). A computational model of action selection in the basal ganglia. II. Analysis and simulation of behaviour. Biol. Cybern. 84, 411-423.
39. Gurney, K. N. (2009). Reverse engineering the vertebrate brain: Methodological principles for a biologically grounded programme of cognitive modelling. Cogn. Comput. 1, 29-41.
40. Gurney, K. N., Humphries, M. D., and Redgrave, P. (2009). Cortico-striatal plasticity for action-outcome learning using spike timing dependent eligibility. BMC Neurosci. 10(Suppl. 1):P135. doi: 10.1186/1471-2202-10-S1-P135
41. Gurney, K. N., Lepora, N., Shah, A., Koene, A., and Redgrave, P. (2013). Action Discovery and Intrinsic Motivation: A Biologically Constrained Formalisation. Berlin: Verlag, Springer.
42. Hazy, T. E., Frank, M. J., and O'Reilly, R. C. (2010). Neural mechanisms of acquired phasic dopamine responses in learning. Neurosci. Biobehav. Rev. 34, 701-720.
43. Houk, J. C., Bastianen, C., Fansler, D., Fishbach, A., Fraser, D., Reber, P. J., et al. (2007). Action selection and refinement in subcortical loops through basal ganglia and cerebellum. Philos. Trans R. Soc. Lond. B Biol. Sci. 362, 1573-1583.
44. Humphries, M. D., and Gurney, K. N. (2002). The role of intra-thalamic and thalamocortical circuits in action selection. Network 13, 131-156.
45. Ikeda, T., and Hikosaka, O. (2003). Reward-dependent gain and bias of visual responses in primate superior colliculus. Neuron 39, 693-700.
46. Jenkinson, N., Nandi, D., Muthusamy, K., Ray, N. J., Gregory, R., Stein, J. F., et al. (2009). Anatomy, physiology, and pathophysiology of the pedunculopontine nucleus. Mov. Disord. 24, 319-328.
47. Joel, D., Niv, Y., and Ruppin, E. (2002). Actor-critic models of the basal ganglia: new anatomical and computational perspectives. Neural Netw. 15, 535-547.
48. Kakade, S., and Dayan, P. (2002). Dopamine: generalization and bonuses. Neural Netw. 15, 549-559.
49. Khamassi, M., Lallée, S., Enel, P., Procyk, E., and Dominey, P. F. (2011). Robot cognitive control with a neurophysiologically inspired reinforcement learning model. Front. Neurorobot. 5:1. doi: 10.3389/fnbot.2011.00001
50. Kumaran, D., and Maguire, E. A. (2007). Which computational mechanisms operate in the hippocampus during novelty detection? Hippocampus 17, 735-748.
51. Lisman, J. E., and Grace, A. A. (2005). The hippocampal-vta loop: controlling the entry of information into long-term memory. Neuron 46, 703-713.
52. Lloyd, D. R., Gancarz, A. M., Ashrafioun, L., Kausch, M. A., and Richards, J. B. (2012). Habituation and the reinforcing effectiveness of visual stimuli. Behav. Processes 91, 184-191.
53. Lorenz, K. (1935). Der kumpan in der umwelt des vogels. J. Ornithol. 83, 137-213; 289-413.
54. Marsland, S. (2009). Using habituation in machine learning. Neurobiol. Learn. Mem. 92, 260-266.
55. Matsuda, W., Furuta, T., Nakamura, K. C., Hioki, H., Fujiyama, F., Arai, R., et al. (2009). Single nigrostriatal dopaminergic neurons form widely spread and highly dense axonal arborizations in the neostriatum. J. Neurosci. 29, 444-453.
56. Matsumoto, M., and Hikosaka, O. (2007). Lateral habenula as a source of negative reward signals in dopamine neurons. Nature 447, 1111-1115.
57. McGrenere, J., and Ho, W. (2000). "Affordances: clarifying and evolving a concept," in Graphics Interface 2000: Proceedings (Montreal, QC), 179. Available online at: http://www.interaction-design.org/references/conferences/proceedings_of_graphics_interface_2000.html
58. McSweeney, F. K., and Murphy, E. S. (2009). Sensitization and habituation regulate reinforcer effectiveness. Neurobiol. Learn. Mem. 92, 189-198.
59. Mink, J. W., and Thach, W. T. (1993). Basal ganglia intrinsic circuits and their role in behavior. Curr. Opin. Neurobiol. 3, 950-957.
60. Mondada, F., Franzi, E., and Guignard, A. (1999). "The development of Khepera," in Experiments with the Mini-Robot Khepera, Proceedings of the First International Khepera Workshop, HNI-Verlagsschriftenreihe (Heinz Nixdorf Institut), 7-14. Available online at: http://infoscience.epfl.ch/record/89709
61. Morris, G., Nevet, A., Arkadir, D., Vaadia, E., and Bergman, H. (2006). Midbrain dopamine neurons encode decisions for future action. Nat. Neurosci. 9, 1057-1063.
62. Moss, J., and Bolam, J. P. (2008). A dopaminergic axon lattice in the striatum and its relationship with cortical and thalamic terminals. J. Neurosci. 28, 11221-11230.
63. Oudeyer, P.-Y., and Kaplan, F. (2007). What is intrinsic motivation? a typology of computational approaches. Front. Neurorobot. 1:6. doi: 10.3389/neuro.12.006.2007
64. Pawlak, V., and Kerr, J. N. (2008). Dopamine receptor activation is required for corticostriatal spike-timing-dependent plasticity. J. Neurosci. 28, 2435.
65. Pear, J. J. (1985). Spatiotemporal patterns of behavior produced by variable-interval schedules of reinforcement. J. Exp. Anal. Behav. 44, 217-231. (PMID: 16812432.)
66. Pfister, J.-P., and Gerstner, W. (2006). Triplets of spikes in a model of spike timing-dependent plasticity. J. Neurosci. 26, 9673-9682.
67. Pleger, B., Blankenburg, F., Ruff, C. C., Driver, J., and Dolan, R. J. (2008). Reward facilitates tactile judgments and modulates hemodynamic responses in human primary somatosensory cortex. J. Neurosci. 28, 8161-8168.
68. Prescott, T. J., Montes Gonzalez, F. M., Gurney, K., Humphries, M. D., and Redgrave, P. (2006). A robot model of the basal ganglia: behavior and intrinsic processing. Neural Netw. 19, 31-61.
69. Pérez-Uribe, A. (2001). "Using a time-delay actor-critic neural architecture with dopamine-like reinforcement signal for learning in autonomous robots," in Emergent Neural Computational Architectures Based on Neuroscience, Vol. 2036, eds S. Wermter, J. Austin, and D. Willshaw (Berlin, Heidelberg: Springer), 522-533.
70. Ranganath, C., and Rainer, G. (2003). Neural mechanisms for detecting and remembering novel events. Nat. Rev. Neurosci. 4, 193-202.
71. Rankin, C. H., Abrams, T., Barry, R. J., Bhatnagar, S., Clayton, D. F., Colombo, J., et al. (2009). Habituation revisited: an updated and revised description of the behavioral characteristics of habituation. Neurobiol. Learn. Mem. 92, 135-138.
72. Redgrave, P., and Gurney, K. (2006). The short-latency dopamine signal: a role in discovering novel actions? Nat. Rev. Neurosci. 7, 967-975.
73. Redgrave, P., Gurney, K., and Reynolds, J. (2008). What is reinforced by phasic dopamine signals? Brain Res. Rev. 58, 322-339.
74. Redgrave, P., Prescott, T. J., and Gurney, K. (1999). The basal ganglia: a vertebrate solution to the selection problem? Neuroscience 89, 1009-1023.
75. Redgrave, P., Vautrelle, N., and Reynolds, J. N. J. (2011). Functional properties of the basal ganglia's re-entrant loop architecture: selection and reinforcement. Neuroscience 198, 138-151.
76. Reynolds, J. N. J., and Wickens, J. R. (2002). Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw. 15, 507-521.
77. Roesch, M. R., Calu, D. J., and Schoenbaum, G. (2007). Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat. Neurosci. 10, 1615-1624.
78. Ryan, R. M., and Deci, E. L. (2000). Intrinsic and extrinsic motivations: classic definitions and new directions. Contemp. Educ. Psychol. 25, 54-67.
79. Samson, R. D., Frank, M. J., and Fellous, J. M. (2010). Computational models of reinforcement learning: the role of dopamine as a reward signal. Cogn. Neurodyn. 4, 91-105.
80. Schmidhuber, J. (2009). "Driven by compression progress: a simple principle explains essential aspects of subjective beauty, novelty, surprise, interestingness, attention, curiosity, creativity, art, science, music, jokes," in Anticipatory Behavior in Adaptive Learning Systems, volume 5499 of Lecture Notes in Computer Science, 48-76. EU Funded Projects. 4th Workshop on Anticipatory Behavior in Adaptive Learning Systems, Munich, Germany, Jun 26-27, 2008.
81. Schultz, W. (2006). Behavioral theories and the neurophysiology of reward. Annu. Rev. Psychol. 57, 87-115.
82. Schultz, W. (2010). Dopamine signals for reward value and risk: basic and recent data. Behav. Brain Funct. 6, 24.
83. Schultz, W., Dayan, P., and Montague, P. R. (1997). A neural substrate of prediction and reward. Science 275, 1593-1599.
84. Shen, W., Flajolet, M., Greengard, P., and Surmeier, D. J. (2008). Dichotomous dopaminergic control of striatal synaptic plasticity. Science 321, 848-851.
85. Song, S., Miller, K. D., and Abbott, L. F. (2000). Competitive hebbian learning through spike-timing-dependent synaptic plasticity. Nat. Neurosci. 3, 919-926.
86. Sporns, O., and Alexander, W. H. (2002). Neuromodulation and plasticity in an autonomous robot. Neural Netw. 15, 761-774.
87. Suri, R. (2002). TD models of reward predictive responses in dopamine neurons. Neural Netw. 15, 523-533.
88. Suri, R. E., and Schultz, W. (1998). Learning of sequential movements by neural network model with dopamine-like reinforcement signal. Exp. Brain Res. 121, 350-354.
89. Suri, R. E., and Schultz, W. (1999). A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task. Neuroscience 91, 871-890.
90. Sutton, R. S., and Barto, A. G. (1998). Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press.
91. Takakusaki, K., Saitoh, K., Harada, H., and Kashiwayanagi, M. (2004). Role of basal ganglia-brainstem pathways in the control of motor behaviors. Neurosci. Res. 50, 137-151.
92. Thivierge, J.-P., Rivest, F., and Monchi, O. (2007). Spiking neurons, dopamine, and plasticity: timing is everything, but concentration also matters. Synapse 61, 375-390.
93. Thompson, A. M., Porr, B., and Woergoetter, F. (2010). Learning and reversal learning in the subcortical limbic system: a computational model. Adaptive Behav. 18, 211-236.
94. Tinbergen, N. (1951). The Study of Instinct. Oxford, UK: Oxford University Press.
95. Tobler, P., Fiorillo, C., and Schultz, W. (2005). Adaptive coding of reward value by dopamine neurons. Science 307, 1642.
96. Weil, R. S., Furl, N., Ruff, C. C., Symmonds, M., Flandin, G., Dolan, R. J., et al. (2010). Rewarding feedback after correct visual discriminations has both general and specific influences on visual cortex. J. Neurophysiol. 104, 1746-1757.
97. West, A. R., and Grace, A. A. (2002). Opposite influences of endogenous dopamine d1 and d2 receptor activation on activity states and electrophysiological properties of striatal neurons: studies combining in vivo intracellular recordings and reverse microdialysis. J. Neurosci. 22, 294-304.
98. Wurtz, R. H., and Albano, J. E. (1980). Visual-motor function of the primate superior colliculus. Annu. Rev. Neurosci. 3, 189-226.
99. Wurtz, R. H., and Goldberg, M. E. (1972). The primate superior colliculus and the shift of visual attention. Invest. Ophthalmol. 11, 441-450.
100. Zink, C. F., Pagnoni, G., Chappelow, J., Martin-Skurski, M., and Berns, G. S. (2006). Human striatal activation reflects degree of stimulus saliency. Neuroimage 29, 977-983.