The role of the basal ganglia in discovering novel actions

Intrinsically Motivated Learning in Natural and Artificial Systems

Volumn 9783642323751, Issue , 2013, Pages 129-150

  Redgrave, Peter ^a   Gurney, Kevin ^a   Stafford, Tom ^a   Thirkettle, Martin ^a   Lewis, Jen ^a  

^a UNIVERSITY OF SHEFFIELD   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

AMINES; CONVEYING; ELECTROPHYSIOLOGY; ENCODING (SYMBOLS); REINFORCEMENT; SIGNAL ENCODING;

ACTION LEARNING; ACTION SELECTION; BASAL GANGLIA; BASIC MECHANISM; DOPAMINE; ENCODINGS; REINFORCEMENT LEARNINGS; RELATIVE SENSITIVITY; RELATIVE STRENGTH; REWARD-PREDICTION ERROR; ECONOMIC VALUES; TEMPORAL CONSTRAINTS;

REINFORCEMENT LEARNING; REINFORCEMENT;

EID: 84929046152     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1007/978-3-642-32375-1_6     Document Type: Chapter

References (106)

1. Alexander, G., DeLong, M., Strick, P.: Parallel organization of functionally segregated circuits linking basal ganglia and cortex. Annu. Rev. Neurosci. 9, 357-381 (1986)
2. Arbuthnott, G.W., Wickens, J.: Space, time and dopamine. Trends Neurosci. 30(2), 62-69 (2007)
3. BarGad, I., Morris, G., Bergman, H.: Information processing, dimensionality reduction and reinforcement learning in the basal ganglia. Prog. Neurobiol. 71(6), 439-473 (2003)
4. Bayer, H.M., Glimcher, P.W.: Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47(1), 129-141 (2005)
5. Berridge, K.C.: The debate over dopamine's role in reward: The case for incentive salience. Psychopharmacology 191(3), 391-431 (2007)
6. Bickford, M., Hall, W.: Collateral projections of predorsal bundle cells of the superior colliculus in the rat. J. Comp. Neurol. 283, 86-106 (1989)
7. Black, J., Belluzzi, J.D., Stein, L.: Reinforcement delay of one second severely impairs acquisition of brain self-stimulation. Brain Res. 359(1-2), 113-119 (1985)
8. Boehnke, S.E., Munoz, D.P.: On the importance of the transient visual response in the superior colliculus. Curr. Opin. Neurobiol. 18(6), 544-551 (2008)
9. Chevalier, G., Deniau, J.: Disinhibition as a basic process in the expression of striatal functions. Trends Neurosci. 13(7), 277-280 (1990)
10. Coizet, V., Dommett, E.J., Redgrave, P., Overton, P.G.: Nociceptive responses of midbrain dopaminergic neurones are modulated by the superior colliculus in the rat. Neuroscience 139(4), 1479-1493 (2006)
11. Coizet, V., Graham, J.H., Moss, J., Bolam, J.P., Savasta, M., McHaffie, J.G., Redgrave, P., Overton, P.G.: Short-latency visual input to the subthalamic nucleus is provided by the midbrain superior colliculus. J. Neurosci. 29(17), 5701-5709 (2009)
12. Coizet, V., Overton, P.G., Redgrave, P.: Collateralization of the tectonigral projection with other major output pathways of superior colliculus in the rat. J. Comp. Neurol. 500(6), 1034-1049 (2007)
13. Comoli, E., Coizet, V., Boyes, J., Bolam, J.P., Canteras, N.S., Quirk, R.H., Overton, P.G., Redgrave, P.: A direct projection from superior colliculus to substantia nigra for detecting salient visual events. Nat. Neurosci. 6(9), 974-980 (2003)
14. Corbit, L., Balleine, B.: The role of prelimbic cortex in instrumental conditioning. Behav. Brain Res. 146(1-2), 145-157 (2003)
15. Crutcher, M.D., DeLong, M.R.: Single cell studies of the primate putamen. II. Relations to direction of movement and pattern of muscular activity. Exp. Brain Res. 53(2), 244-258 (1984)
16. Dayan, P., Balleine, B.: Reward, motivation, and reinforcement learning. Neuron 36(2), 285-298 (2002)
17. Dean, P., Redgrave, P., Westby, G.: Event or emergency? two response systems in the mammalian superior colliculus. Trends Neurosci. 12(4), 137-147 (1989)
18. Dickinson, A.: The 28th bartlett memorial lecture causal learning: An associative analysis. Quart. J. Exp. Psych. B Comp. Phys. P 54(1), 3-25 (2001)
19. Ding, L., Hikosaka, O.: Comparison of reward modulation in the frontal eye field and caudate of the macaque. J. Neurosci. 26(25), 6695-6703 (2006)
20. Dommett, E., Coizet, V., Blaha, C.D., Martindale, J., Lefebvre, V., Walton, N., Mayhew, J.E., Overton, P.G., Redgrave, P.: How visual stimuli activate dopaminergic neurons at short latency. Science 307 (5714), 1476-1479 (2005)
21. Elsner, B., Hommel, B.: Contiguity and contingency in action-effect learning. Psychol. Res. 68(2-3), 138-154 (2004)
22. Everitt, B.J., Robbins, T.W.: Neural systems of reinforcement for drug addiction: From actions to habits to compulsion. Nat. Neurosci. 8(11), 1481-1489 (2005)
23. Fiorillo, C., Tobler, P., Schultz, W.: Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299 (5614), 1898-1902 (2003)
24. Floresco, S., West, A., Ash, B., Moore, H., Grace, A.: Afferent modulation of dopamine neuron firing differentially regulates tonic and phasic dopamine transmission. Nat. Neurosci. 6(9), 968-973 (2003)
25. Freeman, A.S., Meltzer, L.T., Bunney, B.S.: Firing properties of substantia nigra dopaminergic neurons in freely moving rats. Life Sci. 36(20), 1983-1994 (1985)
26. Gerfen, C., Wilson, C.: The basal ganglia. In: Swanson, L., Bjorklund, A., Hokfelt, T. (eds.) Handbook of Chemical Neuroanatomy, vol 12: Integrated Systems of the CNS, Part III., pp. 371-468. Elsevier, Amsterdam (1996)
27. Grace, A.A.: The tonic/phasic model of dopamine system regulation: Its relevance for understanding how stimulant abuse can alter basal ganglia function. Drug Alcohol Depend. 37, 111-129 (1995)
28. Grantyn, R.: Gaze control through superior colliculus: Structure and function. In: Buttner-Ennever, J. (ed.) Neuroanatomy of the Oculomotor System, pp. 273-333. Elsevier, Amsterdam (1988)
29. Greengard, P., Allen, P.B., Nairn, A.C.: Beyond the dopamine receptor: The darpp-32/protein phosphatase-1 cascade. Neuron 23(3), 435-447 (1999)
30. Grillner, S., Helligren, J., Ménard, A., Saitoh, K., Wikström, M.A.: Mechanisms for selection of basic motor programs - roles for the striatum and pallidum. Trends Neurosci. 28(7), 364-370 (2005)
31. Guarraci, F., Kapp, B.: An electrophysiological characterization of ventral tegmental area dopamin-ergic neurons during differential pavlovian fear conditioning in the awake rabbit. Behav. Brain Res. 99(2), 169-179 (1999)
32. Gurney, K., Lepora, N., Shah, A., Koene, A., Redgrave, P.: Action discovery and intrinsic motivation: A biologically constrained formalisation. In: Baldassarre, G., Mirolli, M. (eds.) Intrinsically Motivated Learning in Natural and Artificial Systems, pp. 151-181. Springer, Berlin (2012)
33. Gurney, K., Prescott, T., Redgrave, P.: A computational model of action selection in the basal ganglia. I. A new functional anatomy. Biol. Cybern. 84(6), 401-410 (2001a)
34. Gurney, K., Prescott, T., Redgrave, P.: A computational model of action selection in the basal ganglia. II. Analysis and simulation of behaviour. Biol. Cybern. 84(6), 411-423 (2001b)
35. Heien, M., Khan, A.S., Ariansen, J.L., Cheer, J.F., Phillips, P.E.M., Wassum, K.M., Wightman, R.M.: Real-time measurement of dopamine fluctuations after cocaine in the brain of behaving rats. Proc. Natl. Acad. Sci. U. S. A. 102(29), 10023-10028 (2005)
36. Hikosaka, O.: GABAergic output of the basal ganglia. Prog. Brain Res. 160, 209-226 (2007)
37. Hikosaka, O., Sakamoto, M., Usui, S.: Functional properties of monkey caudate neurons III. Activities related to expectation of target and reward. J. Neurophysiol. 61(4), 814-832 (1989)
38. Hikosaka, O., Wurtz, R.: Visual and oculomotor function of monkey substantia nigra pars reticulata. I. Relation of visual and auditory responses to saccades. J. Neurophysiol. 49(5), 1230-1253 (1983)
39. Horn, G., Hill, R.M.: Effect of removing the neocortex on the response to repeated sensory stimulation of neurones in the mid-brain. Nature 211, 754-755 (1966)
40. Horvitz, J.: Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events. Neuroscience 96(4), 651-656 (2000)
41. Horvitz, J., Stewart, T., Jacobs, B.: Burst activity of ventral tegmental dopamine neurons is elicited by sensory stimuli in the awake cat. Brain Res. 759(2), 251-258 (1997)
42. Houk, J.C.: Agents of the mind. Biol. Cybern. 92(6), 427-437 (2005)
43. Humphries, M.D., Stewart, R.D., Gurney, K.N.: A physiologically plausible model of action selection and oscillatory activity in the basal ganglia. J. Neurosci. 26(50), 12921-12942 (2006)
44. Ikeda, T., Hikosaka, O.: Reward-dependent gain and bias of visual responses in primate superior colliculus. Neuron 39(4), 693-700 (2003)
45. Izhikevich, E.M.: Solving the Distal Reward Problem through linkage of STDP and dopamine signaling. Cereb. Cortex 17(10), 2443-2452 (2007)
46. Jay, M., Sparks, D.: Sensorimotor integration in the primate superior colliculus. I. Motor convergence. J. Neurophysiol. 57(1), 22-34 (1987)
47. Klop, E.M., Mouton, L.J., Hulsebosch, R., Boers, J., Holstege, G.: In cat four times as many lamina I neurons project to the parabrachial nuclei and twice as many to the periaqueductal gray as to the thalamus. Neuroscience 134(1), 189-197 (2005)
48. Kobayashi, S., Lauwereyns, J., Koizumi, M., Sakagami, M., Hikosaka, O.: Influence of reward expectation on visuospatial processing in macaque lateral prefrontal cortex. J. Neurophysiol. 87(3), 1488-1498 (2002)
49. Kobayashi, S., Nomoto, K., Watanabe, M., Hikosaka, O., Schultz, W., Sakagami, M.: Influences of rewarding and aversive outcomes on activity in macaque lateral prefrontal cortex. Neuron 51(6), 861-870 (2006)
50. Lacey, C.J., Bolam, J.P., Magill, P.J.: Novel and distinct operational principles of intralaminar thalamic neurons and their striatal projections. J. Neurosci. 27(16), 4374-4384 (2007)
51. Lévesque, M., Charara, A., Gagnon, S., Parent, A., Deschênes, M.: Corticostriatal projections from layer V cells in rat are collaterals of long-range corticofugal axons. Brain Res. 709(2), 311-315 (1996)
52. Lindvall, O., Björklund, A.: The organization of the ascending catcholamine neuron systems in the rat brain as revealed by the glyoxylic acid fluoresence method. Acta Physiol. Scand. Suppl. 412, 1-48 (1974)
53. Ljungberg, T., Apicella, P., Schultz, W.: Responses of monkey dopamine neurons during learning of behavioural reactions. J. Neurophysiol. 67(1), 145-163 (1992)
54. Matsumoto, M., Hikosaka, O.: Representation of negative motivational value in the primate lateral habenula. Nat. Neurosci. 12(1), 77-84 (2009)
55. McHaffie, J.G., Stanford, T.R., Stein, B.E., Coizet, V., Redgrave, P.: Subcortical loops through the basal ganglia. Trends Neurosci. 28(8), 401-407 (2005)
56. Mink, J.: The basal ganglia: Focused selection and inhibition of competing motor programs. Prog. Neurobiol. 50(4), 381-425 (1996)
57. Montague, P.R., Dayan, P., Sejnowski, T.J.: A framework for mesencephalic dopamine systems based on predictive hebbian learning. J. Neurosci. 16(5), 1936-1947 (1996)
58. Montague, P.R., Hyman, S.E., Cohen, J.D.: Computational roles for dopamine in behavioural control. Nature 431 (7010), 760-767 (2004)
59. Morris, G., Arkadir, D., Nevet, A., Vaadia, E., Bergman, H.: Coincident but distinct messages of midbrain dopamine and striatal tonically active neurons. Neuron 43(1), 133-143 (2004)
60. Nakahara, H., Itoh, H., Kawagoe, R., Takikawa, Y., Hikosaka, O.: Dopamine neurons can represent context-dependent prediction error. Neuron 41(2), 269-280 (2004)
61. O'Doherty, J., Dayan, P., Friston, K., Critchley, H., Dolan, R.: Temporal difference models and reward-related learning in the human brain. Neuron 38(2), 329-337 (2003)
62. Omelchenko, N., Sesack, S.R.: Glutamate synaptic inputs to ventral tegmental area neurons in the rat derive primarily from subcortical sources. Neuroscience 146(3), 1259-1274 (2007)
63. Overton, P.G., Coizet, V., Dommett, E., Redgrave, P.: The parabrachial nucleus is a source of short latency nociceptive input to midbrain dopaminergic neurones in rat. Program No. 301.5, Neuroscience 2005 Abstracts. Society for Neuroscience, Washington. Online (2005)
64. Padoa-Schioppa, C., Assad, J.A.: Neurons in the orbitofrontal cortex encode economic value. Nature 441 (7090), 223-226 (2006)
65. Pan, W.X., Schmidt, R., Wickens, J.R, Hyland, B.I.: Dopamine cells respond to predicted events during classical conditioning: Evidence for eligibility traces in the reward-learning network. J. Neurosci. 25(26), 6235-6242 (2005)
66. Pleger, B., Blankenburg, F., Ruff, C.C., Driver, J., Dolan, R.J.: Reward facilitates tactile judgments and modulates hemodynamic responses in human primary somatosensory cortex. J. Neurosci. 28(33), 8161-8168 (2008)
67. Potts, G.F., Martin, L.E., Burton, P., Montague, P.R.: When things are better or worse than expected: The medial frontal cortex and the allocation of processing resources. J. Cogn. Neurosci. 18(7), 1112-1119 (2006)
68. Prescott, T.J., Montes González, F.M., Gurney, K., Humphries, M.D., Redgrave, P.: A robot model of the basal ganglia: Behavior and intrinsic processing. Neural Netw. 19(1), 31-61 (2006)
69. Prescott, T.J., Redgrave, P., Gurney, K.: Layered control architectures in robots and vertebrates. Adap. Behav. 7(1), 99-127 (1999)
70. Redgrave, P.: Basal ganglia. Scholarpedia 2(6), 1825 (2007)
71. Redgrave, P., Gurney, K.: The short-latency dopamine signal: A role in discovering novel actions? Nat. Rev. Neurosci. 7(12), 967-975 (2006)
72. Redgrave, P., Gurney, K., Reynolds, J.: What is reinforced by phasic dopamine signals? Brain Res. Rev. 58(2), 322-339 (2008)
73. Redgrave, P., Prescott, T., Gurney, K.: The basal ganglia: A vertebrate solution to the selection problem? Neuroscience 89(4), 1009-1023 (1999a)
74. Redgrave, P., Prescott, T., Gurney, K.: Is the short latency dopamine response too short to signal reward error? Trends Neurosci. 22(4), 146-151 (1999b)
75. Reiner, A., Jiao, Y., Del Mar, N., Laverghetta, A., Lei, W.: Differential morphology of pyramidal tract-type and intratelencephalically projecting-type corticostriatal neurons and their intrastri-atal terminals in rats. J. Comp. Neurol. 457, 420-440 (2003)
76. Reynolds, J.N., Wickens, J.R.: Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw. 15(4-6), 507-521 (2002)
77. Robbins, T., Sahakian, B.: Behavioral and neurochemical determinants of drug-induced stereotypy. In: Rose, F. (ed.) Metabolic Disorders of the Nervous System, pp. 244-291. Pitman Pr., London (1981)
78. Roitman, M., Stuber, G., Phillips, P., Wightman, R., Carelli, R.: Dopamine operates as a subsecond modulator of food seeking. J. Neurosci. 24(6), 1265-1271 (2004)
79. Rolls, E.T.: The orbitofrontal cortex and reward. Cereb. Cortex 10(3), 284-294 (2000)
80. Salamone, J., Correa, M.: Motivational views of reinforcement: Implications for understanding the behavioral functions of nucleus accumbens dopamine. Behav. Brain Res. 137(1-2), 3-25 (2002)
81. Satoh, T., Nakai, S., Sato, T., Kimura, M.: Correlated coding of motivation and outcome of decision by dopamine neurons. J. Neurosci. 23(30), 9913-9923 (2003)
82. Schultz, W.: Predictive reward signal of dopamine neurons. J. Neurophysiol. 80(1), 1-27 (1998)
83. Schultz, W.: Multiple reward signals in the brain. Nat. Rev. Neurosci. 1(3), 199-207 (2000)
84. Schultz, W.: Behavioral theories and the neurophysiology of reward. Annu. Rev. Psychol. 57, 87-115 (2006)
85. Schultz, W.: Multiple dopamine functions at different time courses. Annu. Rev. Neurosci. 30, 259-288 (2007)
86. Schultz, W., Dayan, P., Montague, P.: A neural substrate of prediction and reward. Science 275 (5306), 1593-1599 (1997)
87. Schulz, J., Redgrave, P., Clements, K., Reynolds, J.: Short latency activation of striatal spiny neurons via subcortical visual pathways. Program No. 180.2, 2008 Neuroscience Meeting Planner. Society for Neuroscience, Washington. Online (2008)
88. Singh, S., Barto, A., Chentanez, N.: Intrinsically motivated reinforcement learning. In: Saul, K., Weiss, Y., Bottou, L. (eds.) Advances in Neural Information Processing Systems 17, pp. 1281-1288 (2005)
89. Smith, Y., Raju, D.V., Pare, J.F., Sidibe, M.: The thalamostriatal system: A highly specific network of the basal ganglia circuitry. Trends Neurosci. 27(9), 520-527 (2004)
90. Snaith, S., Holland, O.: An investigation of two mediation strategies suitable for behavioural control in animals and animats. In: Meyer, J.-A., Wilson, S. (eds.) From Animals to Animats: Proceedings of the First International Conference on the Simulation of Adaptive Behaviour, pp. 255-262. MIT, Cambridge (1990)
91. Sparks, D.: Translation of sensory signals into commands for control of saccadic eye movements: Role of the primate superior colliculus. Physiol. Rev. 66(1), 118-171 (1986)
92. Sprague, J.M., Marchiafava, P.L., Rixxolatti, G.: Unit responses to visual stimuli in the superior colliculus of the unanesthetized, mid-pontine cat. Arch. Ital. Biol. 106(3), 169-193 (1968)
93. Stafford, T., Walton, T., Hetherington, L., Thirkettle, M., Gurney, K., Redgrave, P.: A novel behavioural task for researching intrinsic motivation. In: Baldassarre, G., Mirolli, M. (eds.) Intrinsically Motivated Learning in Natural and Artificial Systems, pp. 395-410. Springer, Berlin (2012)
94. Stein, B., Meredith, M.: The Merging of the Senses. MIT, Cambridge (1993)
95. Stoerig, P.: Blindsight, conscious vision, and the role of primary visual cortex. Prog. Brain Res. 155B, 217-234 (2006)
96. Sutton, R., Barto, A.: Reinforcement Learning: An Introduction. MIT, Cambridge (1998)
97. Thorndike, E.: Animal Intelligence. Macmillan, New York (1911)
98. Thorpe, S., Fabre-Thorpe, M.: Seeking categories in the brain. Science 291 (5502), 260-263 (2001)
99. Ungless, M.: Dopamine: The salient issue. Trends Neurosci. 27(12), 702-706 (2004)
100. Ungless, M., Magill, P., Bolam, J.: Uniform inhibition of dopamine neurons in the ventral tegmental area by aversive stimuli. Science 303 (5666), 2040-2042 (2004)
101. Venton, B.J., Wightman, R.M.: Pharmacologically induced, subsecond dopamine transients in the caudate-putamen of the anesthetized rat. Synapse 61(1), 37-39 (2007)
102. Watanabe, M., Hikosaka, K., Sakagami, M., Shirakawa, S.: Coding and monitoring of motivational context in the primate prefrontal cortex. J. Neurosci. 22(6), 2391-2400 (2002)
103. Wickens, J.: A theory of the striatum. Pergamon, Oxford (1993)
104. Wickens, J., Reynolds, J., Hyland, B.: Neural mechanisms of reward-related motor learning. Curr. Opin. Neurobiol. 13(6), 685-690 (2003)
105. Wise, R.: Dopamine, learning and motivation. Nat. Rev. Neurosci. 5(6), 483-494 (2004)
106. Wurtz, R., Albano, J.: Visual-motor function of the primate superior colliculus. Annu. Rev. Neurosci. 3, 189-226 (1980)