Neural coding of basic reward terms of animal learning theory, game theory, microeconomics and behavioural ecology

Current Opinion in Neurobiology

Volumn 14, Issue 2, 2004, Pages 139-147

  Schultz, Wolfram ^a  

^a UNIVERSITY OF CAMBRIDGE   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

AMYGDALOID NUCLEUS; ANIMAL BEHAVIOR; BEHAVIORAL SCIENCE; BRAIN FUNCTION; BRAIN REGION; CORPUS STRIATUM; DECISION MAKING; DOPAMINERGIC NERVE CELL; FOOD; GAME; LEARNING; MOTOR PERFORMANCE; NEUROPHYSIOLOGY; NONHUMAN; NUCLEUS ACCUMBENS; ORBITAL CORTEX; PREDICTION; PREFRONTAL CORTEX; PRIORITY JOURNAL; REINFORCEMENT; REVIEW; REWARD; STIMULUS RESPONSE; SUBSTANTIA NIGRA; TASK PERFORMANCE;

ANIMALS; BEHAVIOR, ANIMAL; BRAIN; DECISION MAKING; ECONOMICS; GAME THEORY; HUMANS; LEARNING; NEURONS; REWARD;

EID: 1842684992     PISSN: 09594388     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.conb.2004.03.017     Document Type: Review

References (75)

1. Dickinson A., Balleine B. Motivational control of goal-directed action. Anim Learn Behav. 22:1994;1-18.
2. Flaherty CF: Incentive relativity. Cambridge, UK: Cambridge University Press; 1996.
3. Brosnan S.F., de Waal F.B.M. Monkeys reject unequal pay. Nature. 425:2003;297-299.
4. Wise R.A. Brain reward circuitry: insights from unsensed incentives. Neuron. 36:2002;229-240.
5. Kamin LJ: Selective association and conditioning. In Fundamental Issues in Instrumental Learning. Edited by Mackintosh NJ, Honig WK. Dalhousie, Canada: Dalhousie University Press; 1969: 42-64.
6. Rescorla RA, Wagner AR: A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. In Classical Conditioning II: Current Research and Theory. Edited by Black AH, Prokasy WF. New York, USA: Appleton Century Crofts; 1972: 64-99.
7. Pearce J.M., Hall G. A model for Pavlovian conditioning: variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychol Rev. 87:1980;532-552.
8. Ho M.Y., Mobini S., Chinang T-J., Bradshaw C.M., Szabadi E. Theory and method in the quantitative analysis of 'impulsive choice' behaviour: implications for psychopharmacology. Psychopharmacology. 146:1999;362-372.
9. Bernoulli J. Exposition of a new theory on the measurement of risk. Econometrica. 22:1954;23-36.
10. Kahneman D., Tversky A. Choices, values, and frames. Am Psychol. 4:1984;341-350.
11. Herrnstein R.J. Relative and absolute strength of response as a function of frequency of reinforcement. J Exp Anal Behav. 4:1961;267-272.
12. Lewontin R.C. Evolution and the theory of games. J Theor Biol. 1:1961;382-403.
13. Maynard Smith J: Evolution and the Theory of Games. Cambridge, UK: Cambridge University Press; 1982.
14. Thorpe S.J., Rolls E.T., Maddison S. The orbitofrontal cortex: neuronal activity in the behaving monkey. Exp Brain Res. 49:1983;93-115.
15. Tremblay L., Schultz W. Relative reward preference in primate orbitofrontal cortex. Nature. 398:1999;704-708.
16. Hikosaka K., Watanabe M. Delay activity of orbital and lateral prefrontal neurons of the monkey varying with different rewards. Cereb Cortex. 10:2000;263-271.
17. Nishijo H., Ono T., Nishino H. Single neuron responses in amygdala of alert monkey during complex sensory stimulation with affective significance. J Neurosci. 8:1988;3570-3583.
18. Nakamura K., Mikami A., Kubota K. Activity of single neurons in the monkey amygdala during performance of a visual discrimination task. J Neurophysiol. 67:1992;1447-1463.
19. Bowman E.M., Aigner T.G., Richmond B.J. Neural signals in the monkey ventral striatum related to motivation for juice and cocaine rewards. J Neurophysiol. 75:1996;1061-1073.
20. Hassani O.K., Cromwell H.C., Schultz W. Influence of expectation of different rewards on behavior-related neuronal activity in the striatum. J Neurophysiol. 85:2001;2477-2489.
21. Watanabe M. Reward expectancy in primate prefrontal neurons. Nature. 382:1996;629-632.
22. Aosaki T., Tsubokawa H., Ishida A., Watanabe K., Graybiel A.M., Kimura M. Responses of tonically active neurons in the primate's striatum undergo systematic changes during behavioral sensorimotor conditioning. J Neurosci. 14:1994;3969-3984.
23. Hollerman J.R., Tremblay L., Schultz W. Influence of reward expectation on behavior-related neuronal activity in primate striatum. J Neurophysiol. 80:1998;947-963.
24. Shidara M., Aigner T.G., Richmond B.J. Neuronal signals in the monkey ventral striatum related to progress through a predictable series of trials. J Neurosci. 18:1998;2613-2625.
25. Kawagoe R., Takikawa Y., Hikosaka O. Expectation of reward modulates cognitive signals in the basal ganglia. Nat Neurosci. 1:1998;411-416.
26. Watanabe K., Lauwereyns J., Hikosaka O. Neural correlates of rewarded and unrewarded eye movements in the primate caudate nucleus. J Neurosci. 23:2003;10052-10057.
27. Ravel S., Legallet E., Apicella P. Tonically active neurons in the monkey striatum do not preferentially respond to appetitive stimuli. Exp Brain Res. 128:1999;531-534.
28. Shimo Y., Hikosaka O. Role of tonically active neurons in primate caudate in reward-oriented saccadic eye movement. J Neurosci. 21:2001;7804-7814.
29. Matsumoto K., Suzuki W., Tanaka K. Neuronal correlates of goal-based motor selection in the prefrontal cortex. Science. 301:2003;229-232.
30. Kobayashi S., Lauwereyns J., Koizumi M., Sakagami M., Hikosaka O. Influence of reward expectation on visuospatial processing in macaque lateral prefrontal cortex. J Neurophysiol. 87:2002;1488-1498 After the pioneering study by Watanabe [21] on reward-discriminating prefrontal activity, this work makes the next step by investigating the actual gain in neural information derived from the expectation of reward. Application of basic information theory reveals this activity as a potential input for decision-making mechanisms, although it is presently unclear how close the activity actually is to a neural decision-making mechanism.
31. Tremblay L., Schultz W. Reward-related neuronal activity during go-nogo task performance in primate orbitofrontal cortex. J Neurophysiol. 83:2000;1864-1876.
32. Shidara M., Richmond B.J. Anterior cingulate: single neuron signals related to degree of reward expectancy. Science. 296:2002;1709-1711 After neural activity related to reward expectation has been observed in several brain structures, this study demonstrates a long-term activity that anticipates the reward and progressively increases over several trials through intervening stimuli and movements, indicating a very stable reward representation.
33. Liu Z., Richmond B.J. Response differences in monkey TE and perirhinal cortex: stimulus association related to reward schedules. J Neurophysiol. 83:2000;1677-1692.
34. Ikeda T., Hikosaka O. Reward-dependent gain and bias of visual responses in primate superior colliculus. Neuron. 39:2003;693-700.
35. Sato M., Hikosaka O. Role of primate substantia nigra pars reticulata in reward-oriented saccadic eye movement. J Neurosci. 22:2002;2363-2373.
36. Waelti P., Dickinson A., Schultz W. Dopamine responses comply with basic assumptions of formal learning theory. Nature. 412:2001;43-48.
37. Kawagoe R., Takikawa Y., Hikosaka O. Reward-predicting activity of dopamine and caudate neurons - a possible mechanism of motivational control of saccadic eye movement. J Neurophysiol. 91:2004;1013-1024.
38. Watanabe M., Hikosaka K., Sakagami M., Shirakawa S.I. Coding and monitoring of behavioral context in the primate prefrontal cortex. J Neurosci. 22:2002;2391-2400 This ingenious study reports that neurons in dorsolateral prefrontal cortex show differential delay activations in unrewarded trials that depend on which reward is given in the other rewarded trials within the same trial block, indicating what the animal is actually missing in the current trial.
39. Cromwell H.C., Schultz W. Effects of expectations for different reward magnitudes on neuronal activity in primate striatum. J Neurophysiol. 89:2003;2823-2838.
40. Leon M.I., Shadlen M.N. Effect of expected reward magnitude on the responses of neurons in the dorsolateral prefrontal cortex of the macaque. Neuron. 24:1999;415-425.
41. Roesch M.R., Olson C.R. Impact of expected reward on neuronal activity in prefrontal cortex, frontal and supplementary eye fields and premotor cortex. J Neurophysiol. 90:2003;1766-1789 The authors present a huge effort to separate the potential coding of reward value from the changes of movement-related activity induced by rewards.
42. Wallis J.D., Miller E.K. Neuronal activity in primate dorsolateral and orbital prefrontal cortex during performance of a reward preference task. Eur J Neurosci. 18:2003;2069-2081.
43. Roesch MR, Olson CR: Distinguishing reward representation from motivational modulation in primate frontal cortex [abstract]. Society for Neuroscience;2003: 724.2.
44. Platt M.L., Glimcher P.W. Neural correlates of decision variables in parietal cortex. Nature. 400:1999;233-238.
45. Corneil BD, Musallam S, Andersen RA: Representation of reward expectancy in the medial bank of the intraparietal sulcus: implications for neural prosthetics [abstract]. Society for Neuroscience; 2003: 607.8.
46. McCoy AN, Platt ML: Reward signals in posterior cingulate cortex [abstract]. Society for Neuroscience; 2003: 385.6.
47. Satoh T., Nakai S., Sato T., Kimura M. Correlated coding of motivation and outcome of decision by dopamine neurons. J Neurosci. 23:2003;9913-9923.
48. Bayer HM, Glimcher PW: Midbrain dopamine neurons encode a quantitative reward prediction error signal [abstract]. Society for Neuroscience; 2003: 557.9.
49. Lauwereyns J., Watanabe K., Coe B., Hikosaka O. A neural correlate of response bias in monkey caudate nucleus. Nature. 418:2002;413-417.
50. Fiorillo C.D., Tobler P.N., Schultz W. Discrete coding of reward probability and uncertainty by dopamine neurons. Science. 299:2003;1898-1902.
51. Morris G, Arkadir D, Nevet A, Bergman H: Basal ganglia neuronal teachers: cooperating different messages [abstract]. Society for Neuroscience; 2003: 704.12.
52. Samejima K, Ueda Y, Kenji D, Minoru K: Activity of striate projection neurons encodes action - selective reward expectations [abstract]. Society for Neuroscience; 2003: 704.9.
53. Dorris MC, Glimcher PW: Lateral intraparietal area activity during manipulations of the expected value and utility of choices [abstract]. Society for Neuroscience; 2003: 767.1.
54. Sugrue LP, Rorie AE, Corrado GS, Newsome WT: Matching behavior is effectively modeled by a simple linear integrator of rewards [abstract]. Society for Neuroscience; 2003: 385.1.
55. Basso M.A., Wurtz R.H. Modulation of neuronal activity by target uncertainty. Nature. 389:1997;66-69.
56. Dorris M.C., Munoz D.P. Saccadic probability influences motor preparation signals and time to saccadic initiation. J Neurosci. 18:1998;7015-7026.
57. Basso M.A., Wurtz R.H. Neuronal activity in substantia nigra pars reticulata during target selection. J Neurosci. 22:2002;1883-1894.
58. Leon M.I., Shadlen M.N. Representation of time by neurons in the posterior parietal cortex of the macaque. Neuron. 38:2003;317-327 The authors present the first modern study on the coding of time by the brain, which is one of the great enigmas of neuroscience. Some preliminary work on this topic was done in the 1970s by other investigators. The work is also interesting from the temporal aspect of decision-making (when to make a behavioural response).
59. Janssen P, Shadlen MN: Representation of the hazard function of elapsed time by neurons in macaque area LIP [abstract]. Society for Neuroscience; 2003: 767.2.
60. Rolls E.T., Yaxley S., Sienkiewicz Z.J. Gustatory responses of single neurons in the caudolateral orbitofrontal cortex of the macaque monkey. J Neurophysiol. 64:1990;1055-1066.
61. Rolls E.T., Critchley H.D., Browning A.S., Hernadi I., Lenard L. Responses to the sensory properties of fat of neurons in the primate orbitofrontal cortex. J Neurosci. 19:1999;1532-1540.
62. Schultz W. Predictive reward signal of dopamine neurons. J Neurophysiol. 80:1998;1-27.
63. Schultz W., Dayan P., Montague R.R. A neural substrate of prediction and reward. Science. 275:1997;1593-1599.
64. Tobler P.N., Dickinson A., Schultz W. Coding of predicted reward omission by dopamine neurons in a conditioned inhibition paradigm. J Neurosci. 23:2003;10402-10410.
65. Apicella P., Legallet E., Trouche E. Responses of tonically discharging neurons in the monkey striatum to primary rewards delivered during different behavioral states. Exp Brain Res. 116:1997;456-466.
66. Watanabe M. The appropriateness of behavioral responses coded in post-trial activity of primate prefrontal units. Neurosci Lett. 101:1989;113-117.
67. Ito S., Stuphorn V., Brown J.W., Schall J.D. Performance monitoring by the anterior cingulate cortex during saccade countermanding. Science. 302:2003;120-122 This study follows up on an earlier issue of error coding in the frontal cortex [66]. The authors examine reward prediction errors in an interesting task in which monkeys are sometimes asked to withhold an already prepared saccadic eye movement with different levels of difficulty.
68. Bendiksby MS, Platt ML: Motivation focuses attention and enhances neuronal selectivity in parietal cortex [abstract]. Society for Neuroscience; 2003: 385.5.
69. Camerer C.F., Ho T.H. Experience-weighted attraction learning in normal form games. Econometrica. 67:1999;827-874.
70. Barraclough DJ, Conroy ML, Lee D: Conjunctive coding of decision variables in prefrontal cortex during a competitive game [abstract]. Society for Neuroscience; 2003: 722.21.
71. Newsome W.T., Britten K.H., Movshon J.A. Neuronal correlates of a perceptual decision. Nature. 341:1989;52-54.
72. Kim J.N., Shadlen M.N. Neural correlates of a decision in the dorsolateral prefrontal cortex of the macaque. Nat Neurosci. 2:1999;176-185.
73. Gold J.I., Shadlen M.N. Banburismus and the brain: decoding the relationship between sensory stimuli, decisions, and reward. Neuron. 36:2002;299-308 An interesting charming and mathematically illustrated description of how the development of code-deciphering mechanisms during WW2 by Alan Turing and colleagues could lead to a general description of decision mechanisms that resemble neural mechanisms involved in decision-making with continuously incoming information.
74. Schall J.D., Stuphorn V., Brown J.W. Monitoring and control of action by the frontal lobes. Neuron. 36:2002;309-322.
75. Romo R., Hernandez A., Zainos A., Lemus L., Brody C.D. Neuronal correlates of decision-making in secondary somatosensory cortex. Nat Neurosci. 5:2002;1217-1225.