Beetles, boxes and brain cells: Neural mechanisms underlying valuation and learning

Current Opinion in Neurobiology

Volumn 15, Issue 6, 2005, Pages 721-729

  Salzman, C Daniel ^a   Belova, Marina A ^a   Paton, Joseph J ^a  

^a Howard Hughes Medical Institute   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DOPAMINE; GLUTAMIC ACID;

BASAL GANGLION; BEETLE; BEHAVIOR; BRAIN CELL; BRAIN NERVE CELL; CELL ACTIVITY; CINGULATE GYRUS; EYE MOVEMENT; FLAVOR; FRONTAL CORTEX; LEARNING; MESENCEPHALON; MOTIVATION; NERVE PROJECTION; NONHUMAN; ORBITAL CORTEX; PREDICTION; PREMOTOR CORTEX; PRIORITY JOURNAL; REVIEW; REWARD; SENSORY STIMULATION; SYNAPSE; TASK PERFORMANCE; TASTE; VISUAL STIMULATION;

ANIMALS; BRAIN; CEREBRAL CORTEX; COGNITION; CUES; DOPAMINE; HAPLORHINI; LEARNING; MESENCEPHALON; MOTIVATION; NEURONS; REWARD;

EID: 27944492732     PISSN: 09594388     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.conb.2005.10.016     Document Type: Review

References (59)

1. W. Wittgenstein G. Anscombe R. Rhees Philosophical Investigations 2nd edn 1958 Blackwell Section 293
2. R.A. Wise Dopamine, learning and motivation Nat Rev Neurosci 5 2004 483 494
3. P.R. Montague, S.E. Hyman, and J.D. Cohen Computational roles for dopamine in behavioural control Nature 431 2004 760 767
4. M.A. Ungless Dopamine: the salient issue Trends Neurosci 27 2004 702 706
5. J.E. Lisman, and A.A. Grace The hippocampal-VTA loop: controlling the entry of information into long-term memory Neuron 46 2005 703 713
6. L.P. Sugrue, G.S. Corrado, and W.T. Newsome Choosing the greater of two goods: neural currencies for valuation and decision making Nat Rev Neurosci 6 2005 363 375 The authors present an up-to-date review about the neural signals underlying valuation and decision making with a focus on several recent studies in LIP and PFC.
7. P.W. Glimcher, and A. Rustichini Neuroeconomics: the consilience of brain and decision Science 306 2004 447 452
8. P.C. Holland, and M. Gallagher Amygdala-frontal interactions and reward expectancy Curr Opin Neurobiol 14 2004 148 155
9. W. Schultz Neural coding of basic reward terms of animal learning theory, game theory, microeconomics and behavioural ecology Curr Opin Neurobiol 14 2004 139 147
10. W. Schultz, L. Tremblay, and J.R. Hollerman Changes in behavior-related neuronal activity in the striatum during learning Trends Neurosci 26 2003 321 328
11. W. Schultz, P. Dayan, and P.R. Montague A neural substrate of prediction and reward Science 275 1997 1593 1599
12. P.R. Montague, and G.S. Berns Neural economics and the biological substrates of valuation Neuron 36 2002 265 284
13. R. Sutton, and A. Barto Reinforcement Learning 1998 MIT Press
14. P. Dayan, and L.F. Abbott Classical conditioning and reinforcement learning Theoretical Neuroscience 2001 MIT Press 331-358
15. D.J. Barraclough, M.L. Conroy, and D. Lee Prefrontal cortex and decision making in a mixed-strategy game Nat Neurosci 7 2004 404 410
16. M.C. Dorris, and P.W. Glimcher Activity in posterior parietal cortex is correlated with the relative subjective desirability of action Neuron 44 2004 365 378 This study examines neural activity recorded from LIP while monkeys play a strategic computer game. The authors argue that LIP activity reflects the relative subjective desirability of actions in the game.
17. L.P. Sugrue, G.S. Corrado, and W.T. Newsome Matching behavior and the representation of value in the parietal cortex Science 304 2004 1782 1787 This study describes the activity of LIP neurons while monkeys chose to make eye movements to targets that supplied rewards at different rates. They compared the neural representation of the value of choices both to monkeys' performance and to predictions derived from a computational model. LIP neural activity reflects the relative value of competing actions, matching predictions from their model and the monkeys' behavior. Although the authors do not argue that LIP computes the value of choices, their data demonstrates that LIP has access to information about value.
18. J.H. Maunsell Neuronal representations of cognitive state: reward or attention? Trends Cogn Sci 8 2004 261 265 This is an interesting review that emphasizes the challenges facing cognitive neuroscientists when trying to determine exactly what cognitive process neural signals represent.
19. M.R. Roesch, and C.R. Olson Neuronal activity related to reward value and motivation in primate frontal cortex Science 304 2004 307 310 This study shows that neural activity reflecting different anticipated reward magnitudes might actually represent either the value of the reward or the degree of motivation that the monkeys have. They used a task that modulated reward magnitude and penalty time to vary both reward value and motivation independently. Although there was some overlap in response properties, in general, orbitofrontal cortex preferentially represented reward value, and premotor cortex activity reflected motivation.
20. A.N. McCoy, and M.L. Platt Risk-sensitive neurons in macaque posterior cingulate cortex Nat Neurosci 8 2005 1220 1227 This work strives to disambiguate neural signals related to objective and subjective value by recording neural activity while monkeys performed a visual gambling task. Monkeys chose between making an eye movement to a target with a certain reward amount, or to a target with a variable reward amount but the average reward of which was the same as the other target. Monkeys chose the uncertain reward target more often, and neural activity in the posterior cingulate cortex reflected this subjective evaluation of reward.
21. C.R. Olson, S.Y. Musil, and M.E. Goldberg Single neurons in posterior cingulate cortex of behaving macaque: eye movement signals J Neurophysiol 76 1996 3285 3300
22. A.N. McCoy, J.C. Crowley, G. Haghighian, H.L. Dean, and M.L. Platt Saccade reward signals in posterior cingulate cortex Neuron 40 2003 1031 1040
23. C.V. Buhusi, and W.H. Meck What makes us tick? Functional and neural mechanisms of interval timing Nat Rev Neurosci 6 2005 755 765
24. S. Tsujimoto, and T. Sawaguchi Neuronal activity representing temporal prediction of reward in the primate prefrontal cortex J Neurophysiol 93 2005 3687 3692
25. M.R. Roesch, and C.R. Olson Neuronal activity in primate orbitofrontal cortex reflects the value of time J Neurophysiol 94 2005 2457 2471
26. P. Janssen, and M.N. Shadlen A representation of the hazard rate of elapsed time in macaque area LIP Nat Neurosci 8 2005 234 241 Inspired by the recognition that humans and animals routinely perform tasks that require interval timing, this study demonstrates that activity in LIP flexibly reflects monkeys' knowledge of time intervals during a task requiring eye movements.
27. M.I. Leon, and M.N. Shadlen Representation of time by neurons in the posterior parietal cortex of the macaque Neuron 38 2003 317 327
28. M.L. Platt, and P.W. Glimcher Neural correlates of decision variables in parietal cortex Nature 400 1999 233 238
29. J.W. Bisley, and M.E. Goldberg Neuronal activity in the lateral intraparietal area and spatial attention Science 299 2003 81 86
30. L. Itti, and C. Koch Computational modelling of visual attention Nat Rev Neurosci 2 2001 194 203
31. J. Gottlieb Parietal mechanisms of target representation Curr Opin Neurobiol 12 2002 134 140
32. M.N. Shadlen, and W.T. Newsome Neural basis of a perceptual decision in the parietal cortex (area LIP) of the rhesus monkey J Neurophysiol 86 2001 1916 1936
33. S. Wirth, M. Yanike, L.M. Frank, A.C. Smith, E.N. Brown, and W.A. Suzuki Single neurons in the monkey hippocampus and learning of new associations Science 300 2003 1578 1581
34. P.J. Brasted, and S.P. Wise Comparison of learning-related neuronal activity in the dorsal premotor cortex and striatum Eur J Neurosci 19 2004 721 740 The authors compare the time course of changes of neural activity in the dorsal premotor cortex and striatum (primarily the putamen) during performance of a visuomotor associative learning task. On average, neurons in both areas exhibited changes at approximately the same time with respect to behavioral learning. Recordings in the two brain areas were conducted in separate sessions, but the authors alternated between brain areas every few days to ensure that any differences in the two brain areas could not be attributed to long-term changes in the monkeys' performance.
35. A. Pasupathy, and E.K. Miller Different time courses of learning-related activity in the prefrontal cortex and striatum Nature 433 2005 873 876 This technically challenging study employed simultaneous recording of single neurons from dorsolateral prefrontal cortex and the caudate in monkeys performing a visuomotor associative learning task involving frequent stimulus-action contingency reversals. During the learning of such reversals, directional signals in the striatum strengthened more rapidly than similar signals in prefrontal cortex, which were actually more closely correlated with the gradual improvement in monkeys' behavior. The authors suggest that rapid changes in activity in the striatum might gradually train neurons in the prefrontal cortex, rather than the other way around.
36. L. Tremblay, J.R. Hollerman, and W. Schultz Modifications of reward expectation-related neuronal activity during learning in primate striatum J Neurophysiol 80 1998 964 977
37. W.F. Asaad, G. Rainer, and E.K. Miller Neural activity in the primate prefrontal cortex during associative learning Neuron 21 1998 1399 1407
38. J. Lauwereyns, K. Watanabe, B. Coe, and O. Hikosaka A neural correlate of response bias in monkey caudate nucleus Nature 418 2002 413 417
39. L. Chen, and S. Wise Neuronal activity in the supplementary eye field during acquisition of conditional oculomotor associations J Neurophysiol 73 1995 1101 1121
40. G. Schoenbaum, A.A. Chiba, and M. Gallagher Neural encoding in orbitofrontal cortex and basolateral amygdala during olfactory discrimination learning J Neurosci 19 1999 1876 1884
41. F. Hadj-Bouziane, and D. Boussaoud Neuronal activity in the monkey striatum during conditional visuomotor learning Exp Brain Res 153 2003 190 196
42. G.E. Alexander, M.R. DeLong, and P.L. Strick Parallel organization of functionally segregated circuits linking basal ganglia and cortex Annu Rev Neurosci 9 1986 357 381
43. K. Watanabe, and O. Hikosaka Immediate changes in anticipatory activity of caudate neurons associated with reversal of position-reward contingency J Neurophysiol 94 2005 1879 1887 This interesting study demonstrates that neural activity in the caudate can change within one trial after a reversal in position-reward contingencies. The rapid change in activity reflects the fact that the neurons, like the monkeys, might learn about one trial type from their experience with another trial type, which the authors call learning a 'reversal set rule'.
44. W. Schultz, and A. Dickinson Neuronal coding of prediction errors Annu Rev Neurosci 23 2000 473 500
45. H.M. Bayer, and P.W. Glimcher Midbrain dopamine neurons encode a quantitative reward prediction error signal Neuron 47 2005 129 141
46. P.N. Tobler, C.D. Fiorillo, and W. Schultz Adaptive coding of reward value by dopamine neurons Science 307 2005 1642 1645 This study demonstrates that the reward prediction error signals carried by midbrain DA neurons are adaptive to behavioral context, thereby providing a more efficient representation of information across a range of experimental conditions. By manipulating reward magnitudes, probabilities and relative reward amounts, the authors show that the reward prediction error signal encoded by DA neurons is flexible and adaptive.
47. ••], this study indicates that DA neurons adapt their firing rate depending upon behavioral context.
48. G. Morris, D. Arkadir, A. Nevet, E. Vaadia, and H. Bergman Coincident but distinct messages of midbrain dopamine and striatal tonically active neurons Neuron 43 2004 133 143 This study compared the timing of responses of TANs in the striatum and DA neurons in the midbrain during performance of a conditioning task. DA neurons encoded a well described reward prediction error signal, and TANs respond to all motivationally significant events with a pause in activity that was coincident with the DA error signal, and synchronized across TANs. The authors speculate that this pause in TAN activity could act as a temporal gate for DA signaling in the striatum. The experiments were conducted so that the dynamics of operant responses did not vary across trial types, removing a major potential confound.
49. M. Benoit-Marand, M. Jaber, and F. Gonon Release and elimination of dopamine in vivo in mice lacking the dopamine transporter: functional consequences Eur J Neurosci 12 2000 2985 2992
50. P. Greengard The neurobiology of slow synaptic transmission Science 294 2001 1024 1030
51. F.M. Zhou, C. Wilson, and J.A. Dani Muscarinic and nicotinic cholinergic mechanisms in the mesostriatal dopamine systems Neuroscientist 9 2003 23 36
52. N. Chuhma, H. Zhang, J. Masson, X. Zhuang, D. Sulzer, R. Hen, and S. Rayport Dopamine neurons mediate a fast excitatory signal via their glutamatergic synapses J Neurosci 24 2004 972 981
53. N.S. Bamford, H. Zhang, Y. Schmitz, N.P. Wu, C. Cepeda, M.S. Levine, C. Schmauss, S.S. Zakharenko, L. Zablow, and D. Sulzer Heterosynaptic dopamine neurotransmission selects sets of corticostriatal terminals Neuron 42 2004 653 663 A technical tour de force that combined two-photon imaging of cortico-striatal terminal destaining, electrochemical recordings of striatal DA release, pharmacology, intracellular electrophysiology, and electrical stimulation to examine how DA acts on corticostriatal terminals. Results from this study suggest that DA acts through D2 receptors to filter out the least active among motor area cortico-striatal synapses. The authors suggest that this mechanism could aid in establishing appropriate behaviors during reinforcement learning.
54. Z. Liu, B.J. Richmond, E.A. Murray, R.C. Saunders, S. Steenrod, B.K. Stubblefield, D.M. Montague, and E.I. Ginns DNA targeting of rhinal cortex D2 receptor protein reversibly blocks learning of cues that predict reward Proc Natl Acad Sci USA 101 2004 12336 12341 This innovative study demonstrates the importance of dopaminergic inputs to rhinal cortex on a monkey's ability to use visual stimuli that cue proximity to reward. Dopamine D2 receptor antisense was targeted to the rhinal cortex. This intervention blocked monkeys' ability to learn the significance of visual cues that indicated how many trials must be completed before reward receipt.
55. M. Akil, and D.A. Lewis The dopaminergic innervation of monkey entorhinal cortex Cereb Cortex 3 1993 533 550
56. S.K. Goldsmith, and J.N. Joyce Dopamine D2 receptors are organized in bands in normal human temporal cortex Neuroscience 74 1996 435 451
57. E.K. Richfield, A.B. Young, and J.B. Penney Comparative distributions of dopamine D-1 and D-2 receptors in the cerebral cortex of rats, cats, and monkeys J Comp Neurol 286 1989 409 426
58. B. Berger, S. Trottier, C. Verney, P. Gaspar, and C. Alvarez Regional and laminar distribution of the dopamine and serotonin innervation in the macaque cerebral cortex: a radioautographic study J Comp Neurol 273 1988 99 119
59. L.W. Zhou, S.P. Zhang, and B. Weiss Intrastriatal administration of an oligodeoxynucleotide antisense to the D2 dopamine receptor mRNA inhibits D2 dopamine receptor-mediated behavior and D2 dopamine receptors in normal mice and in mice lesioned with 6-hydroxydopamine Neurochem Int 29 1996 583 595