From biophysics to cognition: reward-dependent adaptive choice behavior

Current Opinion in Neurobiology

Volumn 18, Issue 2, 2008, Pages 209-216

  Soltani, Alireza ^a   Wang, Xiao Jing ^b  

^a CALIFORNIA INSTITUTE OF TECHNOLOGY   (United States)

^b YALE UNIVERSITY SCHOOL OF MEDICINE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ADAPTIVE BEHAVIOR; BIOPHYSICS; COGNITION; CORPUS STRIATUM; DECISION MAKING; DOPAMINERGIC NERVE CELL; HUMAN; LONG TERM DEPRESSION; LONG TERM POTENTIATION; NEUROBIOLOGY; NONHUMAN; ORBITAL CORTEX; PREFRONTAL CORTEX; PRIORITY JOURNAL; REVIEW; REWARD; SIGNAL TRANSDUCTION;

ADAPTATION, PSYCHOLOGICAL; BIOPHYSICS; CHOICE BEHAVIOR; COGNITION; DECISION MAKING; HUMANS; MODELS, NEUROLOGICAL; REWARD; TASK PERFORMANCE AND ANALYSIS;

EID: 52049086876     PISSN: 09594388     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.conb.2008.07.003     Document Type: Review

References (58)

1. Sugrue L.P., Corrado G.C., and Newsome W.T. Choosing the greater of two goods: neural currencies for valuation and decision making. Nat Rev Neurosci 6 (2005) 363-375
2. Gold J.I., and Shadlen M.N. The neural basis of decision making. Annu Rev Neurosci 30 (2007) 535-574. This is an authoritative and comprehensive overview of electrophysiological findings in different types of decision-making tasks, as well as theoretical approaches based on the signal detection theory and sequential analysis.
3. Heekeren H.R., Marrett S., and Ungerleider L.G. The neural systems that mediate human perceptual decision making. Nat Rev Neurosci 9 (2008) 467-479
4. Loewenstein G., Rick S., and Cohen J.D. Neuroeconomics. Annu Rev Psychol 59 (2008) 647-672
5. In: Doya K., Ishii S., Pouget A., and Rao R.P.N. (Eds). Bayesian Brain: Probabilistic Approaches to Neural Coding (2007), MIT Press
6. Sutton R.S., and Barto A.G. Reinforcement Learning: An Introduction (1998), MIT Press
7. Schultz W., Dayan P., and Montague P.R. A neural substrate of prediction and reward. Science 275 (1997) 1593-1599
8. Roesch M.R., Calu D.J., and Schoenbaum G. Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat Neurosci 10 (2007) 1615-1624. This is an important paper showing that, while rats perform a time discounting task, dopamine neurons in the ventral tegmental area encode the reward prediction error, as well as the size and delay of reward.
9. Bayer H.M., and Glimcher P.W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47 (2005) 129-141
10. Matsumoto M., and Hikosaka O. Lateral habenula as a source of negative reward signals in dopamine neurons. Nature 447 (2007) 1111-1115
11. Nakahara H., Itoh H., Kawagoe R., Takikawa Y., and Hikosaka O. Dopamine neurons can represent context-dependent prediction error. Neuron 41 (2004) 269-280. This is the first study to show that dopamine neurons can signal reward prediction error which depends on the context. The context in this task is the number of trials since the last reward, which indicates the growing probability that a reward will appear in the upcoming trial.
12. Tobler P.N., Fiorillo C.D., and Schultz W. Adaptive coding of reward value by dopamine neurons. Science 307 (2005) 1642-1645
13. Matsumoto M., Matsumoto K., Abe H., and Tanaka K. Medial prefrontal cell activity signaling prediction errors of action values. Nat Neurosci 10 (2007) 647-656. This work reveals that both positive and negative reward prediction error signals are coded by an increase in firing activity, in two distinct subpopulations of neurons in the medial prefrontal cortex. The response of these neurons changes as monkeys learn the action-outcome association, in a way that validates the hypothesis that these neurons signal positive and negative prediction errors.
14. Rushworth M.F.S., and Behrens T.E.J. Choice, uncertainty and value in prefrontal and cingulate cortex. Nat Neurosci 11 (2008) 389-397
15. Barraclough D.J., Conroy M.L., and Lee D. Prefrontal cortex and decision making in a mixed-strategy game. Nat Neurosci 7 (2004) 404-410
16. Seo H., and Lee D. Temporal filtering of reward signals in the dorsal anterior cingulate cortex during a mixed-strategy game. J Neurosci 27 (2007) 8366-8377. The authors record from ACC neurons in behaving monkeys during a competitive game and show that spiking activity in some neurons is correlated with rewards in previous trials while activity in others encodes a signal related to reward prediction errors.
17. Samejima K., Ueda K., Doya K., and Kimura M. Representation of action-specific reward values in the striatum. Science 310 (2005) 1337-1340. This paper provides the clearest evidence so far that in a free-choice task, a significant fraction of recorded striatal neurons encode possible action values but are not selective for chosen responses.
18. Lau B., and Glimcher P.W. Action and outcome encoding in the primate caudate nucleus. J Neurosci 27 (2007) 14502-14514
19. Lau B., and Glimcher P.W. Value representations in the primate striatum during matching behavior. Neuron 58 (2008) 451-463
20. Montague P.R., and Berns G.S. Neural economics and the biological substrates of valuation. Neuron 36 (2002) 265-284
21. Padoa-Schioppa C., and Assad J.A. Neurons in the orbitofrontal cortex encode economic value. Nature 441 (2006) 223-226
22. Sugrue L.P., Corrado G.C., and Newsome W.T. Matching behavior and representation of value in parietal cortex. Science 304 (2004) 1782-1787
23. Lau B., and Glimcher P.W. Dynamic response-by-response models of matching behavior in rhesus monkeys. J Exp Anal Behav 84 (2005) 555-579
24. Kennerley S.W., Walton M.E., Behrens T.E.J., Buckley M.J., and Rushworth M.F.S. Optimal decision making and the anterior cingulate cortex. Nat Neurosci 9 (2006) 940-947
25. Reynolds J.N., and Wickens J.R. Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw 15 (2002) 507-521
26. Calabresi P., Picconi B., Tozzi A., and di Filippo M. Dopamine-mediated regulation of corticostriatal synaptic plasticity. Trends Neurosci 30 (2007) 211-219
27. Reynolds J.N., Hyland B.I., and Wickens J.R. A cellular mechanism of reward-related learning. Nature 413 (2001) 67-70
28. Fino E., Glowinski J., and Venance L. Bidirectional activity-dependent plasticity at corticostriatal synapses. J Neurosci 25 (2005) 11279-11287
29. 5 dopamine receptors is required for this plasticity.
30. Wang Z., Kai L., Day M., Ronesi J., Yin H.H., Ding J., Tkatch T., Lovinger D.M., and Surmeier D.J. Dopaminergic control of corticostriatal long-term synaptic depression in medium spiny neurons is mediated by cholinergic interneurons. Neuron 50 (2006) 443-452
31. Matsuda Y., Marzo A., and Otani S. The presence of background dopamine signal converts long-term synaptic depression to potentiation in rat prefrontal cortex. J Neurosci 26 (2006) 4803-4810
32. Huang Y.Y., Simpson E., Kellendonk C., and Kandel E.R. Genetic evidence for the bidirectional modulation of synaptic plasticity in the prefrontal cortex by D1 receptors. Proc Natl Acad Sci U S A 101 (2004) 3236-3241
33. Mockett B.G., Guevremont D., Williams J.M., and Abraham W.C. Dopamine d1/d5 receptor activation reverses nmda receptor-dependent long-term depression in rat hippocampus. J Neurosci 27 (2007) 2918-2926
34. Grace A.A. Phasic versus tonic dopamine release and the modulation of dopamine system responsivity: a hypothesis for the etiology of schizophrenia. Neuroscience 41 (1991) 1-24
35. Schultz W. Multiple dopamine functions at different time courses. Annu Rev Neurosci 30 (2007) 259-288. This paper focuses on different time courses of dopamine signaling and proposes a role for these signals in guiding the behavior at different time scales.
36. Seung H.S. Learning in spiking neural networks by reinforcement of stochastic synaptic transmission. Neuron 40 (2003) 1063-1073
37. Soltani A., and Wang X.J. A biophysically-based neural model of matching law behavior: melioration by stochastic synapses. J Neurosci 26 (2006) 3731-3744. The authors show that in a neural circuit model for a foraging task, plastic synapses with a biophysically plausible learning rule can estimate reward values in terms of return. The information stored in synapses is reflected in the spiking activity of decision neurons, and reward-dependent plasticity generates adaptive choice behavior in accordance with the matching law.
38. Soltani A., Lee D., and Wang X.J. Neural mechanism for stochastic behavior during a competitive game. Neural Netw 19 (2006) 1075-1090
39. Izhikevich E.M. Solving the distal reward problem through linkage of STDP and dopamine signaling. Cereb Cortex 17 (2007) 2443-2452. The author shows that a spiking network model endowed with a STDP learning rule, which is modulated by dopamine signals, can reproduce observations in classical and instrumental conditioning.
40. Montague P.R., McClure S.M., Baldwin P.R., Phillips P.E., Budygin E.A., Stuber G.D., Kilpatrick M.R., and Wightman R.M. Dynamic gain control of dopamine delivery in freely moving animals. J Neurosci 24 (2004) 1754-1759
41. Bogacz R., McClure S.M., Li J., Cohen J.D., and Montague P.R. Short-term memory traces for action bias in human reinforcement learning. Brain Res 1153 (2007) 111-121
42. Fusi S., Asaad W.F., Miller E.K., and Wang X.J. A neural circuit model of flexible sensorimotor mapping: learning and forgetting on multiple timescales. Neuron 54 (2007) 319-333
43. Wang X.J. Probabilistic decision making by slow reverberation in cortical circuits. Neuron 36 (2002) 955-968
44. Fusi S. Hebbian spike-driven synaptic plasticity for learning patterns of mean firing rates. Biol Cybern 87 (2002) 459-470
45. Loewenstein Y., and Seung H.S. Operant matching is a generic outcome of synaptic plasticity based on the covariance between reward and neural activity. Proc Natl Acad Sci U S A 103 (2006) 15224-15229
46. Seo H., Barraclough D.J., and Lee D. Dynamic signals related to choices and outcomes in the dorsolateral prefrontal cortex. Cereb Cortex 17 Suppl 1 (2007) i110-i117
47. Basso M.A., and Wurtz R.H. Modulation of neuronal activity by target uncertainty. Nature 389 (1997) 66-69. This is the first study to show the effect of uncertainty in the motor output (saccade location) on the preparatory activity of neurons that control this behavior. The authors find that the premovement activity of saccade-related neurons in SC decreases as the probability that a particular saccade will be selected decreases.
48. Basso M.A., and Wurtz R.H. Modulation of neuronal activity in superior colliculus by changes in target probability. J Neurosci 18 (1998) 7519-7534
49. Basso M.A., and Wurtz R.H. Neuronal activity in substantia nigra pars reticulata during target selection. J Neurosci 22 (2002) 1883-1894
50. Dorris M.C., and Munoz D.P. Saccadic probability influences motor preparation signals and time to saccadic initiation. J Neurosci 18 (1998) 7015-7026
51. Ikeda T., and Hikosaka O. Reward-dependent gain and bias of visual responses in primate superior colliculus. Neuron 39 (2003) 693-700
52. Lee K.M., and Keller E.L. Neural activity in the frontal eye fields modulated by the number of alternatives in target choice. J Neurosci 28 (2008) 2242-2251. The authors show that at different times during a color-to-location saccade task, FEF neurons show different forms of modulations with respect to the number of alternatives (NAs); visual neurons decrease their activity during stimulus presentation as NA is increased, whereas the visuo-motor neurons increase their activity during presaccadic time.
53. Fiorillo C.D., Tobler P.N., and Schultz W. Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299 (2003) 1898-1902
54. Takikawa Y., Kawagoe R., and Hikosaka O. Reward dependent spatial selectivity of anticipatory activity in monkey caudate neurons. J Neurophys 87 (2002) 508-515
55. Platt M.L., and Glimcher P.W. Neural correlates of decision variables in parietal cortex. Nature 400 (1999) 233-238
56. Ernst M.O., and Bulthoff H.H. Merging the senses into a robust percept. Trends Cogn Sci 8 (2004) 162-169
57. Ma W.J., Beck J.M., Latham P.E., and Pouget A. Bayesian inference with probabilistic population codes. Nat Neurosci 9 (2006) 1432-1438
58. Yang T., and Shadlen M.N. Probabilistic reasoning by neurons. Nature 447 (2007) 1075-1080