Dopamine Neurons Can Represent Context-Dependent Prediction Error

Neuron

Volumn 41, Issue 2, 2004, Pages 269-280

  Nakahara, Hiroyuki ^a   Itoh, Hideaki ^b   Kawagoe, Reiko ^c   Takikawa, Yoriko ^c   Hikosaka, Okihide ^d  

^a RIKEN BRAIN SCIENCE INSTITUTE   (Japan)

^b TOKYO INSTITUTE OF TECHNOLOGY   (Japan)

^c JUNTENDO UNIVERSITY   (Japan)

^d NATIONAL EYE INSTITUTE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ANIMAL BEHAVIOR; ANIMAL EXPERIMENT; ARTICLE; CELL FUNCTION; CONTROLLED STUDY; DOPAMINERGIC NERVE CELL; ERROR; LEARNING; MENTAL TASK; MONKEY; NONHUMAN; PREDICTION; PRIORITY JOURNAL; PROBABILITY; REWARD; STIMULUS RESPONSE; TASK PERFORMANCE; VISUAL STIMULATION;

EID: 1642575165     PISSN: 08966273     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0896-6273(03)00869-9     Document Type: Article

References (57)

1. Arbib M.A., Dominey P.F. Modeling the roles of basal ganglia in timing and sequencing saccadic eye movements. Houk J.C., Davis J.L., Beiser D.G. Models of Information Processing in the Basal Ganglia. 1995;149-162 MIT Press, Cambridge, MA.
2. Barto A.G. Adaptive critics and the basal ganglia. Houk J.C., Davis J.L., Beiser D.G. Models of Information Processing in the Basal Ganglia. 1995;215-232 MIT Press, Cambridge, MA.
3. Berns G.S., Sejnowski T.J. A computational model of how the basal ganglia produce sequences. J. Cogn. Neurosci. 10:1998;108-121.
4. Coe B., Tomihara K., Matsuzawa M., Hikosaka O. Visual and anticipatory bias in three cortical eye fields of the monkey during an adaptive decision-making task. J. Neurosci. 22:2002;5081-5090.
5. Dayan P. Motivated reinforcement learning. NIPS. 14:2002;11-18.
6. Doya K. Metalearning and neuromodulation. Neural Netw. 15:2002;495-506.
7. Fiorillo C.D., Tobler P.N., Schultz W. Discrete coding of reward probability and uncertainty by dopamine neurons. Science. 299:2003;1898-1902.
8. Graybiel A.M., Aosaki T., Flaherty A.W., Kimura M. The basal ganglia and adaptive motor control. Science. 265:1994;1826-1831.
9. Hollerman J.R., Schultz W. Dopamine neurons report an error in the temporal prediction of reward during learning. Nat. Neurosci. 1:1998;304-309.
10. Houk J.C., Adams J.L., Barto A.G. A model of how the basal ganglia generate and use neural signals that predict reinforcement. Houk J.C., Davis J.L., Beiser D.G. Models of Information Processing in the Basal Ganglia. 1995;249-270 MIT Press, Cambridge, MA.
11. Judge S.J., Richmond B.J., Chu F.C. Implantation of magnetic search coils for measurement of eye position. an improved method Vision Res. 20:1980;535-538.
12. Kawagoe R., Takikawa Y., Hikosaka O. Expectation of reward modulates cognitive signals in the basal ganglia. Nat. Neurosci. 1:1998;411-416.
13. Kawagoe, R., Takikawa, Y., and Hikosaka, O. (2003). Reward-predicting activity of dopamine and caudate neurons - a possible mechanism of motivational control of saccadic eye movement. J. Neurophysiol., in press. Published online October 1, 2003. 10.1152/jn.00721.2003 .
14. Kobayashi S., Lauwereyns J., Koizumi M., Sakagami M., Hikosaka O. Influence of reward expectation on visuospatial processing in macaque lateral prefrontal cortex. J. Neurophysiol. 87:2002;1488-1498.
15. Lauwereyns J., Takikawa Y., Kawagoe R., Kobayashi S., Koizumi M., Coe B., Sakagami M., Hikosaka O. Feature-based anticipation of cues that predict reward in monkey caudate nucleus. Neuron. 33:2002;463-473. a.
16. Lauwereyns J., Watanabe K., Coe B., Hikosaka O. A neural correlate of response bias in monkey caudate nucleus. Nature. 418:2002;413-417. b.
17. Leon M.I., Shadlen M.N. Effect of expected reward magnitude on the response of neurons in the dorsolateral prefrontal cortex of the macaque. Neuron. 24:1999;415-425.
18. Ljungberg T., Apicella P., Schultz W. Responses of monkey dopamine neurons during learning of behavioral reactions. J. Neurophysiol. 67:1992;145-163.
19. Lu X., Matsuzawa M., Hikosaka O. A neural correlate of oculomotor sequences in supplementary eye field. Neuron. 34:2002;317-325.
20. Matsumoto K., Suzuki W., Tanaka K. Neuronal correlates of goal-based motor selection in the prefrontal cortex. Science. 301:2003;229-232.
21. Meck W.H. Neuropharmacology of timing and time perception. Brain Res. Cogn. Brain Res. 3:1996;227-242.
22. Mirenowicz J., Schultz W. Importance of unpredictability for reward responses in primate dopamine neurons. J. Neurophysiol. 72:1994;1024-1027.
23. Montague P.R., Dayan P., Person C., Sejnowski T.J. Bee foraging in uncertain environments using predictive Hebbian learning. Nature. 377:1995;725-728.
24. Montague P., Dayan P., Sejnowski T. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16:1996;1936-1947.
25. Nakahara H., Doya K., Hikosaka O. Parallel cortico-basal ganglia mechanisms for acquisition and execution of visuomotor sequences - a computational approach. J. Cogn. Neurosci. 13:2001;626-647.
26. Nakamura K., Sakai K., Hikosaka O. Neuronal activity in medial frontal cortex during learning of sequential procedures. J. Neurophysiol. 80:1998;2671-2687.
27. Nieder A., Miller E.K. Coding of cognitive magnitude. compressed scaling of numerical information in the primate prefrontal cortex Neuron. 37:2003;149-157.
28. Parent A., Hazrati L.-N. Multiple striatal representation in primate substantia nigra. J. Comp. Neurol. 344:1994;305-320.
29. Platt M.L., Glimcher P.W. Neural correlates of decision variables in parietal cortex. Nature. 400:1999;233-238.
30. Procyk E., Tanaka Y.L., Joseph J.P. Anterior cingulate activity during routine and non-routine sequential behaviors in macaques. Nat. Neurosci. 3:2000;502-508.
31. Rescorla R.A., Wagner A.R. A theory of Pavlovian conditioning. variations in the effectiveness of reinforcement and nonreinforcement Black A.H., Prokasy W.F. Classical Conditioning. Current Research and Theory:1972;64-99 Applenton Century Crofts, New York.
32. Reynolds J.N., Hyland B.I., Wickens J.R. A cellular mechanism of reward-related learning. Nature. 413:2001;67-70.
33. Robinson D.A. A method of measuring eye movement using a scleral search coil in a magnetic field. IEEE Trans. Biomed. Eng. 10:1963;137-145.
34. Romo R., Schultz W. Dopamine neurons of the monkey midbrain. contingencies of responses to active touch during self-initiated arm movements J. Neurophysiol. 63:1990;592-606.
35. Satoh T., Nakai S., Sato T., Kimura M. Correlated coding of motivation and outcome of decision by dopamine neurons. J. Neurosci. 23:2003;9913-9923.
36. Schall J.D., Stuphorn V., Brown J.W. Monitoring and control of action by the frontal lobes. Neuron. 36:2002;309-322.
37. Schlag-Rey M., Amador N., Sanchez H., Schlag J. Antisaccade performance predicted by neuronal activity in the supplementary eye field. Nature. 390:1997;398-401.
38. Schultz W. Predictive reward signal of dopamine neurons. J. Neurophysiol. 80:1998;1-27.
39. Schultz W., Romo R. Responses of nigrostriatal dopamine neurons to high-intensity somatosensory stimulation in the anesthetized monkey. J. Neurophysiol. 57:1987;201-217.
40. Schultz W., Romo R., Ljungberg T., Mirenowicz J., Hollerman J.R., Dickinson A. Reward-related signals carried by dopamine neurons. Houk J.C., Davis J.L., Beiser D.G. Models of Information Processing in the Basal Ganglia. 1995;233-248 MIT Press, Cambridge, MA.
41. Schultz W., Dayan P., Montague P.R. A neural substrate of prediction and reward. Science. 275:1997;1593-1599.
42. Shidara M., Richmond B.J. Anterior cingulate. single neuronal signals related to degree of reward expectancy Science. 296:2002;1709-1711.
43. Shima K., Tanji J. Role for cingulate motor area cells in voluntary movement selection based on reward. Science. 282:1998;1335-1338.
44. Shima K., Mushiake H., Saito N., Tanji J. Role for cells in the presupplementary motor area in updating motor plans. Proc. Natl. Acad. Sci. USA. 93:1996;8694-8698.
45. Stuphorn V., Taylor T.L., Schall J.D. Performance monitoring by the supplementary eye field. Nature. 408:2000;857-860.
46. Suri R.E. Anticipatory responses of dopamine neurons and cortical neurons reproduced by internal model. Exp. Brain Res. 140:2001;234-240.
47. Suri R.E., Schultz W. Learning of sequential movements by neural network model with dopamine-like reinforcement signal. Exp. Brain Res. 121:1998;350-354.
48. Suri R.E., Schultz W. Temporal difference model reproduces anticipatory neural activity. Neural Comput. 13:2001;841-862.
49. Sutton R.S. Dyna, an integrated architecture for learning, planning, and reacting. SIGART Bulletin. 2:1991;160-163.
50. Sutton R.S., Barto A.G. Toward a modern theory of adaptive networks. expectation and prediction Psychol. Rev. 88:1981;135-170.
51. Sutton R.S., Barto A.G. Reinforcement Learning. An Introduction:1998;A Bradford Book, Cambridge, MA.
52. Takikawa Y., Kawagoe R., Hikosaka O. Reward-dependent spatial selectivity of anticipatory activity in monkey caudate neurons. J. Neurophysiol. 87:2002;508-515. a.
53. Takikawa Y., Kawagoe R., Itoh H., Nakahara H., Hikosaka O. Modulation of saccadic eye movements by predicted reward outcome. Exp. Brain Res. 142:2002;284-291. b.
54. Tanji J. Sequential organization of multiple movements. involvement of cortical motor areas Annu. Rev. Neurosci. 24:2001;631-651.
55. Tesauro G.J. TD-Gammon, a self-teaching backgammon program, achieves masterlevel play. Neural Comput. 6:1994;215-219.
56. Waelti P., Dickinson A., Schultz W. Dopamine responses comply with basic assumptions of formal learning theory. Nature. 412:2001;43-48.
57. Watanabe M. Reward expectancy in primate prefrontal neurons. Nature. 382:1996;629-632.