A framework for mesencephalic dopamine systems based on predictive Hebbian learning

Journal of Neuroscience

Volumn 16, Issue 5, 1996, Pages 1936-1947

  Montague, P Read ^a   Dayan, Peter ^b   Sejnowski, Terrence J ^c,d  

^a BAYLOR COLLEGE OF MEDICINE   (United States)

^b MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^c HOWARD HUGHES MEDICAL INSTITUTE   (United States)

^d UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

diffuse ascending systems; dopamine; prediction; reinforcement learning; reward; synaptic plasticity

Indexed keywords

ARTICLE; DOPAMINERGIC SYSTEM; LEARNING; MESENCEPHALON; NERVE CELL PLASTICITY; NONHUMAN; PRIORITY JOURNAL; REINFORCEMENT; SIGNAL TRANSDUCTION; SYNAPSE;

EID: 0029981543     PISSN: 02706474     EISSN: None     Source Type: Journal    
DOI: 10.1523/jneurosci.16-05-01936.1996     Document Type: Article

References (53)

1. Acquas E, Carboni E, Di Chiara G (1991) Profound depression of mesolimbic dopamine release after morphine withdrawal in dependent rats. Eur J Pharmacol 193:133-134.
2. Barto AG, Sutton RS, Watkins CJCH (1989) Technical Report 89-95. Amherst, MA: University of Massachusetts.
3. Bechara A, Damasio A, Damasio H, Anderson S (1994) Insensitivity to future consequences following damage to human prefrontal cortex. Cognition 50:7-15.
4. Bernheimer H, Birkmayer W, Hornykiewicz O, Jellinger K, Seitelberger F (1973) Brain dopamine and the syndromes of Parkinson and Huntington: clinical, morphological and neurochemical correlations. J Neurol Sci 20:415-55.
5. Bush RR, Mosteller F (1955) Stochastic models for learning. New York: Wiley.
6. Crippens D, Robinson TE (1994) Withdrawal from morphine or amphetamine: different effects on dopamine in the ventral-medial striatum studied with microdialysis. Brain Res 650:56-62.
7. Delong MR, Crutcher MD, Georgopoulos AP (1983) Relations between movement and single cell discharge in the substantia nigra of the behaving monkey. J Neurosci 3:1599-1606.
8. Diana M, Pistis M, Carboni S, Gessa GL, Rossetti ZL (1993) Profound decrement of mesolimbic dopaminergic neuronal activity during ethanol withdrawal syndrome in rats: electrophysiological and biochemical evidence. Proc Natl Acad Sci USA 90:7966-7969.
9. Dickinson A (1980) Contemporary animal learning theory. Cambridge: Cambridge UP.
10. Doya K, Sejnowski TJ (1995) A novel reinforcement model of birdsong vocalization learning. In: Advances in neural information processing systems, Vol 7. (Tesauro G, Touretzky D, Alspector J, eds). Cambridge: MIT, in press.
11. Egelman DM, Person C, Montague PR (1995) A predictive model for diffuse systems matches human choices in a simple decision-making task. Soc Neurosci Abstr 21:2087.
12. Fibiger HC, Phillips AG (1986) Reward, motivation, cognition: psycholobiology of mesotelencephalic dopamine systems. In: Handbook of physiology. The nervous system, Intrinsic regulatory systems of the brain, Vol 4, pp 647-675. Bethesda: American Physiological Society.
13. Freeman AS, Bunney BS (1987) Activity of A9 and A10 dopaminergic neurons in unrestrained rats: further characterization and effects of cholecystokinin. Brain Res 405:46-55.
14. Gallagher M, Holland PC (1994) The amygdala complex: multiple roles in associative learning and attention. Proc Natl Acad Sci USA 91:11771-11776.
15. Gallistel CR (1990) The organization of learning. Cambridge: MIT.
16. Grossberg S, Levine DS (1987) Neural dynamics of attentionally modulated Pavlovian conditioning: blocking, interstimulus interval, and secondary reinforcement. Appl Optics 26:5015-5030.
17. Grossberg S, Schmajuk NA (1989) Neural dynamics of adaptive timing and temporal discrimination during associative learning. Neural Networks 2:79-102.
18. Harder LD, Real LA (1987) Why are bumble bees risk averse? Ecology 68:1104-1108.
19. Herrnstein RJ (1961) Relative and absolute strength of response as a function of frequency of reinforcement. J Exp Anal Behav 4:267-272.
20. Herrnstein RJ (1991) Experiments on stable suboptimality in individual behavior. Am Econ Rev Papers Proc 83:360-364.
21. Houk JC, Adams JL, Barto AG (1995) A model of how the basal ganglia generate and use neural signals that predict reinforcement. In: Models of information processing in the basal ganglia (Houk JC, Davis JL, Beiser DG, eds). Cambridge: MIT.
22. Kalman RE (1960) A new approach to linear filtering and prediction problems. J Basic Eng Trans ASME 82:35-45.
23. Klopf AH (1982) The hedonistic neuron. New York: Taylor and Francis.
24. Koob GF, Bloom FE (1988) Cellular and molecular mechanisms of drug dependence. Science 242:715-723.
25. Ljungberg T, Apicella P, Schultz W (1992) Responses of monkey dopamine neurons during learning of behavioral reactions. J Neurophysiol 67:145-163.
26. Luce RD, Raiffa H (1957) Games and decisions: introduction and critical survey. New York: Wiley.
27. Mackintosh NJ (1983) Conditioning and associative learning. New York: Oxford UP.
28. Montague PR (1996) Biological substrates of predictive mechanisms in learning and action choice. In: Neural-network approaches to cognition biobehavioral foundations (Donahoe PP, ed). New York: Elsevier Science, in press.
29. Montague PR, Sejnowksi TJ (1994) The predictive brain: temporal coincidence and temporal order in synaptic learning mechanisms. Learn Memory 1:1-33.
30. Montague PR, Dayan P, Nowlan SJ, Sejnowski TJ (1993) Using aperiodic reinforcement for directed self-organization. In: Advances in neural information processing systems (Giles CL, Hanson SJ, Cowan JD eds). San Mateo, CA: Morgan Kaufmann.
31. Montague PR, Dayan P, Person C, Sejnowski TJ (1995) Bee foraging in uncertain environments using predictive Hebbian learning. Nature 376:725-728.
32. Moore JW, Desmond JE, Bethier NE, Blazis DE, Sutton RS, Barto AG (1986) Simulation of the classically-conditioned nictitating membrane response by a neuron-like adaptive element: response topography, neuronal firing, and interstimulus intervals. Behav Brain Res 12:143-154.
33. Oades RD, Halliday GM (1987) Ventral tegmental (A10) system: neurobiology. 1. Anatomy and connectivity. Brain Res 434:117-165.
34. Parsons LH, Smith AD, Justice Jr JB (1991) Basal extracellular dopamine is decreased in the rat nucleus accumbens during abstinence from chronic cocaine. Synapse 9:60-65.
35. Quartz SR, Dayan P, Montague PR, Sejnowski TJ (1992) Expectation learning in the brain using diffuse ascending projections. Soc Neurosci Abstr 18:1210.
36. Real LA (1991) Animal choice behavior and the evolution of cognitive architecture. Science 253:980-986.
37. Rescorla RA, Wagner AR (1972) A theory of Pavlovian conditioning: the effectiveness of reinforcement and non-reinforcement. In: Classical conditioning. 2. Current research and theory (Black AH, Prokasy WF, eds), pp 64-69. New York: Appleton Century-Crofts.
38. Romo R, Schultz W (1990) Dopamine neurons of the monkey midbrain: contingencies of responses to active touch during self-initiated arm movements. J Neurophysiol 63:592-606.
39. Rossetti ZL, Hmaidan Y, Gessa GL (1992) Marked inhibition of mesolimbic dopamine release: a common feature of ethanol, morphine, cocaine and amphetamine abstinence in rats. Eur J Pharmacol 221:227-234.
40. Sawaguchi T, Goldman-Rakic PS (1991) D1 dopamine receptors in prefrontal cortex: involvement in working memory. Science 251:947-950.
41. Schultz W (1992) Activity of dopamine neurons in the behaving primate. Semin Neurosci 4:129-138.
42. Schultz W, Romo R (1990) Dopamine neurons of the monkey midbrain: contingencies of responses to stimuli eliciting immediate behavioral reactions. J Neurophysiol 63:607-624.
43. Schultz W, Apicella P, Ljungberg T (1993) Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J Neurosci 13:900-13.
44. Simon RS (1988) Learning to predict by the methods of temporal difference. Machine Learning 3:9-44.
45. Sutton RS, Barto AG (1981) Toward a modern theory of adaptive networks: expectation and prediction. Psychol Rev 88:135-170.
46. Sutton RS, Barto AG (1987) A temporal-difference model of classical conditioning. Proceedings of the Ninth Annual Conference of the Cognitive Science Society, Seattle.
47. Sutton RS, Barto AG (1990) Time-derivative models of Pavlovian reinforcement. In: Learning and computational neuroscience (Gabriel M, Moore J. eds). Cambridge: MIT.
48. Tranel D, Damasio AR (1985) Knowledge without awareness: an autonomic index of facial recognition by prosopagnosics. Science 228:1453-1454.
49. von Neumann J, Morgenstern O (1947) Theory of games and economic behavior. Princeton: Princeton UP.
50. Wickens J, Kotter R (1995) Cellular models of reinforcement. Models of information processing in the basal ganglia (Houk JC, Davis JL, Beiser DG, eds). Cambridge: MIT.
51. Widrow B, Stearns SD (1985) Adaptive signal processing. Englewood Cliffs, NJ: Prentice-Hall.
52. Wise RA (1982) Neuroleptics and operant behavior: the anhedonia hypothesis. Behav Brain Sci 5:39.
53. Wise RA, Bozarth MA (1984) Brain reward circuitry: four circuit elements "wired" in apparent series. Brain Res Bull 12:203-208.