Reinforcement learning by Hebbian synapses with adaptive thresholds

Neuroscience

Volumn 81, Issue 2, 1997, Pages 303-319

  Pennartz, C M A ^a,b  

^a CALIFORNIA INSTITUTE OF TECHNOLOGY   (United States)

^b Netherlands Inst for Brain Research ^*  (Netherlands)

Author keywords

Amygdala; Long term depression; Long term potentiation; NMDA receptor; Prefrontal cortex; Reinforcement

Indexed keywords

N METHYL DEXTRO ASPARTIC ACID RECEPTOR;

ADAPTIVE BEHAVIOR; AMYGDALOID NUCLEUS; ANIMAL CELL; ARTICLE; CINGULATE GYRUS; DEPRESSION; LEARNING; LONG TERM POTENTIATION; MAMMAL; NERVE CELL PLASTICITY; NONHUMAN; PERCEPTIVE THRESHOLD; PREFRONTAL CORTEX; PRIORITY JOURNAL; REINFORCEMENT;

ADAPTATION, PHYSIOLOGICAL; ANIMALS; LEARNING; MODELS, NEUROLOGICAL; REINFORCEMENT (PSYCHOLOGY); SYNAPSES;

EID: 0031559957     PISSN: 03064522     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0306-4522(97)00118-8     Document Type: Article

References (102)

1. Ackley D. H. and Littman M. L. (1990) Generalization and scaling in reinforcement learning. In Advances in Neural Information Processing Systems 2, pp. 550-557. Kaufmann, San Mateo, CA.
2. Alger B. E., Nicoll R. A. Feed-forward dendritic inhibition in rat hippocampal pyramidal cells studied in vitro. J. Physiol. 328:1982;105-123.
3. Amaral D. G., Price J. L., Pitkänen A. and Carmichael S. T. (1992) Anatomical organization of the primate amygdaloid complex. In The Amygdala: Neurobiological Aspects of Emotion, Memory, and Mental Dysfunction (ed. Aggleton J. P.), pp. 1-66. Wiley-Liss, London.
4. Andersen R. A. Encoding of intention and spatial location in the posterior parietal cortex. Cerebr. Cort. 5:1995;457-469.
5. Apicella P., Ljungberg T., Scarnati E., Schultz W. Responses to reward in monkey dorsal and ventral striatum. Expl Brain Res. 85:1991;491-500.
6. Avendaño C., Price J. L., Amaral D. G. Evidence for an amygdaloid projection to premotor cortex but not to motor cortex in the monkey. Brain Res. 264:1983;111-117.
7. Barto A. G. and Jordan M. I. (1987) Gradient following without back-propagation in layered networks. In IEEE First Int. Conf. Neural Networks, Vol. II, pp. 629-636.
8. Barto A. G., Sutton R. S., Anderson C. W. Neuron-like adaptive elements that can solve difficult learning control problems. IEEE Trans. Syst. Man Cybern. 13:1983;834-846.
9. Bashir Z. I., Bortolotto Z. A., Davies C. H., Berretta N., Irving A. J., Seal A. J., Henley J. M., Jane D. E., Watkins J. C., Collingridge G. L. Induction of LTP in the hippocampus needs synaptic activation of glutamate metabotropic receptors. Nature. 363:1993;347-350.
10. Bear M. F., Cooper L. N., Ebner F. F. A physiological basis for a theory of synaptic modification. Science. 237:1987;42-48.
11. Berger B., Trottier S., Verney C., Gaspar P., Alvarez C. Regional and laminar distribution of the dopamine and serotonin innervation in the macaque cerebral cortex: a radioautographic study. J. comp. Neurol. 273:1988;99-119.
12. Bienenstock E. L., Cooper L. N., Munro P. W. Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex. J. Neurosci. 2:1982;32-48.
13. Bliss T. V. P., Collingridge G. L. A synaptic model of memory: long-term potentiation in the hippocampus. Nature. 361:1993;31-39.
14. Brown R. M., Crane A. M., Goldman P. S. Regional distribution of monoamines in the cerebral cortex and subcortical structures of the rhesus monkey: concentrations and in vivo synthesis rates. Brain Res. 168:1979;133-150.
15. Brown T. H., Kairiss E. W., Keenan C. L. Hebbian synapses: biophysical mechanisms and algorithms. A. Rev. Neurosci. 13:1990;475-511.
16. Caminiti R., Johnson P. B., Urbano A. Making arm movements within different parts of space: dynamic aspects in the primate motor cortex. J. Neurosci. 10:1990;2039-2058.
17. Castro-Alamancos M. A., Donoghue J. P., Connors B. W. Different forms of synaptic plasticity in somatosensory and motor areas of the neocortex. J. Neurosci. 15:1995;5324-5333.
18. Cavada C., Goldman-Rakic P. S. Posterior parietal cortex in rhesus monkey: II. Evidence for segregated corticocortical networks linking sensory and limbic areas with the frontal lobe. J. comp. Neurol. 287:1989;422-445.
19. b receptors in lamprey. J. Neurophysiol. 66:1991;2141-2149.
20. Christie B. R., Abraham W. C. Priming of associative long-term depression in the dentate gyrus by theta frequency synaptic activity. Nature. 9:1992;79-84.
21. Crick F. The recent excitement about neural networks. Nature. 337:1989;129-132.
22. 2+ signaling requirements for long-term depression in the hippocampus. Neuron. 16:1996;825-833.
23. Davis M., Rainnie D., Cassell M. Neurotransmission in the rat amygdala related to fear and anxiety. Trends Neurosci. 17:1994;208-214.
24. Dayan P., Montague P. R. and Sejnowski T. J. (1994) Foraging in an uncertain environment using predictive Hebbian learning. In Advances in Neural Information Processing Systems (eds Cowan J. D., Tesauro G. and Alspector J.), Vol. 6. Morgan Kaufmann, San Mateo, CA.
25. Dehaene S., Changeux J. P. The Wisconsin card sorting test: theoretical analysis and modeling in a neuronal network. Cerebr. Cort. 1:1991;62-79.
26. Dudek S. M., Bear M. F. Homosynaptic long-term depression in area CA1 of hippocampus and effects of N-methyl-D-aspartate receptor blockade. Proc. natn. Acad. Sci. U.S.A. 89:1992;4363-4367.
27. Farb C. R., Aoki C., LeDoux J. E. Differential localization of NMDA and AMPA receptor subunits in the lateral and basal nuclei of the amygdala: a light and electron microscopic study. J. comp. Neurol. 362:1995;86-108.
28. Finch D., Tan A. M., Isokawa-Akesson M. Feed-forward inhibition of the rat entorhinal cortex and subicular complex. J. Neurosci. 8:1988;2213-2226.
29. Fonnum F., Storm-Mathisen J., Divac I. Biochemical evidence for glutamate as neurotransmitter in corticostriatal and corticothalamic fibres in rat brain. Neuroscience. 6:1981;863-873.
30. Fregnac Y., Shulz D., Thorpe S., Bienenstock E. A cellular analogue of visual cortical plasticity. Nature. 333:1988;367-370.
31. Friston K. J., Tononi G., Reeke G. N., Sporns O., Edelman G. M. Value-dependent selection in the brain: simulation in a synthetic neural model. Neuroscience. 59:1994;229-243.
32. Funahashi S., Bruce C. J., Goldman-Rakic P. S. Neuronal activity related to saccadic eye movements in the monkey's dorsolateral prefrontal cortex. J. Neurophysiol. 65:1991;1464-1483.
33. Gaffan D. (1992) Amygdala and the memory of reward. In The Amygdala: Neurobiological Aspects of Emotion, Memory and Mental Dysfunction (ed. Aggleton J. P.), pp. 471-483. Wiley-Liss, London.
34. Georgopoulos A. P. (1990) Neural coding of the direction of reaching and a comparison with saccadic eye movements. In Cold Spring Harbor Symposia on Quantitative Biology, Vol. LV, pp. 849-859. Cold Spring Harbor Laboratory Press, Cold Spring Harbor.
35. Georgopoulos A. P. New concepts in generation of movement. Neuron. 13:1994;257-268.
36. Georgopoulos A. P., Lurito J. T., Petrides M., Schwartz A. B., Massey J. T. Mental rotation of the neuronal population vector. Science. 243:1989;234-236.
37. Van Gisbergen J. A. M., van Opstal A. J., Tax A. A. M. Collicular ensemble coding of saccades based on vector summation. Neuroscience. 21:1987;541-555.
38. Groenewegen H. J., Berendse H. W. The specificity of the "nonspecific" midline and intralaminar thalamic nuclei. Trends Neurosci. 17:1994;52-57.
39. Groenewegen H. J., Berendse H. W., Wolters J. G. and Lohman A. H. M. (1990) The anatomical relationship of the prefrontal cortex with the striatopallidal system, the thalamus and the amygdala: evidence for a parallel organization. In Progress in Brain Research (eds Uylings H. B. M., Van Eden C. G., De Bruin J. P. C., Corner M. A and Feenstra M. G. P.), Vol. 85, pp. 95-118. Elsevier, Amsterdam.
40. Hertz J., Krogh A. and Palmer R. G. (1991) Introduction to the Theory of Neural Computation. Addison-Wesley, Redwood City.
41. Hestrin S., Nicoll R. A., Perkel D. J., Sah P. Analysis of excitatory synaptic action in pyramidal cells using whole-cell recording from rat hippocampal slices. J. Physiol. 422:1990;203-225.
42. Hirsch J. C., Crepel F. Postsynaptic calcium is necessary for the induction of LTP and LTD of monosynaptic EPSPs in prefrontal neurons: an in vitro study in the rat. Synapse. 10:1992;173-175.
43. Hopfield J. J. Neurons with graded response have collective computational properties like those of two-state neurons. Proc. natn. Acad. Sci. U.S.A. 81:1984;3088-3092.
44. Huang Y.-Y., Colino A., Selig D. K., Malenka R. C. The influence of prior synaptic activity on the induction of long-term potentiation. Science. 255:1992;730-733.
45. Iriki A., Pavlides C., Keller A., Asanuma H. Long-term potentiation in the motor cortex. Science. 245:1989;1385-1387.
46. Keinänen K., Wisden W., Sommer B., Werner P., Herb A., Verdoorn T. A., Sakmann B., Seeburg P. H. A family of AMPA-selective glutamate receptors. Science. 249:1990;556-560.
47. Keller B. U., Konnerth A., Yaari Y. Patch clamp analysis of excitatory synaptic currents in granule cells of rat hippocampus. J. Physiol. 435:1991;275-293.
48. Kesner R. P., Williams J. M. Memory for magnitude of reinforcement: dissociation between the amygdala and hippocampus. Neurobiol. Learning Memory. 64:1995;237-244.
49. Kirkwood A., Dudek S. M., Gold J. T., Aizenman C. D., Bear M. F. Common forms of synaptic plasticity in the hippocampus and neocortex in vitro. Science. 260:1993;1518-1521.
50. Kirkwood A., Rioult M. G., Bear M. F. Experience-dependent modification of synaptic plasticity in visual cortex. Nature. 381:1996;526-528.
51. Kitamura Y., Miyazaki A., Yamanaka Y., Nomura Y. Stimulatory effects of protein kinase C and calmodulin kinase II on N-methyl-D-aspartate receptor/channels in the postsynaptic density of rat brain. J. Neurochem. 61:1993;100-109.
52. Kohonen T. (1989) Self-organization and Associative Memory. Springer, Berlin.
53. Kosar E., Grill H. J., Norgren R. Gustatory cortex in the rat. I. Physiological properties and cytoarchitecture. Brain Res. 379:1986;329-341.
54. Krettek J. E., Price J. L. Projections from the amygdaloid complex to the cerebral cortex and thalamus in the rat and cat. J. comp. Neurol. 172:1977;687-722.
55. Kriegstein A. R. Synaptic responses of cortical pyramidal neurons to light stimulation in the isolated turtle visual system. J. Neurosci. 7:1987;2488-2492.
56. 2+-dependent phosphatase. Nature. 369:1994;235-239.
57. Lisberger S. G. The neural basis for learning of simple motor skills. Science. 242:1988;728-735.
58. Lisman J. A mechanism for the Hebb and the anti-Hebb processes underlying learning and memory. Proc. natn. Acad. Sci. U.S.A. 86:1989;9574-9578.
59. Lynch G. S., Dunwiddie T., Gribkoff V. Heterosynaptic depression: a postsynaptic correlate of long-term potentiation. Nature. 266:1977;737-739.
60. Markowitsch H. J., Pritzel M. Reward related neurons in cat association cortex. Brain Res. 111:1976;185-188.
61. Mazzoni P., Andersen R. A., Jordan M. I. A more biologically plausible learning rule for neural networks. Proc. natn. Acad. Sci. U.S.A. 88:1991;4433-4437.
62. McDonald A. J. Glutamate and aspartate immunoreactive neurons of the rat basolateral amygdala: colocalization of excitatory amino acids and projections to the limbic circuit. J. comp. Neurol. 365:1996;367-379.
63. Mirenowicz J., Schultz W. Preferential activation of midbrain dopaine neurons by appetitive rather than aversive stimuli. Nature. 379:1996;449-451.
64. Miserendino M. J. D., Sananes C. B., Melia K. R., Davis M. Blocking of acquisition but not expression of conditioned fear-potentiated startle by NMDA antagonists in the amygdala. Nature. 345:1990;716-718.
65. Montague P. R., Dayan P., Nowlan S. J., Pouget A. and Sejnowski T. J. (1993) Using aperiodic reinforcement for directed self-organization during development. In Advances in Neural Information Processing Systems (eds Hanson S. J., Cowan J. D. and Giles C. L.), Vol. 5. Morgan Kaufmann, San Mateo, CA.
66. Morecraft R. J., Geula C., Mesulam M.-M. Cytoarchitecture and neural afferents of orbitofrontal cortex in the brain of the monkey. J. comp. Neurol. 323:1992;341-358.
67. Morecraft R. J., Van Hoesen G. W. Cingulate input to the primary and supplementary motor cortices in the rhesus monkey: evidence for somatotopy in areas 24c and 23c. J. comp. Neurol. 322:1992;471-489.
68. Mulkey R. M., Herron C. E., Malenka R. C. An essential role for protein phosphatases in hippocampal long-term depression. Science. 261:1993;1051-1055.
69. Mulkey R. M., Malenka R. C. Mechanisms underlying induction of homosynaptic long-term depression in area CA1 of the hippocampus. Neuron. 9:1992;967-975.
70. i needed to trigger LTD and LTP. Neuron. 16:1996;619-629.
71. Niki H., Watanabe M. Prefrontal and cingulate unit activity during timing behavior in the monkey. Brain Res. 171:1979;213-224.
72. Nishijo H., Ono T., Nishino H. Single neuron responses in amygdala of alert monkey during complex sensory stimulation with affective significance. J. Neurosci. 8:1988;3570-3583.
73. Pennartz C. M. A. The ascending neuromodulatory systems in learning by reinforcement: comparing computational conjectures with experimental findings. Brain Res. Rev. 21:1996;219-245.
74. Pennartz C. M. A., Ameerun R. F., Groenewegen H. J., Lopes da Silva F. H. Synaptic plasticity in an in vitro slice preparation of the rat nucleus accumbens. Eur. J. Neurosci. 5:1993;107-117.
75. Pennartz C. M. A., Groenewegen H. J., Lopes da Silva F. H. The nucleus accumbens as a complex of functionally distinct neuronal ensembles: an integration of behavioural, electrophysiological and anatomical data. Prog. Neurobiol. 42:1994;719-761.
76. Pennartz C. M. A., Kitai S. T. Hippocampal inputs to identified neurons in an in vitro slice preparation of the rat nucleus accumbens: evidence for feed-forward inhibition. J. Neurosci. 11:1991;2838-2847.
77. Pitts W., McCulloch W. S. How we know universals. The perception of auditory and visual forms. Bull. math. Biophys. 9:1947;127.
78. Robinson D. A. Eye movements evoked by collicular stimulation in the alert monkey. Vision Res. 12:1972;1795.
79. Rogan M. T., LeDoux J. E. LTP is accompanied by commensurate enhancement of auditory-evoked responses in a fear conditioning circuit. Neuron. 15:1995;127-136.
80. Rolls E. T., Sanghera M. K., Roper-Hall A. The latency of activation of neurones in the lateral hypothalamus and substantia innominata during feeding in the monkey. Brain Res. 164:1979;121-135.
81. Romanski L. M., LeDoux J. E. Information cascade from primary auditory cortex to the amygdala: corticocortical and corticoamygdaloid projections of temporal cortex in the rat. Cerebr. Cort. 3:1993;515-532.
82. Rosenkilde C. E., Bauer R. H., Fuster J. M. Single cell activity in ventral prefrontal cortex of behaving monkeys. Brain Res. 209:1981;375-394.
83. Rumelhart D. E., Hinton G. E. and Williams R. J. (1986) Learning internal representations by error propagation. In Parallel Distributed Processing (eds Rumelhart D. E., McClelland J. L. and the PDP research group), Vol. 1, chapter 8. MIT Press, Cambridge, MA.
84. Sah P., Nicoll R. A. Mechanisms underlying potentiation of synaptic transmission in rat anterior cingulate cortex in vitro. J. Physiol. 433:1991;615-630.
85. Sanghera M. K., Rolls E. T., Roper-Hall A. Visual responses of neurons in the dorsolateral amygdala of the alert monkey. Expl Neurol. 63:1979;610-626.
86. Sarter M., Markowitsch H. J. Involvement of the amygdala in learning and memory: a critical review, with emphasis on anatomical relations. Behav. Neurosci. 99:1985;342-380.
87. Selig D. K., Hjelmstad G. O., Herron C. E., Nicoll R. A., Malenka R. C. Independent mechanisms for long-term depression of AMPA and NMDA responses. Neuron. 15:1995;417-426.
88. Sikes R. W., Vogt B. A. Nociceptive neurons in area 24 of rabbit cingulate cortex. J. Neurophysiol. 68:1992;1720-1732.
89. Streit P. (1984) Glutamate and aspartate as transmitter candidates for systems of the cerebral cortex. In Cerebral Cortex (eds Peters A. and Jones E. G.), Vol. 2, pp. 119-143. Plenum, New York.
90. Sutton R. S. Learning to predict by the methods of temporal differences. Machine Learning. 3:1988;9-44.
91. Sutton R. S. and Barto A. G. (1987) A temporal-difference model of classical conditioning. In Ninth Annual Conf. Cogn. Society, pp. 355-378. Lawrence Erlbaum Assoc.
92. Takayama K., Miura M. Glutamate-immunoreactive neurons of the central amygdaloid nucleus projecting to the subretrofacial nucleus of SHR and WKY rats: a double-labeling study. Neurosci. Lett. 134:1991;62-66.
93. Thorpe S. J., Rolls E. T., Maddison S. The orbitofrontal cortex: neuronal activity in the behaving monkey. Expl Brain Res. 49:1983;93-115.
94. Vogt B. A. (1985) Cingulate Cortex. In Cerebral Cortex (eds Peters A. and Jones E. G.), Vol. 4, pp. 89-149. Plenum, New York.
95. Wang L. Y., Orser B. A., Brautigan D. L., McDonald J. F. Regulation of NMDA receptors in cultured hippocampal neurons by protein phosphatases 1 and 2A. Nature. 369:1994;230-232.
96. Wang Y. T., Salter M. W. Regulation of NMDA receptors by tyrosine kinases and phosphatases. Nature. 369:1994;233-235.
97. Watanabe M. Reward expectancy in primate prefrontal neurons. Nature. 382:1996;629-632.
98. Werbos P. (1974) Beyond regression: new tools for prediction and analysis in the behavioral sciences. Ph.D. thesis, Harvard University.
99. White E. L. (1989) Cortical Circuits. Synaptic Organization of the Cerebral Cortex. Structure, Function, and Theory. Birkhäuser, Basel.
100. Yasui Y., Breder C. D., Saper C. B., Cechetto D. F. Autonomic responses and efferent pathways from the insular cortex in the rat. J. comp. Neurol. 303:1991;355-374.
101. Yim C. Y., Mogenson G. J. Neuromodulatory action of dopamine in the nucleus accumbens: an in vivo intracellular study. Neuroscience. 26:1988;403-415.
102. Zhuo M., Hu Y., Schultz C., Kandel E. R., Hawkins R. D. Role of guanylyl cyclase and cGMP-dependent protein kinase in long-term potentiation. Nature. 368:1994;635-639.