Temporal sequence learning, prediction, and control: A review of different models and their relation to biological mechanisms

Neural Computation

Volumn 17, Issue 2, 2005, Pages 245-319

  Wörgötter, Florentin ^a   Porr, Bernd ^a  

^a UNIVERSITY OF STIRLING   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL NEURAL NETWORK; FORECASTING; LEARNING; REVIEW; TIME;

FORECASTING; LEARNING; NEURAL NETWORKS (COMPUTER); TIME FACTORS;

EID: 13244267004     PISSN: 08997667     EISSN: None     Source Type: Journal    
DOI: 10.1162/0899766053011555     Document Type: Review

References (276)

1. Abarbanel, H. D. I., Gibb, L., Huerta, R., & Rabinovich, M. I. (2003). Biophysical model of synaptic plasticity dynamics. Biol. Cybern., 89(3), 214-226.
2. Abarbanel, H. D. I., Huerta, R., & Rabinovich, M. I. (2002). Dynamical model of long-term synaptic plasticity. Proc. Natl. Acad. Sci. (USA), 99(15), 10132-10137.
3. Abbott, L. F., & Blum, K. I. (1996). Functional significance of long-term potentiation for sequence learning and prediction. Cereb. Cortex, 6, 406-416.
4. Abbott, L. S., & Song, S. (1999). Temporal asymmetric Hebbian learning, spike timing and neuronal response variability. In M. S. Kearns, S. Solla, & D. A. Cohn, (Eds.), Advances in neural information processing systems, (pp. 69-75). Cambridge, MA: MIT Press.
5. Abeles, M. (1991). Corticotronics: Neural circuits of the cerebral cortex. Cambridge. Cambridge University Press.
6. Akopian, G., Musleh, W., Smith, R., & Walsh, J. P. (2000). Functional state of corticostriatal synapses determines their expression of short- and long-term plasticity. Synapse, 38, 271-280.
7. Altar, C. A., Boyar, W. C., & Kim, H. S. (1990). Discriminatory roles for D1 and D2 dopamine receptor subtypes in the in vivo control of neostriatal cyclic GMP. Eur. J. Pharmacol., 181, 17-21.
8. Anderson, C., & Crawford-Hines, S. (1994). Multigrid q-learning (Tech. Rep. No. CS-94-121), Fort Collins: Colorado State University.
9. Artola, A., & Singer, W. (1993). Long-term depression of excitatory synaptic transmission and its relationship to long-term potentation. Trends Neurosci., 16, 480-487.
10. Ashby, W. R. (1952). Design for a brain. London: Chapman & Hall.
11. Balkenius, C., & Moren, J. (1998). Computational models of classical conditioning: A comparative study. (LUCS 62 ISSN 1101-8453). Lund, Sweden Lund University Cognitive Studies.
12. Barto, A. (1995). Adaptive critics and the basal ganglia. In J. C. Houk, J. L. Davis, & D. G. Beiser (Eds.), Models of information processing in the basal ganglia (pp. 215-232). Cambridge, MA: MIT Press.
13. Barto, A. G., Sutton, R. S., & Anderson, C. W. (1983). Neuronlike elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics, 13, 835-846.
14. Bell, C. C., Han, V. Z., Sugawara, Y., & Grankt, K. (1997). Synaptic plasticity in a cerebellum-like structure depends on temporal order. Nature, 387, 278-281.
15. Bellman, R. E. (1957). Dynamic programming. Princeton, NJ: Princeton University Press.
16. Benett, M. R. (2000). The concept of long term potentiation of transmission at synapses. Prog. Neurobiol., 60, 109-137.
17. Berger, B., Trottier, S., Verney, C., Gaspar, P., & Alvarez, C. (1988). Regional and laminar distribution of the dopamine and serotonin innervation in the macaque cerebral cortex: A radioautographic study. J. Comput. Neural., 273, 99-119.
18. Berns, G. S., McClure, S. M., Pagoni, G., & Montague, P. R. (2001). Predictability modulates human brain responses to reward. J. Neurosci., 21, 2793-2798.
19. Berns, G. S., & Sejnowski, T. J. (1998). A computational model of how the basal ganglia produce sequences. J. Cogn. Neurosci., 10 (1), 108-121.
20. Berridge, K. C., & Robinson, T. E. (1998). What is the role of dopamine in reward: Hedonic impact, reward learning or incentive salience? Brain Res. Rev., 28, 309-369.
21. Berridge, M. J. (1998). Neuronal calcium signaling. Neuron, 21, 13-26.
22. Bi, G. Q. (2002). Spatiotemporal specificity of synaptic plasticity: Cellular rules and mechanisms. Biol. Cybern., 87, 319-332.
23. Bi, G. Q., & Poo, M. M. (1998). Synaptic modifications in cultured hippocampal neurons: Dependence on spike timing, synaptic strength, and postsynaptic cell type. J. Neurosci., 18, 10464-10472.
24. Bi, G.-Q., & Poo, M. (2001). Synaptic modification by correlated activity: Hebb's postulate revisited. Annu. Rev. Neurosci., 24, 139-166.
25. Bibb, J. A., Snyder, G. L., Nishi, A., Yan, Z., Meijer, L., Fienberg, A. A., Tsai, L. H., Kwon, Y. T., Girault, J. A., Czernik, A. J., Huganir, R. L., Hemmings, H. C., Nairn, A. C., and Greengard, P. (1999). Phosphorylation of DARPP-32 by Cdk5 modulates dopamine signalling in neurons. Nature, 402, 669-671.
26. Bienenstock, E., Cooper, L., & Munro, P. (1982). Theory for the development of neuron selectivity: Orientation specificity and binocular interaction in visual cortex. J. Neurosci., 2(1), 32-48.
27. Bliss, T. V., & Gardner-Edwin, A. R. (1973). Long-lasting potentiation of synaptic transmission in the dentate area of the unanaesthetized rabbit following stimulation of the perforant path. J. Physiol. (Lond.), 232, 357-374.
28. Bliss, T. V., & Lomo, T. (1970). Plasticity in a monosynaptic cortical pathway. J. Physiol. (Lond.), 207, 61P.
29. Bliss, T. V., & Lomo, T. (1973). Long-lasting potentiation of synaptic transmission in the dentate area of the anaesthetized rabbit following stimulation of the perforant path. J. Physiol. (Lond.), 232, 331-356.
30. Breiter, H. C., Aharon, I., Kahneman, D., Dale, A., & Shizgal, P. (2001). Functional imaging of neural responses to expectancy and experience of monetary gains and losses. Neuron, 30, 619-639.
31. Brown, J., Bullock, D., & Grossberg, S. (1999). How the basal ganglia use parallel excitatory and inhibitory learning pathways to selectively respond to unexpected rewarding cues. J. Neurosci., 19(23), 10502-10511.
32. Brown, J. R., & Arbuthnott, G. W. (1983). The electrophysiology of dopamine D2 receptors: A study of the actions of dopamine on corticostriatal transmission. Neurosci., 10, 349-355.
33. Buchs, N. J., & Senn, W. (2002). Spike-based synaptic plasticity and the emergence of direction selective simple cells: Simultation results. J. Comput. Neurosci., 13, 167-186.
34. Bunney, B. S., Chiodo, L. A., & Grace, A. A. (1991). Midbrain dopamine system electrophysiological functioning: A review and new hypothesis. Synapse, 9, 79-94.
35. Buzsaki, G., Penttonen, M., Nadasdy, Z., & Bragin, A. (1996). Pattern and inhibition-dependent invasion of pyramidal cell dendrites by fast spikes in the hippocampus in vivo. Proc. Natl. Acad. Sci. USA, 93, 9921-9925.
36. Calabresi, P., Centonze, D., Gubellini, P., Marfia, G. A., & Bernardi, G. (1999). Glutamate-triggered events inducing corticostriatal long-term depression. J. Neurosci., 19, 6102-6110.
37. Calabresi, P., Fedele, E., Pisani, A., Fontana, G., Mercuri, N. B., Bernardi, G., & Raiteri, M. (1995). Transmitter release associated with long-term synaptic depression in rat corticostriatal slices. European J. Neurosci., 7, 1889-1894.
38. Calabresi, P., Gubellini, P., Centonze, D., Picconi, B., Bernardi, G., Chergui, K., Svenningsson, P., Fienberg, A. A., & Greengard, P. (2000). Dopamine and cAMP-regulated phosphoprotein 32 kDa controls both striatal long-term depression and long-term potentiation opposing forms of synaptic plasticity. J. Neurosci., 20(22), 8443-8451.
39. Calabresi, P., Maj, R., Mercuri, N. B., & Bernardi, G. (1992a). Coactivation of D1 and D2 dopamine receptors is required for long-term synaptic depression in the striatum. Neurosci. Letters, 142, 95-99.
40. Calabresi, P., Maj, R., Pisani, A., Mercuri, N. B., & Bernardi, G. (1992b). Long-term synaptic depression in the striatum: Physiological and pharmacological characterization. J. Neurosci., 12, 4224-4233.
41. Calabresi, P., Mercuri, N., Stanzione, P., Stefani, A., & Bernardi, G. (1987). Intracellular studies on the dopamine-induced firing inhibition of neostriatal neurons in vitro: Evidence for Dl receptor involvement. Neurosci., 20, 757-771.
42. Calabresi, P., Pisani, A., Mercuri, N. B., & Bernardi, G. (1992c). Long-term potentiation in striatum is unmasked by removing the voltage-dependent magnesium block of NMDA receptor channels. European J. Neurosci., 4, 929-935.
43. Calabresi, P., Pisani, A., Mercuri, N. B., & Bernardi, G. (1996). The corticostriatal projection: From synaptic plasticity to dysfunctions of the basal ganglia. Trends Neurosci., 19, 19-24.
44. Cannon, C. M., & Palmiter, R. D. (2003). Reward without dopamine. J. Neurosci, 23(34), 10827-10831.
45. Carpenter, G. A., & Grossberg, S. (1987). ART-2: Self-organization of stable category recognition codes for analog input pattern. Applied Optics, 26, 4919-4930.
46. Castellani, G. C., Quinlan, E. M., Cooper, L. N., & Shouval, H. Z. (2001). A biophysical model of bidirectional synaptic plasticity: Dependence on AMPA and NMDA receptors. Proc. Natl. Acad. Sci. (USA), 98(22), 12772-12777.
47. Cateau, H., Kitano, K., & Fukai, T. (2002). An accurate and widely applicable method to determine the distribution of synaptic strengths formed by the spike-timing-dependent learning. Neurocomputing, 44(46), 343-351.
48. Centonze, D., Gubellini, P., Picconi, B., Calabresi, P., Giacomini, P., & Bernardi, G. (1999). Unilateral dopamine denervation blocks corticostriatal LTP. J. Neurophysiol., 82, 3575-3579.
49. Centonze, D., Picconi, B., Gubellini, P., Bernardi, G., & Calabresi, P. (2001). Dopaminergic control of synaptic plasticity in the dorsal striatum. European J. Neurosci., 13, 1071-1077.
50. Charpier, S., & Deniau, J. M. (1997). In vivo activity-dependent plasticity at cortico-striatal connections: Evidence for physiological long-term potentiation. Proc. Natl. Acad. Sci. USA, 94, 7036-7040.
51. Charpier, S., Mahon, S., & Deniau, J. M. (1999). In vivo induction of striatal long-term potentiation by low-frequency stimulation of the cerebral cortex. Neurosci., 91, 1209-1222.
52. Choi, S., & Lovinger, D. M. (1997). Decreased probability of neuro-transmitter release underlies long-term depression and postnatal development of corticostriatal synapses. Proc. Natl. Acad. Sci. USA, 94, 2665-2670.
53. Contreras-Vidal, J. L., & Schultz, W. (1999). A predictive reinforcement model of dopamine neurons for learning approach behavior. J. Comput. Neurosci., 6, 191-214.
54. Daw, N. D. (2003). Reinforcement learning models of the dopamine system and their behavioral implications. Unpublished doctoral dissertation, Carnegie Mellon University, Pittsburgh, PA.
55. Dayan, P. (1992). The convergence of TD(X). Mach. Learn., 8(3/4), 341-362.
56. Dayan, P. (2002). Matters temporal. Trends Cogn. Sci., 6(3), 105-106.
57. Dayan, P., & Balleine, B. W. (2002). Reward, motivation, and reinforcement learning. Neuron, 36(2), 285-298.
58. Dayan, P., Kakade, S., & Montague, P. R. (2000). Learning and selective attention. Nature Neurosci., 3, 1218-1223.
59. Dayan, P., & Seynowski, T. (1994). TD(λ) converges with probability 1. Mach. Learn., 14, 295-301.
60. Debanne, D., Gahwiler, B. T., & Thompson, S. H. (1994). Asynchronous pre- and postsynaptic activity induces associative long-term depression in area CAI of the rat hippocampus in vitro. Proc. Natl. Acad. Sci. (USA), 91, 1148-1152.
61. Debanne, D., Gahwiler, B., & Thompson, S. (1998). Long-term synaptic plasticity between pairs of individual CA3 pyramidal cells in rat hippocampal slice cultures. J. Physiol (Lond.), 507, 237-247.
62. Deisseroth, K., Heist, E. K., & Tsien, R. W. (1998). Translocation of calmodulin to the nucleus supports CREB phosphorylation in hippocampal neurons. Nature, 392, 198-202.
63. DeKoninck, P., & Schulmann, H. (1998). Sensitivity of CaM kinase II to the frequency of Ca2+ oscillations. Science, 279, 227-230.
64. Delgado, M. R., Nystrom, L. E., Fissell, C., Noll, D. C., & Fiez, J. A. (2000). Tracking the hemodynamic responses to reward and punishment in the striatum. J. Neurophysiol., 84, 3072-3077.
65. Dennett, D. C. (1984). Cognitive wheels: The frame problem of AI. In C. Hookway (Eds.), Minds, machines and evolution (pp. 129-151). Cambridge: Cambridge University Press.
66. Dolmetsch, R. E., Pajvani, U., Fife, K., Spotts, J. M., & Greenberg, M. E. (2001). Signaling to the nucleus by an L-type calcium channel-calmodulin complex through the MAP kinase pathway. Science, 294, 333-339.
67. Doya, K. (2000). Reinforcement learning in continuous time and space. Neural Comp., 12(1), 219-245.
68. Egger, V., Feldmeyer, D., & Sakmann, B. (1999). Coincidence detection and changes of synaptic efficacy in spiny stellate neurons in rat barrel cortex. Nature Neurosci., 2, 1098-1105.
69. Elliott, R., Friston, K. J., & Dolan, R. J. (2000). Dissociable neural responses in human reward systems. J. Neurosci., 20, 6159-6165.
70. Eurich, C. W., Pawelzik, K., Ernst, U., Cowan, J. D., & Milton, J. G. (1999). Dynamics of self-organized delay adaptation. Phys. Rev. Lett., 82, 1594-1597.
71. Feldman, D. (2000). Timing-based LTP and LTD at vertical inputs to layer II/III pyramidal cells in rat barrel cortex. Neuron, 27, 45-56.
72. Froemke, R. C., & Dan, Y. (2002). Spike-timing-dependent synaptic modification induced by natural spike trains. Nature, 416, 433-438.
73. Fusi, S. (2002). Hebbian spike-driven synaptic plasticity for learning patterns of mean firing rates. Biol. Cybern., 87, 459-470.
74. Futatsugi, A., Kato, K., Ogura, H., Li, S. T., Nagata, E., Kuwajima, G., Tanaka, K., Itohara, S., & Mikoshiba, K. (1999). Facilitation of NMDAR-independent LTP and spatial learning in mutant mice lacking ryanodine receptor type 3. Neuron, 24, 701-713.
75. Gerfen, C. R. (1984). The neostriatal mosaic: Compartmentalization of corticostriatal input and striatonigral output systems. Nature, 311, 461-464.
76. Gerfen, C. R. (1985). The neostriatal mosaic. I. Compartmental organization of projections from the striatum to the substantia nigra in the rat. J. Comp. Neural, 236, 454-176.
77. Gerfen, C. R. (1992). The neostriatal mosaic: Multiple levels of compartmental organization in the basal ganglia. Annu. Rev. Neurosci., 15, 285-320.
78. Gerfen, C. R., Herkenham, M., & Thibault, J. (1987). The neostriatal mosaic: II. Patch- and matrix- directed mesostriatal dopaminergic and non-dopaminergic systems. J. Neurosci., 7, 3915-3934.
79. Gerstner, W., Kempter, R., van Hemmen, J. L., & Wagner, H. (1996). A neuronal learning rule for sub-millisecond temporal coding. Nature, 383, 76-78.
80. Gerstner, W., & Kistler, W. M. (2002). Mathematical formulations of Hebbian learning. Biol. Cybern., 87, 404-415.
81. Goldberg, J., Holthoff, K., & Yuste, R. (2002). A problem with Hebb and local spikes. Trends Neurosci., 25(9), 433-435.
82. Golding, N., Kath, W. L., & Spruston, N. (2001). Dichotomy of action-potential backpropagation in CA1 pyramidal neuron dendrites. J. Neurophysiol., 86, 2998-3010.
83. Golding, N. L., Staff, P. N., & Spruston, N. (2002). Dendritic spikes as a mechanism for cooperative long-term potentiation. Nature, 418, 326-331.
84. Gormezano, I., Kehoe, E. J., & Marshall, B. S. (1983). Twenty years of classical conditioning research with the rabbit. In J. M. Sprague, & A. N. Epstein, (Eds.), Progress of psychobiology and physiological psychology (pp. 198-274). Orlando, FL: Academic Press.
85. Graef, I. A., Mermelstein, P. G., Stankunas, K., Neilson, J. R., Deisseroth, K., Tsien, R. W., & Crabtree, G. R. (1999). L-type calcium channels and GSK-3 regulate the activity of NF-ATc4 in hippocampal neurons. Nature, 401, 703-708.
86. Greengard, P., Allen, P. B., & Nairn, A. C. (1999). Beyond the dopamine receptor: The DARPP-32/protein phosphatase-1 cascade. Neuron, 23, 435-447.
87. Groenewegen, H. J., Berendse, H. W., Wolters, J. G., & Lohman, A. H. M. (1990). The anatomical relationship of the prefrontal cortex with the striatopallidal system, the thalamus and the amygdala: Evidence for a parallel organization. Prog. Brain Res., 85, 95-118.
88. Grossberg, S. (1995). A spectral network model of pitch perception. J. Acoust. Soc. Am., 98(2), 862-879.
89. Grossberg, S., & Merrill, J. (1996). The hippocampus and cerebellum in adaptively timed learning, recognition and movement. J. Cogn. Neurosci., 8, 257-277.
90. Grossberg, S., & Schmajuk, N. (1989). Neural dynamics of adaptive timing and temporal discrimination during associative learning. Neural Networks, 2, 79-102.
91. Groves, P. M., Garcia-Munoz, M., Linder, J. C., Manley, M. S., Martone, M. E., & Young, S. J. (1995). Elements of the intrinsic organization and information processing in the neostriatum. In J. C. Houk, J. L. Davis, & D. G., Beiser, (Eds.), Models of information processing in the basal ganglia (pp. 51-96). Cambridge, MA: MIT Press.
92. Gruber, A. J., Solla, S. A., Surmeier, D. J., & Houk, J. C. (2003). Modulation of striatal single units by expected reward: A spiny neuron model displaying dopamine-induced bistability. J. Neurophysiol., 90, 1095-1114.
93. Gustafsson, B., Wigstrom, H., Abraham, W. C., & Huang, Y.-Y. (1987). Long-term potentiation in the hippocampus using depolarizing current pulses as the conditioning stimulus to single volley synaptic potentials. J. Neurosci., 7, 774-780.
94. Gütig, R., Aharonov, R., Rotter, S., & Sompolinsky, H. (2003). Learning input correlations through nonlinear temporally asymmetric Hebbian plasticity. J. Neurosci., 23(9), 3697-3714.
95. Haber, S. N., Fudge, J. L., & McFarland, N. R. (2000). Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. J. Neurosci., 20, 2369-2382.
96. Han, V. Z., Grant, K., & Bell, C. C. (2000). Reversible associative depression and nonassociative potentiation at a parallel fiber synapse. Neuron, 27, 611-622.
97. Hassani, O. K., Cromwell, H. C., & Schultz, W. (2001). Influence of expectation of different rewards on behavior-related neuronal activity in the striatum. J. Neurophysiol., 85, 2477-2489.
98. Hauber, W., Bohn, I., & Giertler, C. (2000). NMDA, but not dopamine D2, receptors in the rat nucleus accumbens are involved in guidance of instrumental behavior by stimuli predicting reward magnitude. J. Neurosci., 20(16), 6282-6288.
99. Häusser, M., Spruston, N., & Stuart, G. J. (2000). Diversity and dynamics of dendritic signaling. Science, 11, 739-744.
100. Hebb, D. O. (1949). The organization of behavior: A neuropsychological study. New York: Wiley-Interscience.
101. Hemmings, H. C., J., Greengard, P., Tung, H. Y., & Cohen, P. (1984). DARPP-32, a dopamine-regulated neuronal phosphoprotein, is a potent inhibitor of protein phosphatase-1. Nature, 310, 503-505.
102. 2+ conductance. J. Neurosci., 17, 3334-3342.
103. + channel regulation of signal propagation in dendrites of hippocampal pyramidal neurons. Nature, 387, 869-875.
104. Hollerman, J. R., & Schultz, W. (1998). Dopamine neurons report an error in the temporal prediction of reward during learning. Nature Neurosci., 1(4), 304-309.
105. Hollerman, J. R., Tremblay, L., & Schultz, W. (1998). Influence of reward expectation on behavior-related neuronal activity in primate striatum. J. Neurophysiol, 80, 947-963.
106. Houk, J. C., Adams, J. L., & Barto, A. G. (1995). A model of how the basal ganglia generate and use neural signals that predict reinforcement. In J. C. Houk, J. L. Davis, & D. G., Beiser (Eds.), Models of information processing in the basal ganglia (pp. 249-270). Cambridge, MA: MIT Press.
107. Hull, C. L. (1939). The problem of stimulus equivalence in behavior theory. Psychological Review, 46, 9-30.
108. Hull, C. L. (1943). Principles of behavior. New York: Appleton-Century-Crofts. Izhikevich, E. M., & Desai, N. S. (2003). Relating STDP to BCM. Neural Comp., 15, 1511-1523.
109. Hull, C. L. (1943). Principles of behavior. New York: Appleton-Century-Crofts. Izhikevich, E. M., & Desai, N. S. (2003). Relating STDP to BCM. Neural Comp., 15, 1511-1523.
110. Jaakkola, T., Jordan, M. I., & Singh, S. P. (1995). On the convergence of stochastic iterative dynamic programming algorithms. Neural Comp., 6(6), 1185-1201.
111. Joel, D., Niv, Y., & Ruppin, E. (2002). Actor-critic models of the basal ganglia: New anatomical and computational perspectives. Neural Networks, 15, 535-547.
112. Joel, D., & Weiner, I. (2000). The connections of the dopaminergic system with the striatum in rats and primates: An analysis with respect to the functional and compartmental organization of the striatum. Neurosci., 96, 451-474.
113. Johnston, D., Magee, J. C., Colbert, C. M., & Cristie, B. R. (1996). Active properties of neuronal dendrites. Annu. Rev. Neurosci., 19, 165-186.
114. Joyce, J. N., & Marshall, J. F. (1987). Quantitative autoradiography of dopamine D2 sites in rat caudate-putamen: Localization to intrinsic neurons and not to neocortical afferents. Neurosci., 20, 773-795.
115. Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996). Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4, 237-285.
116. Karmarkar, U. R., & Buonomano, D. V. (2002). A model of spike-timing dependent plasticity: One or two coincidence detectors? J. Neurophysiol., 88, 507-513.
117. Karmarkar, U. R., Najarian, M. T., & Buonomano, D. V. (2002). Mechanisms and significance of spike-timing dependent plasticity. Biol. Cybern., 87, 373-382.
118. Kawagoe, R., Takikawa, Y., & Hikosaka, O. (1998). Expectation of reward modulates cognitive signals in the basal ganglia. Nat. Neurosci., 1, 411-416.
119. Kempter, R., Gerstner, W., & van Hemmen, J. L. (1999). Hebbian learning and spiking neurons. Phys. Rev. E., 59, 4498-4515.
120. Kempter, R., Gerstner, W., & van Hemmen J. L. (2001a). Intrinsic stabilization of output rates by spike-based Hebbian learning. Neural Comp., 13, 2709-2741.
121. Kempter, R., Leibold, C., Wagner, H., & van Hemmen, J. L. (2001b). Formation of temporal-feature maps by axonal propagation of synaptic learning. Proc. Natl. Acad. Sci. (USA), 98(7), 4166-4171.
122. King, M. M., Huang, C. Y., Chock, P. B., Nairn, A. C., Hemmings, H. C., J., Chan, K. F., & Greengard, P. (1984). Mammalian brain phosphoproteins as substrates for calcineurin. J. Biol. Chem., 259, 8080-8083.
123. Kistler, W. M. (2002). Spike-timing dependent synaptic plasticity: A phenomenological framework. Biol. Cybern., 87, 416-427.
124. Kistler, W. M., & van Hemmen, J. L. (2000). Modeling synaptic plasticity in conjunction with the timing of pre- and post-synaptic action potentials. Neural Comput., 12, 385-405.
125. Kitano, K., Okamoto, H., & Fukai, T. (2003). Time representing cortical activities: Two models inspired by prefrontal persistent activity. Biol. Cybern, 88(5), 387-394.
126. Klopf, A. H. (1972). Brain function and adaptive systems - a heterostatic theory. (Tech. Rep. Air Force Cambridge Research Laboratories Special Report No. 133), Alexandria, VA: Defense Technical Information Center, Cameron Station.
127. Klopf, A. H. (1982). The hedonistic neuron: A theory of memory, learning, and intelligence. Washington, DC: Hemisphere.
128. Klopf, A. H. (1986). A drive-reinforcement model of single neuron function. In J. S. Denker (Eds.), Neural networks for computing: AIP Conf. Proc., New York: American Institute of Physics.
129. Klopf, A. H. (1988). A neuronal model of classical conditioning. Psychobiol., 16(2), 85-123.
130. Knutson, B., Fong, G. W., Adams, C. M., Varner, J. L., & Hommer, D. (2001). Dissociation of reward anticipation and outcome with event-related fMRI. Neuroreport, 12, 3683-3687.
131. Koester, H. J., & Sakmann, B. (1998). Calcium dynamics in single spines during coincident pre- and postsynaptic activity depend on relative timing of back-propagating action potentials and subthreshold excitatory postsynaptic potentials. Proc. Natl. Acad. Sci. USA, 95, 9596-9601.
132. Kosco, B. (1986). Differential Hebbian learning. In J. S. Denker, (Ed.), Neural networks for computing: AIP Conference Proc. New York: American Institute of Physics.
133. Kurata, K., & Wise, S. P. (1988). Premotor and supplementary motor cortex in rhesus monkeys: Neuronal activity during externally- and internally-instructed motor tasks. Exp. Brain Res., 72, 237-248.
134. Lasser-Ross, N., & Ross, W. N. (1992). Imaging voltage and synaptically activated sodium transients in cerebellar Purkinje cells. Proc. R. Soc. B. Biol. Sci., 247, 35-39.
135. Leibold, C., Kempter, R., & van Hemmen, J. L. (2001). Temporal map formation in the barn owl's brain. Phys. Rev. Lett., 87(24), 248101-1-248101-4.
136. Leibold, C., & van Hemmen, J. L. (2001). Temporal receptive fields, spikes, and Hebbian delay selection. Neural Networks, 24(6-7), 805-813.
137. Leibold, C., & van Hemmen, J. L. (2002). Mapping time. Biol. Cybern., 87, 428-439.
138. Levy, N., Horn, D., Meilijson, I., & Ruppin, E. (2001). Distributed synchrony in a cell assembly of spiking neurons. Neural Networks, 24, 815-824.
139. Levy, W. B., & Steward, O. (1983). Temporal contiguity requirements for long-term associative potentiation/depression in the hippocampus. Neurosci., 8, 791-797.
140. Linden, D. J. (1999). The return of the spike: Postsynaptic action potentials and the induction of LTP and LTD. Neuron, 22, 661-666.
141. Lisman, J. (1989). A mechanism for the Hebb and the anti-Hebb processes underlying learning and memory. Proc. Natl. Acad. Sci. USA, 86, 9574-9578.
142. Ljungberg, T., Apicella, P., & Schultz, W. (1992). Responses of monkey dopamine neurons during learning of behavioral reactions. J. Neurophysiol., 67, 145-163.
143. Lovinger, D. M., Tyler, E. C., & Merritt, A. (1993). Short- and long-term synaptic depression in rat neostratium. J. Neurophysiol., 70, 1937-1949.
144. Lüscher, C., & Frerking, M. (2001). Restless AMPA receptors: Implications for synaptic transmission and plasticity. Trends Neurosci., 24(11), 665-670.
145. Lynd-Balta, E., & Haber, S. N. (1994). Primate striatonigral projections: A comparison of the sensorimotor-related striatum and the ventral striatum. J. Comp. Neurol., 345, 562-578.
146. Mackintosh, N. J. (1974). The psychology of animal learning. Orlando, FL: Academic Press.
147. Mackintosh, N. J. (1983). Conditioning and associative learning. New York: Oxford University Press.
148. Magee, J. C., & Johnston, D. (1997). A synaptically controlled, associative signal for Hebbian plasticity in hippocampal neurons. Science, 275, 209-213.
149. Malenka, R. C., Kauer, J. A., Perkel, D. J., Mauk, M. D., Kelly, P. T., Nicoll, R. A., & Waxham, M. N. (1989). An essential role for postsynaptic calmodulin and protein kinase activity in long-term potentiation. Nature, 340, 554-557.
150. Malenka, R. C., & Nicoll, R. A. (1999). Long-term potentiation - a decade of progress? Science, 285, 1870-1874.
151. Malinow, R., Schulman, H., & Tsien, R. W. (1989). Inhibition of postsynaptic PKC or CaMKII blocks induction but not expression of LTP. Science, 245, 862-866.
152. Markram, H., Lübke, J., Frotscher, M., & Sakmann, B. (1997). Regulation of synaptic efficacy by coincidence of postsynaptic APs and EPSPs. Science, 275, 213-215.
153. Martin, S. J., Grimwood, P. D., & Morris, R. G. M. (2000). Synaptic plasticity and memory: An evaluation of the hypothesis. Annu. Rev. Neurosci., 23, 649-711.
154. Martinez, J. L., & Derrick, B. E. (1996). Long-term potentiation and learning. Annu. Rev. Psychol., 47, 173-203.
155. 2+ of NMDA responses in spinal cord neurones. Nature, 309, 261-263.
156. McFarland, D. J. (1989). Problems of animal behavior. Harlow: Longman Scientific & Technical.
157. Mehta, M. R. (2004). Cooperative LTP can map memory sequences on dendritic branches. TINS, 27, 69-72.
158. Mehta, M. R., Lee, A. K., & Wilson, M. A. (2002). Role of experience and oscillations in transforming a rate code into a temporal code. Nature, 417(6890), 741-746.
159. Melamed, O., Gerstner, W., Maass, W., Tsodyks, M., & Markram, H. (2004). Coding and learning of behavioral sequences. Trends Neurosci, 27(1), 11-14.
160. Memo, M., Lovenberg, W., & Hanbauer, I. (1982). Agonist-induced subsensitivity of adenylate cyclase coupled with a dopamine receptor in slices from rat corpus striatum. Proc. Natl. Acad. Sci. USA, 79, 4456-4460.
161. + conductance in the back-propagation of action potentials in the dendrites of hippocampal pyramidal neurons. J. Comput. Neurosci., 7, 5-15.
162. Miller, J. D., Sanghera, M. K., & German, D. C. (1981). Mesencephalic dopaminergic unit activity in the behaviorally conditioned rat. Life Sci., 29, 1255-1263.
163. Mirenowicz, J., & Schultz, W. (1994). Importance of unpredictability for reward responses in primate dopamine neurons. J. Neurophysiol., 72(2), 1024-1027.
164. Montague, P. R., Dayan, P., Person, C., & Sejnowski, T. J. (1995). Bee foraging in uncertain environments using predictive Hebbian learning. Nature, 377, 725-728.
165. Montague, P. R., Dayan, P., & Sejnowski, T. J. (1996). A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci., 16(5), 1936-1947.
166. Morris, B. J., Simpson, C. S., Mundell, S., Maceachern, K., Johnston, H. M., & Nolan, A. M. (1997). Dynamic changes in NADPH-diaphorase staining reflect activity of nitric oxide synthase: Evidence for a dopaminergic regulation of striatal nitric oxide release. Neuropharmacology, 36, 1589-1599.
167. Mulkey, R. M., Endo, S., Shenolikar, S., & Malenka, R. C. (1994). Involvement of a calcineurin/inhibitor-1 phosphatase cascade in hippocampal long-term depression. Nature, 369, 486-488.
168. Nicola, S. M., Surmeier, J., & Malenka, R. C. (2000). Dopaminergic modulation of neuronal excitability in the striatum and nucleus accumbens. Ann. Rev. Neurosci., 23, 185-215.
169. Nishi, A., Snyder, G. L., & Greengard, P. (1997). Bidirectional regulation of DARPP-32 phosphorylation by dopamine. J. Neurosci., 17, 8147-8155.
170. Nishiyama, M., Hong, K., Mikoshiba, K., Poo, M., & Kato, K. (2000). Calcium release from internal stores regulates polarity and input specificity of synaptic modification. Nature, 408, 584-588.
171. Nobre, A. C., Coull, J. T., Frith, C. D., & Mesulam, M. M. (1999). Orbitofrontal cortex is activated during breaches of expectation in tasks of visual attention. Nature Neurosci., 2, 11-12.
172. Nowak, L., Bregestovski, P., Ascher, P., Herbet, A., & Prochiantz, A. (1984). Magnesium gates glutamate-activated channels in mouse central neurones. Nature, 307, 462-465.
173. O'Doherty, J., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003). Temporal difference models and reward-related learning in the human brain. Neuron, 28, 329-337.
174. O'Doherty, J., Deichman, R., Critchley, H. D., & Dolan, R. J. (2002). Neural responses during anticipation of primary taste reward. Neuron, 33, 815-826.
175. O'Doherty, J., Rolls, E. T., Francis, S., Bowtell, R., & McGlone, F. (2001). Representation of pleasant and aversive taste in the human brain. J. Neurophysiol., 85, 1315-1321.
176. Overton, P. G., & Clark, D. (1997). Burst firing in midbrain dopaminergic neurons. Brain Res. Rev., 25, 312-334.
177. +-sensitive conductance in neostriatal neurons. Exp. Brain Res., 110, 205-211.
178. Pagoni, G., Zink, C. F., Montague, P. R., & Berns, G. S. (2002). Activity in human ventral striatum locked to errors of reward prediction. Nature Neurosci., 5, 97-98.
179. Panchev, C., Wermter, S., & Chen, H. (2002). Spike-timing dependent competitive learning of integrate-and-fire neurons with active dendrites. In Lecture Notes in Computer Science. Proc. Int. Conf. Artificial Neural Networks (pp. 896-901). Berlin: Springer.
180. Parent, A. (1990). Extrinsic connections of the basal ganglia. Trends Neurosci., 13, 254-258.
181. Partridge, J. G., Tang, K. C., & Lovinger, D. M. (2000). Regional and postnatal heterogeneity of activity-dependent long-term changes in synaptic efficacy in the dorsal striatum. J. Neurophysiol, 84, 1422-1429.
182. Pavlov, P. I. (1927). Conditioned reflexes. New York: Oxford University Press.
183. Peng, J. (1993). Efficient dynamic programming-based learning for control. Unpublished doctoral dissertation, Northeastern University.
184. Pennartz, C. M., Ameerun, R. F., Groenewegen, H. J., & Lopes da Silva, F. H. (1993). Synaptic plasticity in an in vitro slice preparation of the rat nucleus accumbens. European J. Neurosci., 5, 107-117.
185. Pike, F. G., Meredith, R. M., Olding, A. A., & Paulsen, O. (1999). Postsynaptic bursting is essential for "Hebbian" induction of associative long-term potentiation at excitatory synapses in rat hippocampus. J. Physiol. (Lond.), 518, 571-576.
186. Pollack, J. B., & Blair, A. D. (1997). Why did TD-Gammon work? In M. Mozer, M. Jordan, & T. Petsche (Eds.), Advances in neural information processing systems, (pp. 10-16). Cambrdige, MA: MIT Press.
187. Porr, B., von Ferber, C., & Wörgötter, F. (2003). ISO-learning approximates a solution to the inverse-controller problem in an unsupervised behavioral paradigm. Neural Comp., 15, 865-884.
188. Porr, B., & Wörgötter, F. (2002). Isotropic sequence order learning using a novel linear algorithm in a closed loop behavioural system. Biosystems, 67(1-3), 195-202.
189. Porr, B., & Wörgötter, F. (2003a). Isotropic sequence order learning. Neural Comp., 15, 831-864.
190. Porr, B., & Wörgötter, F. (2003b). Isotropic sequence order learning in a closed loop behavioural system. Proc. Roy. Soc. B.
191. Porr, B., & Wörgötter, F. (2003c). Isotrpic sequence order learning in a closed loop behavioural system. Roy. Soc. Phil. Trans. Mathematical, Physical, & Engineering Sciences, 361, 2225-2244.
192. Prokasy, W. F., Hall, J. F., & Fawcett, J. T. (1962). Adaptation, sensitization, forward and backward conditioning, and pseudo-conditioning of the GSR. Psychol. Rep., 10, 103-106.
193. Pucak, M. L., & Grace, A. A. (1994). Regulation of substantia nigra dopamine neurons. Crit. Rev. Neurobiol., 9, 67-89
194. Rao, R. P. N., & Sejnowski, T. J. (2001). Spike-timing-dependent Hebbian plasticity as temporal difference learning. Neural Comp., 13, 2221-2237.
195. Redgrave, P., Prescott, T. J., & Gurney, K. (1999). Is the short-latency dopamine response too short to signal reward? Trends Neurosci., 22, 146-151.
196. Regehr, W. G., Konnerth, A., & Armstrong, C. M. (1992). Sodium action potentials in the dendrites of cerebellar Purkinje cells. Proc. Natl. Acad. Sci. USA, 89, 5492-5496.
197. Rescorla, R. A., & Wagner, A. R. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In A. H. Black & W. F. Prokasy (Eds.), Classical conditioning II: Current research and theory. New York: Appleton Century Crofts.
198. Reynolds, J. N. J., & Wickens, J. R. (2002). Dopamine-dependent plasticity of corticostriatal synapses. Neural Networks, 15, 507-521.
199. Reynolds, S. I. (2002). Reinforcement learning with exploration. Unpublished doctoral dissertation, University of Birmingham.
200. Roberts, P. D. (1999). Computational consequences of temporally asymmetric learning rules: I. Differential Hebbian learning. J. Comput. Neurosci., 7(3), 235-246.
201. Roberts, P. D., & Bell, C. C. (2002). Spike timing dependent synaptic plasticity in biological systems. Biol. Cybern., 87, 392-403.
202. Roche, K. W., O'Brian, R. J., Mammen, A. L., Bernhardt, J., & Huganir, R. L. (1996). Characterization of multiple phosphorylation sites on the AMPA receptor GluR1 subunit. Neuron, 16, 1179-1188.
203. Rose, C. R., & Konnerth, A. (2001). Stores not just for storage, intracellular calcium release and synaptic plasticity. Neuron, 31, 519-522.
204. Rubin, J., Lee, D. D., & Sompolinsky, H. (2001). Equilibrium properties of temporally asymmetric Hebbian plasticity. Phys. Rev. Lett., 86(2), 364-367.
205. Rummery, G. A. (1995). Problem solving with reinforcement learning. Unpublished doctoral dissertation, Cambridge University.
206. Russell, V. A., Allin, R., Lamm, M. C., & Taljaard, J. J. (1992). Regional distribution of monoamines and dopamine D1- and D2-receptors in the striatum of the rat. Neurochemical Res., 17, 387-395.
207. Santos, J. M., & Touzet, C. (1999a). Dynamic update of the reinforcement function during learning. Connection Science [Special issue], 11 (3-4).
208. Santos, J. M. & Touzet, C. (1999b). Exploration tuned reinforcement function. Neurocomputing [Special issue], 28(1-3), 93-105.
209. Sato, N., & Yamaguchi, Y. (2003). Memory encoding by theta phase precession in the hippocampal network. Neural Computation, 15, 2379-2397.
210. Saudargiene, A., Porr, B., & Wörgötter, F. (2004). How the shape of pre- and postsynaptic signals can influence STDP: A biophysical model. Neural Comp., 16, 595-626.
211. Schiller, J., Schiller, Y., Stuart, G., & Sakmann, B. (1997). Calcium action potentials restricted to distal apical dendrites of rat neocortical pyramidal neurons. J. Physiol., 505, 605-616.
212. Schoenbaum, G., Chiba, A. A., & Gallagher, M. (1998). Orbitofrontal cortex and basolateral amygdala encode expected outcomes during learning. Nature Neurosci., 1, 155-159.
213. Schultz, W. (1986). Responses of midbrain dopamine neurons to behavioral trigger stimuli in the monkey. J. Neurophysiol., 56, 1439-1462.
214. Schultz, W. (1998). Predictive reward signal of dopamine neurons. J. Neurophysiol, 80, 1-27.
215. Schultz, W. (2002). Getting formal with dopamine and reward. Neuron, 36, 241-263.
216. Schultz, W., Apiccela, P., Scarnati, E., & Ljungberg, T. (1992). Neuronal activity in monkey ventral striatum related to the expectation of reward. J. Neurosci., 12, 4595-4610.
217. Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275, 1593-1599.
218. Schultz, W., & Dickinson, A. (2000). Neuronal coding of prediction errors. Annu. Rev. Neurosci., 23, 473-500.
219. Senn, W. & Buchs, N. J. (2003). Spike-based synaptic plasticity and the emergence of direction selective simple cells: Mathematical analysis. J. Comput. Neurosci., 14, 119-138.
220. Senn, W., Markram, H., & Tsodyks, M. (2000). An algorithm for modifying neurotransmitter release probability based on pre-and postsynaptic spike timing. Neural Comp., 13, 35-67.
221. Seung, H. S. (1998). Learning continous attractors in recurrent networks. In M. Kearns, M. Jordan, & S. Solla, (Eds.), Advances in neural information processing systems, 10 (pp. 654-660). Cambridge, MA: MIT Press.
222. Shouval, H. Z., Bear, M. F., & Cooper, L. N. (2002). A unified model of NMDA receptor-dependent bidirectional synaptic plasticity. Proc. Natl. Acad. Sci. (USA), 99(16), 10831-10836.
223. Sibley, D. R., & Monsma, F. J. J. (1992). Molecular biology of dopamine receptors. Trends Pharmakol. Sci., 13, 61-69.
224. Singh, S. P., & Sutton, R. S. (1996). Reinforcement learning with replacing eligibility traces. Mach. Learn., 22, 123-158.
225. Sjöström, P. J., Turrigiano, G. G., & Nelson, S. B. (2001). Rate, timing, and cooperativity jointly determine cortical synaptic plasticity. Neuron, 32, 1149-1164.
226. Song, I., & Huganir, R. L. (2002). Regulation of AMPA receptors during synaptic plasticity. Trends Neurosci., 25(11), 578-588.
227. Song, S., & Abbott, L. F. (2001). Cortical development and remapping through spike timing-dependent plasticity. Neuron, 32, 1-20.
228. Song, S., Miller, K. D., & Abbott, L. F. (2000). Competitive Hebbian learning through spike-timing-dependent synaptic plasticity. Nature Neurosci., 3, 919-926.
229. Spencer, J. P., & Murphy, K. P. (2000). Bi-directional changes in synaptic plasticity induced at corticostriatal synapses in vitro. Exp. Brain Res., 135, 497-503.
230. Sterratt, D. C., & van Ooyen, A. (2002). Does morphology influence temporal plasticity? In J. R. Dorronsoro, (Ed.), ICANN 2002 (Vol. LNCS 2415, pp. 186-191). Berlin: Springer-Verlag.
231. Stoof, J. C., & Kebabian, J. W. (1981). Opposing roles for D-1 and D-2 dopamine receptors in efflux of cyclic AMP from rat neostriatum. Nature, 294, 366-368.
232. Stuart, G. J., & Sakmann, B. (1994). Active propagation of somatic action potentials into neocortical pyramidal cell dendrites. Nature, 367, 69-72.
233. Stuart, G., Spruston, N., Sakmann, B., & Häusser, M. (1997). Action potential initiation and backpropagation in neurons of the mammalian central nervous system. Trends Neurosci., 20, 125-131.
234. Suri, R. E., Bargas, J., & Arbib, M. A. (2001). Modeling functions of striatal dopamine modulation in learning and planning. Neurosci., 103(1), 65-85.
235. Suri, R. E., & Schultz, W. (1998). Learning of sequential movements by neural network model with dopamine-like reinforcement signal. Exp. Brain Res., 121, 350-354.
236. Suri, R. E., & Schultz, W. (1999). A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task. Neurosci., 91 (3), 871-890.
237. Suri, R. E., & Schultz, W. (2001). Temporal difference model reproduces anticipatory neural activity. Neural Comp., 23(4), 841-862.
238. Surmeier, D. J., Bargas, J., Hemmings, H. C., J., Nairn, A. C., & Greengard, P. (1995). Modulation of calcium currents by a D1 dopaminergic protein kinase/phosphatase cascade in rat neostriatal neurons. Neuron, 14, 385-397.
239. Surmeier, D. J., & Kitai, S. T. (1993). D1 and D2 dopamine receptor modulation of sodium and potassium currents in rat neostriatal neurons. Prog. Brain Res., 99, 309-324.
240. Sutton, R. S. (1984). Temporal credit assignment in reinforcement learning. Unpublished doctoral dissertation, University of Massachusetts.
241. Sutton, R. S. (1988). Learning to predict by the methods of temporal differences. Mach. Learn, 3, 9-44.
242. Sutton, R. S. (1996). Generalization in reinforcement learning: Successful examples using sparse coarse coding. In D. S. Touretzky, M. C. Mozer, & M. E. Hasselmo (Eds.), Advances in neural information processing systems, 8 (pp. 1038-1044). Cambridge, MA: MIT Press.
243. Sutton, R. S. (1999). Open theoretical questions in reinforcement learning. In EUROColt, (pp. 11-17).
244. Sutton, R., & Barto, A. (1981). Towards a modern theory of adaptive networks: Expectation and prediction. Psychol. Review, 88, 135-170.
245. Sutton, R. S., & Barto, A. G. (1990). Time-derivative models of Pavlovian reinforcement. In M. Gabriel, & J. Moore (Eds.), Learning and computational neuroscience: Foundation of adaptive networks. Cambridge, MA: MIT Press.
246. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
247. Svenningsson, P., Lindskog, M., Rognoni, F., Fredholm, B. B., Greengard, P., & Fisone, G. (1998). Activation of adenosine A2A and dopamine Dl receptors stimulates cyclic AMP-dependent phosphorylation of DARPP-32 in distinct populations of striatal projection neurons. Neurosci., 84, 223-228.
248. 2+: intracellular stores spill their guts. Neuron, 22, 427-430.
249. Tang, K., Low, M. J., Grandy, D. K., & Lovinger, D. M. (2001). Dopamine-dependent synaptic plasticity in striatum during in vivo development. Proc. Natl. Acad. Sci. USA, 98, 1255-1260.
250. Tesauro, G. (1992). Practical issues in temporal difference learning. Mach. Learn., 8, 257-277.
251. Tingley, W. G., Ehlers, M. D., Kameyama, K., Doherty, C., Ptak, J. B., Riley, C. T., & Huganir, R. L. (1997). Characterization of the protein kinase A and protein kinase C phosphorylation of the N-methyl-D-aspartate receptor NR1 subunit using phosphorylation site-specific antibodies. J. Biol. Chem., 272, 5157-5166.
252. Toan, D. L., & Schultz, W. (1985). Responses of rat pallidum cells to cortex stimulation and effects of altered dopaminergic activity. Neurosci., 15, 683-694.
253. Touzet, C. (1999). Neural networks and Q-learning for robotics. Available online at: http://www.sciences-cognitives.org/scico/annuaire/Touzet_Claude/ Touzet_IJCNN_Tut.pdf.
254. Tremblay, L., Hollerman, J. R., & Schultz, W. (1998). Modifications of reward expectation-related neuronal activity during learning in primate striatum. J. Neurophysiol., 80, 964-977.
255. Tremblay, L., & Schultz, W. (1999). Relative reward preference in primate orbitofrontal cortex. Nature, 398, 704-708.
256. Umemiya, M., & Raymond, L. A. (1997). Dopaminergic modulation of excitatory postsynaptic currents in rat neostriatal neurons. J. Neurophysiol., 78, 1248-1255.
257. van Hemmen, J. L. (2001). Theory of synaptic plasticity. In F. Moss, & S. Gielen, (Eds.), Handbook of biological physics, neuro-informatics, neural modelling (Vol. 4, pp. 771-823. Amsterdam: Elsevier.
258. van Rossum, M. C. W., Bi, G. Q., & Turrigiano, G. G. (2000). Stable Hebbian learning from spike timing-dependent plasticity. J. Neurosci., 20(23), 8812-8821.
259. Walsh, J. P. (1993). Depression of excitatory synaptic input in rat striatal neurons. Brain Res., 608, 123-128.
260. Walsh, J. P., & Dunia, R. (1993). Synaptic activation of N-methyl-D-aspartate receptors induces short-term potentiation at excitatory synapses in the striatum of the rat. Neurosci., 57, 241-248.
261. Watkins, C. J. C. H. (1989). Learning from delayed rewards. Unpublished doctoral dissertation, Cambridge University.
262. Watkins, C. J. C. H., & Dayan, P. (1992). Technical note: Q-learning. Mach. Learn., 8, 279-292.
263. West, A. E., Chen, W. G., Dalva, M. B., Dolmetsch, R. E., Kornhauser, J. M., Shaywitz, A. J., Takasu, M. A., Tao, X., & Greenberg, M. E. (2001). Calcium regulation of neuronal gene expression. Proc. Natl. Acad. Sci. USA, 98, 11024-11031.
264. Wickens, J. R., Begg, A. J., & Arbuthnott, G. W. (1996). Dopamine reverses the depression of rat corticostriatal synapses which normally follows high-frequency stimulation of cortex in vitro. Neurosci., 70, 1-5.
265. Widrow, B., & Hoff, M. E. (1960). Adaptive switching circuits. In IRE WESCON Convention Record (Vol. 4, pp. 96-104). New York.
266. Wiering, M., & Schmidhuber, J. (1998). Fast online Q(λ). Mach. Learn., 33, 105-115.
267. Williams, S. M., & Goldman-Rakic, P. S. (1993). Characterization of the dopaminergic innervation of the primate frontal cortex using a dopamine-specific antibody. Cereb. Cortex, 3, 199-222.
268. Witten, I. H. (1977). An adaptive optimal controller for discrete-time Markov environments. Information and Control, 34, 86-295.
269. Xie, X., & Seung, S. (2000). Spike-based learning rules and stabilization of persistent neural activity. In S. A. Solla, T. K. Leen, & K. R. Müller, (Eds.), Advances in neural information processing systems, 12 (pp. 199-208). Cambridge, MA: MIT Press.
270. Yamamoto, Y., Nakanishi, H., Takai, N., Shimazoe, T., Watanabe, S., & Kita, H. (1999). Expression of N-methyl-D-aspartate receptor-dependent long-term potentiation in the neostriatal neurons in an in vitro slice after ethanol withdrawal of the rat. Neurosci., 91, 59-68.
271. 2+ elevation. J. Neurophysiol., 81, 781-787.
272. Yao, H., & Dan, Y. (2001). Stimulus timing-dependent plasticity in cortical processing of orientation. Neuron, 32, 315-323.
273. Yim, C. Y., & Mogenson, G. J. (1982). Response of nucleus accumbens neurons to amygdala stimulation and its modification by dopamine. Brain Res., 239, 401-415.
274. Zhang, L. I., Tao, H. W., Holt, C. E., Harris, W. A., & Poo, M.-M. (1998). A critical window for cooperation and competition among developing retinotectal synapses. Nature, 395, 37-44.
275. Zink, C. F., Pagnoni, G., Martin, M. E., Dhamala, M., & Berns, G. S. (2003). Human striatal response to salient nonrewarding stimuli. J. Neurosci., 23(22), 8092-8097.
276. Zucker, R. S. (1999). Calcium- and activity-dependent synaptic plasticity. Curr. Opin. Neurobiol., 9, 305-313.