Neuromodulated spike-timing-dependent plasticity, and theory of three-factor learning rules

Frontiers in Neural Circuits

Volumn 9, Issue JAN2016, 2015, Pages

  Frémaux, Nicolas ^a   Gerstner, Wulfram ^a  

^a EPFL   (Switzerland)

Author keywords

Neuromodulation; Novelty; Plasticity; Reward learning; Spiking neuron networks; STDP; Synaptic plasticity (LTP LTD)

Indexed keywords

CONDITIONED REFLEX; EXPERIMENTAL MODEL; JOINT; LEARNING; MATHEMATICS; MEMORY; MODEL; NEGLECT; NERVE CELL PLASTICITY; NEUROMODULATION; PLASTICITY; REWARD; SENSORY STIMULATION; SPIKE; ACTION POTENTIAL; ANIMAL; BIOLOGICAL MODEL; HUMAN; NERVE CELL; PHYSIOLOGY;

ACTION POTENTIALS; ANIMALS; HUMANS; LEARNING; MODELS, NEUROLOGICAL; NEURONAL PLASTICITY; NEURONS;

EID: 84957873806     PISSN: 16625110     EISSN: None     Source Type: Journal    
DOI: 10.3389/fncir.2015.00085     Document Type: Review

References (182)

1. Abbott, L. F., and Nelson, S. B. (2000). Synaptic plastictiy-taming the beast. Nat. Neurosci. 3, 1178-1183. doi: 10.1038/81453
2. Apicella, P., Ljungberg, T., Scarnati, E., and Schultz, W. (1991). Responses to reward in monkey dorsal and ventral striatum. Exp. Brain Res. 85, 491-500. doi: 10.1007/BF00231732
3. Arleo, A., and Gerstner, W. (2000). Spatial cognition and neuro-mimetic navigation: a model of hippocampal place cell activity. Biol. Cybern. 83, 287-299. doi: 10.1007/s004220000171
4. Arleo, A., Smeraldi, F., and Gerstner, W. (2004). Cognitive navigation based on non-uniform gabor space sampling, unsupervised growing networks, and reinforcement learning. IEEE Trans. Neural Netw. 15, 639-652. doi: 10.1109/TNN.2004.826221
5. Artola, A., and Singer, W. (1993). Long-term depression of excitatory synaptic transmission and its relationship to long-term potentiation. Trends Neurosci. 16, 480-487. doi: 10.1016/0166-2236(93)90081-V
6. Aston-Jones, G., and Cohen, J. (2005). An integrative theory of locus coeruleus-norepinephrine function: Adaptive gain and optimal performance. Annu. Rev. Neurosci. 28, 403-450. doi: 10.1146/annurev.neuro.28.061604.135709
7. Bailey, C. H., Giustetto, M., Huang, Y.-Y., Hawkins, R. D., and Kandel, E. R. (2000). Is heterosynaptic modulation essential for stabilizing hebbian plasiticity and memory. Nat. Rev. Neurosci. 1, 11-20. doi: 10.1038/35036191
8. Baras, D., and Meir, R. (2007). Reinforcement learning, spike-time-dependent plasticity, and the bcm rule. Neural Comput. 19, 2245-2279. doi: 10.1162/neco.2007.19.8.2245
9. Barnes, C. (1979). Memory deficits associated with senescence: a neurophysiological and behavioral study in the rat. J. Comp. Physiol. Psychol. 93, 74-104. doi: 10.1037/h0077579
10. Barrett, A. B., Billings, G. O., Morris, R. G., and van Rossum, M. C. (2009). State based model oflong-term potentiation and synaptic tagging and capture. PLOS Comput. Biol. 5:e1000259. doi: 10.1371/journal.pcbi.1000259
11. Baxter, J., and Bartlett, P. (2001). Infinite-horizon policy-gradient estimation. J. Artif. Intell. Res. 15, 319-350. doi: 10.1613/jair.806
12. Bellman, R. E. (1957). Dynamic Programming. Princeton: Princeton University Press.
13. Bergado, J. A., Lucas, M., and Richter-Levin, G. (2011). Emotional taggingâATa simple hypothesis in a complex reality. Progr. Neurobiol. 94, 64-76. doi: 10.1016/j.pneurobio.2011.03.004
14. Bertsekas, D. (1987). DynamicProgramming: Deterministic and Stochastic Models. Englewood cliffs, NJ: Prentice-Hall.
15. Bi, G. Q., and Poo, M. M. (1998). Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. J. Neurosci. 18, 10464-10472
16. Bi, G., and Poo, M. (2001). Synaptic modification of correlated activity: Hebb’s postulate revisited. Ann. Rev. Neurosci. 24, 139-166. doi: 10.1146/annurev.neuro.24.1.139
17. Bissière, S., Humeau, Y., and Lüthi, A. (2003). Dopamine gates ltp induction in lateral amygdala by suppressing feedforward inhibition. Nat. Neurosci. 6, 587-592. doi: 10.1038/nn1058
18. Bliss, T., Collingridge, G., and Morris, R. (2003). Long-term potentiation: enhancing neuroscience for 30 years-Introduction. Phil. Trans. R. Soc. Lond B Biol. Sci. 358, 607-611. doi: 10.1098/rstb.2003.1282
19. Bliss, T. V. P., and Gardner-Medwin, A. (1973). Long-lasting potentation of synaptic transmission in the dendate area of unanaesthetized rabbit following stimulation of the perforant path. J. Physiol. 232, 357-374. doi: 10.1113/jphysiol.1973.sp010274
20. Bliss, T., and Lomo, T. (1973). Long-lasting potentation of synaptic transmission in the dendate area of anaesthetized rabbit following stimulation of the perforant path. J. Physiol. 232, 351-356
21. Bliss, T. V. P., and Collingridge, G. L. (1993). A synaptic model of memory: long-term potentiation in the hippocampus. Nature 361, 31-39. doi: 10.1038/361031a0
22. Bohte, S. M., and Mozer, M. C. (2007). Reducing the variability of neural responses: a computational theory of spike-timing-dependent plasticity. Neural Comp. 19, 371-403. doi: 10.1162/neco.2007.19.2.371
23. Brea, J., Senn, W., and Pfister, J.-P. (2011). “Sequence learning with hidden units in spiking neural networks," in Advances in Neural Information Processing Systems 24 (NIPS 2011), eds J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. C. N. Pereira, and K. Q. Weinberger (Granada: Neural Information Processing Systems Foundation, Inc.), 1422-1430
24. Brea, J., Senn, W., and Pfister, J.-P. (2013). Matching recall and storage in sequence learning with spiking neural networks. J. Neurosci. 33, 9565-9575. doi: 10.1523/JNEUROSCI.4098-12.2013
25. Brown, T. H., Zador, A. M., Mainen, Z. F., and Claiborne, B. J. (1991). “Hebbian modifications in hippocampal neurons," in Long-term potentiation, eds M. Baudry and J. Davis (Cambridge: MIT Press), 357-389
26. Brunel, N. (2000). Dynamics of sparsely connected networks of excitatory and inhibitory neurons. Comput. Neurosci. 8, 183-208. doi: 10.1023/A:1008925309027
27. Buchanan, K. A., Petrovic, M. M., Chamberlain, S. E. L., Marrion, N. V., and Mellor, J. R. (2010). Facilitation of long-term potentiation by muscarinic m1 receptors is mediated by inhibition of sk channels. Neuron 68, 948-963. doi: 10.1016/j.neuron.2010.11.018
28. Buonomano, D. V., and Maass, W. (2009). State-dependent computations: spatiotemporal processing in cortical networks. Nat. Rev. Neurosci. 10, 113-125. doi: 10.1038/nrn2558
29. Calabresi, P., Maj, R., Mercuri, N. B., and Bernandi, G. (1992). Coactivation of D1 and D2 dopamine receptors is required for long-term synaptic depression in the striatum. Neurosci. Lett. 142, 95-99. doi: 10.1016/0304-3940(92)90628-K
30. Calabresi, P., Picconi, B., Tozzi, A., and Di Filippo, M. (2007). Dopamine-mediated regulation of corticostriatal synaptic plasticity. Trends Neurosci. 30, 211-219. doi: 10.1016/j.tins.2007.03.001
31. Caporale, N., and Dan, Y. (2008). Spike timing-dependent plasticity: a hebbian learning rule. Ann. Rev. Neurosci. 31, 25-46. doi: 10.1146/annurev.neuro.31.060407.125639
32. Carpenter, G., and Grossberg, S. (1988). The art of adpative pattern recognition by a self-organizing neural network. Computer 21, 77-88. doi: 10.1109/2.33
33. Cassenaer, S., and Laurent, G. (2012). Conditional modulation of spike-timing-dependent plasticity for olfactory learning. Nature 482, 47-52. doi: 10.1038/nature10776
34. Clopath, C., Busing, L., Vasilaki, E., and Gerstner, W. (2010). Connectivity reflects coding: a model of voltage-based spike-timing-dependent-plasticity with homeostasis. Nat. Neurosc. 13, 344-352. doi: 10.1038/nn.2479
35. Clopath, C., Ziegler, L., Vasilaki, E., Büsing, L., and Gerstner, W. (2008). Tag-trigger-consolidation: a model of early and late long-term-potentiation and depression. PLOS Comput. Biol. 4:e1000248. doi: 10.1371/journal.pcbi.1000248
36. Cohen, J., Haesler, S., Vong, L., Lowell, B. B., and Ushida, N. (2012). Neuron-type-specific signals for reward and punishment in the ventral tegmental area. Nature 482, 85-88. doi: 10.1038/nature10754
37. Couey, J. J., Meredith, R. M., Spijker, S., Poorthuis, R. B., Smit, A. B., Brussaard, A. B., et al. (2007). Distributed network actions by nicotine increase the threshold for spike-timing-dependent plasticity in prefrontal cortex. Neuron 54, 73-87. doi: 10.1016/j.neuron.2007.03.006
38. Crow, T. (1968). Cortical synapses and reinforcement: a hypothesis. Nature 219, 736-737. doi: 10.1038/219736a0
39. Dayan, P. (1992). The convergens ofTD(X) for general X. Mach. Learn. 8, 341-362. doi: 10.1007/BF00992701
40. Di Castro, D., Volkinshtein, D., and Meir, R. (2009). “Temporal difference based actor critic learning-convergence and neural implementation," in Proceedings of NIPS 2009, Advances in Neural Information Processing Systems, Vol. 22 (Cambridge, MA: MIT Press), 385-392
41. Disney, A. A., Aoki, C., and Hawken, M. J. (2007). Gain modulaton by nicotine in macaque v1. Neuron 56, 701-713. doi: 10.1016/j.neuron.2007.09.034
42. Disney, A. A., Domakonda, K. V., and Aoki, C. (2006). Differential expression of muscarinic acetylcholine receptors across excitatory and inhibitory cells in visual cortical areas v1 and v2 of the macaque monkey. J. Comp. Neurol. 499, 49-63. doi: 10.1002/cne.21096
43. Doya, K. (2002). Metalearning and neuromodulation. Neural Netw. 15, 495-506. doi: 10.1016/S0893-6080(02)00044-8
44. Doya, K. (2007). Reinforcement learning: computational theory and biological mechanisms. HFSPJ. 1,30-40. doi: 10.2976/1.2732246/10.2976/1
45. Drever, B. D., Riedel, G., and Platt, B. (2011). The cholinergic system and hippocampal plasticity. Behav. Brain Res. 221, 205-214. doi: 10.1016/j.bbr.2010.11.037
46. Engineer, N. D., Riley, J. R., Seale, J. D., Vrana, W. A., Shetake, J. A., Sudanagunta, S. P., et al. (2011). Reversing pathological neural activity using targeted plasticity. Nature 470, 101-106. doi: 10.1038/nature09656
47. Farries, M. A., and Fairhall, A. L. (2007). Reinforcement learning with modulated spike timing dependent synaptic plasticity. J. Neurophysiol. 98, 3648-3665. doi: 10.1152/jn.00364.2007
48. Florian, R. V. (2007). Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural Comput. 19, 1468-1502. doi: 10.1162/neco.2007.19.6.1468
49. Foster, D., Morris, R., and Dayan, P. (2000). Models of hippocampally dependent navigation using the temporal difference learning rule. Hippocampus 10, 1-16. doi: 10.1002/(SICI)1098-1063(2000)10:1<1::AID-HIP01>3.0.C0;2-1
50. Franzius, M., Wilbert, N., and Wiskott, L. (2007). “Unsupervised learning of invariant 3D-object representations with slow feature analysis," in 3rd Bernstein Symposium for Computational Neuroscience, GAttingen, September 24-27, Bernstein Centerfor ComputationalNeuroscience (BCCN) (Gttingen), 105
51. Frémaux, N., Sprekeler, H., and Gerstner, W. (2010). Functional requirements for reward-modulated spike-timing-dependent plasticity. J. Neurosci. 40, 13326-13337. doi: 10.1523/JNEUR0SCI.6249-09.2010
52. Frémaux, N., Sprekeler, H., and Gerstner, W. (2013). Reinforcement learning using continuous time actor-critic framework with spiking neurons. PLOS Comput. Biol. 9:e1003024. doi: 10.1371/journal.pcbi.1003024
53. Freund, T. F., Powell, J. F., and Smith, A. D. (1984). Tyrosine hydroxylase-immunoreactive boutons in synaptic contact with identified striatonigral neurons, with particular reference to dendritic spines. Neuroscience 13, 11895-1215. doi: 10.1016/0306-4522(84)90294-X
54. Frey, U., and Morris, R. (1997). Synaptic tagging and long-term potentiation. Nature 385, 533-536. doi: 10.1038/385533a0
55. Frey, U., and Morris, R. (1998). Synaptic tagging: implications for late maintenance of hippocampal long-term potentiation. Trends Neurosci. 21, 181-188. doi: 10.1016/S0166-2236(97)01189-2
56. Friedrich, J., Urbanczik, R., and Senn, W. (2011). Spatio-temporal credit assignment in neuronal population learning. PLoS Comput. Biol. 7:e1002092. doi: 10.1371/journal.pcbi.1002092
57. Fusi, S. (2002). Hebbian spike-driven synaptic plasticity for learning patterns of mean firing rates. Biol. Cybern. 87, 459-470. doi: 10.1007/s00422-002-0356-8
58. Fusi, S., and Abbott, L. (2007). Limits on the memory storage capacity of bounded synapses. Nat. Neurosci. 10, 485-493. doi: 10.1038/nn1859
59. Gerstner, W., Kempter, R., van Hemmen, J. L., and Wagner, H. (1996). A neuronal learning rule for sub-millisecond temporal coding. Nature 383, 76-78. doi: 10.1038/383076a0
60. Gerstner, W., Kistler, W., Naud, R., and Paninski, L. (2014). Neuronal Dynamics. From Single Neurons to Networks and Cognition. Cambridge: Cambridge University Press.
61. Gerstner, W., and Kistler, W. M. (2002). Mathematical formulations of hebbian learning. Biol. Cybern. 87, 404-415. doi: 10.1007/s00422-002-0353-y
62. Golding, N. L., Staff, N. P., and Spruston, N. (2002). Dendritic spikes as a mechanism for cooperative long-term potentiation. Nature 418, 326-331. doi: 10.1038/nature00854
63. Goldman-Rakic, P. C., Leranth, C., williams, S. M., Mons, N., and Geffard, M. (1989). Dopamine synaptic complex with pyramidal neurons in primate cerebral cortex. Proc. Natl. Acad. Sci. U.S.A. 86, 9015-9019. doi: 10.1073/pnas.86.22.9015
64. Gotzsche, C. R., and Woldbye, D. P. (2015). The role of npy in learning and memory. Neuropeptides. doi: 10.1016/j.npep.2015.09.010. [Epub ahead of print].
65. Gruber, M. J., Gelman, B. D., and Ranganath, C. (2014). States of curiosity modulate hippocampus-dependent learning via the dopaminergic circuit. Neuron 84, 486-496. doi: 10.1016/j.neuron.2014.08.060
66. Gu, Q. (2002). Neuromodulatory transmitter systems in the cortex and their role in cortical plasticity. Neuroscience 111, 815-835. doi: 10.1016/S0306-4522(02)00026-X
67. Gu, Z., and Yakel, J. L. (2011). Timing-dependent septal cholinergic induction of dynamic hippocampal synaptic plasticity. Neuron 71, 155-165. doi: 10.1016/j.neuron.2011.04.026
68. Hasselmo, M. (2006). The role of acetylcholine in learning and memory. Curr. Opin. Neurobiol. 16, 710-715. doi: 10.1016/j.conb.2006.09.002
69. Hebb, D. O. (1949). The Organization ofBehavior. New York, NY: Wiley.
70. Hennequin, G., Vogels, T. P., and Gerstner, W. (2014). Optimal control of transient dynamics in balanced networks supports generation of complex movements. Neuron 82, 1394-1406. doi: 10.1016/j.neuron.2014.04.045
71. Hertz, J., Krogh, A., and Palmer, R. G. (1991). Introduction to the Theory ofNeural Computation. Redwood City, CA: Addison-Wesley.
72. Herzog, M. H., Aberg, K. C., Frémaux, N., Gerstner, W., and Sprekeler, H. (2012). Perceptual learning, roving, and the unsupervised bias. Vis. Res. 61, 95-99. doi: 10.1016/j.visres.2011.11.001
73. Hinton, G., and Sejnowski, T. (1999). Unsupervised Learning: Foundations of Neural Computation. Cambridge, MA: MIT Press.
74. Hökfelt, T., Broberger, C., Xu, Z.-Q. D., Sergeyev, V., Ubin, R., and Diez, M. (2000). Neuropeptides â ĂŤ an overview. Neuropharmacology 39, 1337-1356. doi: 10.1016/S0028-3908(00)00010-1
75. Hollerman, J. R., and Schultz, W. (1998). Dopamine neurons report an error in the temporal prediction of reward during learning. Nat. Neurosci. 1, 304-309. doi: 10.1038/1124
76. Houk, J., Adams, J., and Barto, A. (1995). “A model of how the basal ganglia generate and use neural signals that predict reinforcement" in Models on Information Processing in the Basal Ganglia, eds J. C. Houk, J. L. Davis, and D. G. Beiser (Cambridge: MIT Press), 249-270
77. Izhikevich, E. (2007). Solving the distal reward problem through linkage of STDP and dopamine signaling. Cereb. Cortex 17, 2443-2452. doi: 10.1093/cercor/bhl152
78. Jay, T. M. (2003). Dopamine: a potential substrate for synaptic plasticity and memory mechanisms. Prog. Neurobiol. 69, 375-390. doi: 10.1016/S0301-0082(03)00085-6
79. Joel, D., Niv, Y., and Ruppin, E. (2002). Actor-critic models of the basal ganglia: new anatomical and computational perspectives. Neural Netw. 15, 535-547. doi: 10.1016/S0893-6080(02)00047-3
80. Kaczmarek, L., and Levitan, I. (1987). Neuromodulation: The Biochemical Control ofNeuronal Excitability. Oxford: Oxford University Press.
81. Kang, J., Huppé-Gourgues, F., and Vaucher, E. (2014). Boosting visual cortex function and plasticity with acetylcholine to enhance visual perception. Front. Syst. Neurosci. 8:172. doi: 10.3389/fnsys.2014.00172
82. Katz, P., and Edwards, D. (1999). “Metamodulation: the control and modulation of neuromodulation,” in Beyond Neurotransmission: Neuromodulation and Its Importance for Information Processing, ed P. Katz (New York, NY: Oxford Univesity Press), 339-381
83. Kempter, R., Gerstner, W., and van Hemmen, J. L. (1999). Hebbian learning and spiking neurons. Phys. Rev. E 59,4498-4514. doi: 10.1103/PhysRevE.59.4498
84. Kerr, J. N. D., and Wickens, J. R. (2001). Dopamine d-1/d-5 receptor activation is required for long-term potentiation in the rat neostriatum in vitro. J. Neurophysiol. 85, 117-124
85. Kety, S. (1972). The possible role of the adrenergic systems of the cortex in learning. Res. Publ. Assoc. Res. Nerv. Ment. Dis. 50, 376-389
86. Kilgard, M. P., and Merzenich, M. M. (1998). Cortical map reorganization enabled by nucleus basalis activity. Science 279, 1714-1718. doi: 10.1126/science.279.5357.1714
87. Klopf, A. H. (1982). The Hedonistic Neuron: A Theory of Memory, Learning, and Intelligence. Washington: Hemisphere.
88. Kohonen, T. (1990). The self-organizing map. Proc. IEEE 78, 1464-1480. doi: 10.1109/5.58325
89. Kolossa, A., Kopp, B., and Fingscheidt, T. (2015). A computational analysis of the neural bases of bayesian inference. Neuroimage 106, 222-237. doi: 10.1016/j.neuroimage.2014.11.007
90. Laje, R., and Buonomano, D. (2013). Robust timing and motor patterns by taming chaos in recurrent neural networks. Nat. Neurosci. 16, 925-933. doi: 10.1038/nn.3405
91. Legenstein, R., Pecevski, D., and Maass, W. (2008). A learning theory for reward-modulated spike-timing-dependent plasticity with application to biofeedback. PLoS Comput. Biol. 4:e1000180. doi: 10.1371/journal.pcbi.1000180
92. Levy, W. B., and Stewart, D. (1983). Temporal contiguity requirements for longterm associative potentiation/depression in hippocampus. Neurosci 8, 791-797. doi: 10.1016/0306-4522(83)90010-6
93. Li, N., and DiCarlo, J. (2012). Neuronal learning of invariant object representation in the ventral visual stream is not dependent on reward. J. Neurosci. 32, 6611-6620. doi: 10.1523/JNEUR0SCI.3786-11.2012
94. Lin, Y.-W., Min, M.-Y., Chiu, T.-H., and Yang, H.-W. (2003). Enhancement of associative long-term potentiation by activation of β-adrenergic receptors at cal synapses in rat hippocampal slices. J. Neurosci. 23, 4173-4181
95. Lisman, J. (2003). Long-term potentiation: outstanding questions and attempted synthesis. Phil. Trans. R. Soc. Lond. B Biol. Sci. 358, 829-842. doi: 10.1098/rstb.2002.1242
96. Lisman, J., Grace, A. A., and Duzel, E. (2011). A neoHebbian framework for episodic memory; role of dopamine-dependent late ltp. Trends Neurosci. 34, 536-547. doi: 10.1016/j.tins.2011.07.006
97. Ljunberg, T., and Schultz, P. A. (1992). Responses of monkey dopamine neurons during learning of behavioral interactions. J. Neurophysiol. 67, 145-163
98. Loewenstein, Y. (2008). Robustness of learning that is based on covariance-driven synaptic plasticity. PLOS Comput. Biol. 4:e1000007. doi: 10.1371/journal.pcbi.1000007
99. Loewenstein, Y., and Seung, H. (2006). Operant matching is a generic outcome of synaptic plasticity based on the covariance between reward and neural activity. Proc. Natl. Acad. Sci. U.S.A. 103, 15224-15229. doi: 10.1073/pnas.0505220103
100. Lomo, T. (1966). Frequency potentiation of excitatory synaptic activity in the dentate area of the hippocampal formation. Acta Physiol. Scand. 68 (Suppl. 277), 28
101. Lynch, G. S., Dunwiddie, T., and Gribkoff, V. (1977). Heterosynaptic depression: a postsynaptic correlate of long-term potentiation. Nature 266, 737-739. doi: 10.1038/266737a0
102. Ma, X., and Suga, N. (2005). Long-term cortical plasticity evoked by electric stimulation and acetylcholine applied to the auditory cortex. Proc. Natl. Acad. Sci. U.S.A. 102, 9335-9340. doi: 10.1073/pnas.0503851102
103. Malenka, R. C., and Nicoll, R. A. (1999). Long-term potentiation-a decade of progress? Science 285, 1870-1874. doi: 10.1126/science.285.5435.1870
104. Marder, E. (2012). Neuromodulation of neural circuits: back to the future. Neuron 76, 1-11. doi: 10.1016/j.neuron.2012.09.010
105. Markram, H., Lübke, J., Frotscher, M., and Sakmann, B. (1997). Regulation of synaptic efficacy by coincidence of postysnaptic AP and EPSP. Science 275, 213-215. doi: 10.1126/science.275.5297.213
106. Markram, H., Gerstner, W., and Sjostrom, P. J.(2011). A history of spike-timing dependent plasticity. Front. Synaptic Neurosci. 3:4. doi: 10.3389/fnsyn.2011.00004
107. Martin, S. J., Grimwood, P. D., and Morris, R. G. (2000). Synaptic plasticity and memory: an evaluation of the hypothesis. Ann. Rev. Neurosci. 23, 649-711. doi: 10.1146/annurev.neuro.23.1.649
108. Meyer, W.-U., Niepel, M., Rudolph, U., and Schützwohl, A. (1991). An experimental analysis of surprise. Cogn. Emot. 5, 295-331. doi: 10.1080/02699939108411042
109. Missale, C., Nash, S. R., Robinson, S. W., Jaber, M., and Caron, M. G. (1998). Dopamine receptors: from structure to function. Physiol. Rev. 78, 189-225
110. Mitsushima, D., Sano, A., and Takahashi, T. (2013). A cholinergic trigger drives learning-induced plasticity at hippocampal synapses. Nat. Commun. 4:2760. doi: 10.1038/ncomms3760
111. Moncada, D., and Viola, H. (2007). Induction of long-term memory by exposure to novelty requires protein synthesis: evidence for a behavioral tagging. J. Neurosci. 27, 7476-7481. doi: 10.1523/JNEUROSCI.1083-07.2007
112. Morris, R. G. M., Anderson, E., Lynch, G. S., and Baudry, M. (1986). Selective impairment of learning and blockade of long-term potentiation by an n-methyl-d-aspartate receptor antagonist, ap5. Nature 319, 774-776. doi: 10.1038/319774a0
113. Morrison, A., Diesmann, M., and Gerstner, W. (2008). Phenomenological models of synaptic plasticity based on spike timing. Biol. Cybern. 98, 459-478. doi: 10.1007/s00422-008-0233-1
114. Moser, E. I., Kropff, E., and Moser, M.-B. (2008). Place cells, grid cells, and the brain's spatial representation system. Annu. Rev. Neurosci. 31, 69-89. doi: 10.1146/annurev.neuro.31.061307.090723
115. Nadim, F., and Bucher, D. (2014). Neuromodulation of neurons and synapses. Curr. Opin. Neurobiol. 29, 48-56. doi: 10.1016/j.conb.2014.05.003
116. Nakano, T., Doi, T., Yoshimoto, J., and Doya, K. (2010). A kinetic model of dopamine-and calcium-dependent striatal synaptic plasticity. PLoS Comput. Biol. 6:e1000670. doi: 10.1371/journal.pcbi.1000670
117. Nasser, M. R., Rumsey, K. M., Wilson, R. C., Parikh, K., Heasly, B., and Gold, J. I. (2012). Rational regulation of learning dynamics by pupil-linked arousal systems. Nat. Neurosci. 15, 1040-1046. doi: 10.1038/nn.3130
118. Nessler, B., Pfeiffer, M., Buesing, L., and Maass, W. (2013). Bayesian computation emerges in generic cortical microcircuits through spike-timing-dependent plasticity. PLOS Comput. Biol. 9:e1003037. doi: 10.1371/journal.pcbi.1003037
119. Nicola, S. M., Surmeier, J., and Malenka, R. C. (2000). Dopaminergic modulation of neuronal excitability in the striatum and nucleus accumbens. Annu. Rev. Neurosci. 23, 185-215. doi: 10.1146/annurev.neuro.23.1.185
120. O'Keefe, J., and Nadel, L. (1978). The Hippocampus as a Cognitive Map, Vol. 3. Oxford: Clarendon Press.
121. Packard, M. G., and McGaugh, J. L. (1992). Double dissociation of fornix and caudate nucleus lesions on acquisition of two water maze tasks: Further evidence for multiple memory systems. Behav. Neurosci. 106, 439-446. doi: 10.1037/0735-7044.106.3.439
122. Parra, L. C., Beck, J. M., and Bell, A. J. (2009). On the maximization of information flow between spiking neurons. Neural Comput. 21,2991-3009. doi: 10.1162/neco.2009.04-06-184
123. Pawlak, V., and Kerr, J. (2008). Dopamine receptor activation is required for corticostriatal spike-timing-dependent plasticity. J. Neurosci. 28, 2435-2446. doi: 10.1523/JNEUROSCI.4402-07.2008
124. Pawlak, V., Wickens, J., Kirkwood, A., and Kerr, J. (2010). Timing is not everything: neuromodulation opens the STDP gate. Front. Synaptic Neurosci. 2:146. doi: 10.3389/fnsyn.2010.00146
125. Pfister, J.-P., Toyoizumi, T., Barber, D., and Gerstner, W. (2006). Optimal spike-timing dependent plasticity for precise action potential firing in supervised learning. Neural Comput. 18, 1318-1348. doi: 10.1162/neco.2006.18.6.1318
126. Potjans, W., Diesmann, M., and Morrison, A. (2011). An imperfect dopaminergic error signal can drive temporal-difference learning. PLOS Comput. Biol. 7:e1001133. doi: 10.1371/journal.pcbi.1001133
127. Potjans, W., Morrison, A., and Diesmann, M. (2009). A spiking neuron model of an actor-critic learning agent. Neural Comput. 21, 301-339. doi: 10.1162/neco.2008.08-07-593
128. Ranganath, C., and Rainer, G. (2003). Neural mechanisms for detecting and remembering novel events. Nat. Rev. Neurosci. 4, 193-202. doi: 10.1038/nrn1052
129. Rasmusson, D. (2000). The role of acetylcholine in cortical synaptic plasticity. Behav. Brain Res. 115, 205-218. doi: 10.1016/S0166-4328(00)00259-X
130. Redgrave, P., and Gurney, K. (2006). The short-latency dopamine signal: a role in discovering novel actions? Nat. Rev. Neurosci. 7, 967-975. doi: 10.1038/nrn2022
131. Redondo, R. L., and Morris, R. G. M. (2011). Making memories last: the synaptic tagging and capture hypothesis. Nat. Rev. Neurosci. 12, 17-30. doi: 10.1038/nrn2963
132. Reymann, K. G., and Frey, J. U. (2007). The late maintenance of hippocampal ltp: requirements, phases,synaptic tagging, late associativity and implications. Neuropharmacology 52, 24-40. doi: 10.1016/j.neuropharm.2006.07.026
133. Reynolds, J. N., and Wickens, J. R. (2002). Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw. 15, 507-521. doi: 10.1016/S0893-6080(02)00045-X
134. Rezende, D., and Gerstner, W. (2014). Stochastic variational learning in recurrent spiking networks. Front. Comput. Neurosci. 8:38. doi: 10.3389/fncom.2014.00038
135. Rezende, D., Wierstra, D., and Gerstner, W. (2011). “Variational learning for recurrent spiking networks" in Advances in Neural Information Processing Systems 24 (NIPS 2011), eds J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. C. N. Pereira, and K. Q. Weinberger (Granada: Neural Information Processing Systems Foundation, Inc.), 136-144
136. Sara, S. (2009). The locus coeruleus and noradrenergic modulation of cognition. Nat. Rev. Neurosci. 10, 211-223. doi: 10.1038/nrn2573
137. Schmidhuber, J. (1991). “Curious model-building control systems," in Proceedings of the International Joint Conference on Neural Networks, Singapore, Vol. 2 (Los Alamitos, CA: IEEE press), 1458-1463. doi: 10.1109/IJCNN.1991.170605
138. Schmitzer-Torbert, N., and Redish, A. D. (2004). Neuronal activity in the rodent dorsal striatum in sequential navigation: separation of spatial and reward responses on the multiple t task. J. Neurophysiol. 91, 2259-2272. doi: 10.1152/jn.00687.2003
139. Schultz, W. (1998). Predictiv reward signal of dopamine neurons. J. Neurophysiol. 80, 1-27
140. Schultz, W. (2002). Getting formal with dopamine and reward. Neuron 36, 241-263. doi: 10.1016/S0896-6273(02)00967-4
141. Schultz, W. (2006). Behavioral theories and the neurophysiology of reward. Annu. Rev. Psychol. 57, 87-115. doi: 10.1146/annurev.psych.56.091103.070229
142. Schultz, W., Apicella, P., and Ljungberg, T. (1993). Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J. Neurosci. 13, 900-913
143. Schultz, W., Dayan, P., and Montague, P. R. (1997). A neural substrate for prediction and reward. Science 275, 1593-1599. doi: 10.1126/science.275.5306.1593
144. Schulz, J. M., Redgrave, P., and Reynolds, J. N. J. (2010). Cortico-striatal spike-timing dependent plasticity after activation of subcortical pathways. Front. SynapticNeurosci. 2:23. doi: 10.3389/fnsyn.2010.00023
145. Seamans, J. (2007). Dopamine anatomy. Scholarpedia 2:3737. doi: 10.4249/scholarpedia.3737
146. Seidenbacher, T., Reyman, K. G., and Balschun, D. (1997). A post-tetanic time window for the reinforcement of long-term potentiation by appetitive and aversive stimuli. Proc. Natl. Acad. Sic. U.S.A. 94, 1494-1499. doi: 10.1073/pnas.94.4.1494
147. Seol, G. H., Ziburkus, J., Huang, S., Song, L., Kim, I. T., Takamiya, K., et al. (2007). Neuromodulators control the polarity of spike-timing-dependent synaptic plasticity. Neuron 55, 919-929. doi: 10.1016/j.neuron.2007.08.013
148. Seung, H. (2003). Learning in spiking neural networks by reinforcement of stochastic synaptic transmission. Neuron 40, 1063-1073. doi: 10.1016/S0896-6273(03)00761-X
149. Shen, W., Flajolet, M., Greengard, P., and Surmeier, D. J. (2008). Dichotomous dopaminergic control of striatal synaptic plasticity. Science 321, 848-851. doi: 10.1126/science.1160575
150. Sheynikhovich, D., Chavarriaga, R., Strosslin, T., Arleo, A., and Gerstner, W. (2009). Is there a geometric module for spatial orientation? insights from a rodent navigation model. Psychol. Rev. 116, 540-566. doi: 10.1037/a0016170
151. Shohamy, D., and Adcock, R. (2010). Dopamine and adaptive memory. Trends Cogn. Sci. 14, 464-472. doi: 10.1016/j.tics.2010.08.002
152. Sjostrom, J., and Gerstner, W. (2010). Spike-timing dependent plasticity. Scholarpedia 5:1362. doi: 10.4249/scholarpedia.1362
153. Sjostrom, P., and Hausser, M. (2006). A cooperative switch determines the sign of synaptic plasticity in distal dendrites of neocortical pyramidal neurons. Neuron 51, 227-238. doi: 10.1016/j.neuron.2006.06.017
154. Sjostrom, P., Ranz, E., Roth, A., and Hausser, M. (2008). Dendritic excitabilityh and synaptic plasticity. Physiol. Rev. 88, 769-840. doi: 10.1152/physrev.00016.2007
155. Sjostrom, P., Turrigiano, G., and Nelson, S. (2001). Rate, timing, and cooperativity jointly determine cortical synaptic plasticity. Neuron 32, 1149-1164. doi: 10.1016/S0896-6273(01)00542-6
156. Song, S., and Abbott, L. F. (2001). Cortical development and remapping through spike timing-dependent plasticity. Neuron 32, 339-350. doi: 10.1016/S0896-6273(01)00451-2
157. Song, S., Miller, K., and Abbott, L. (2000). Competitive Hebbian learning through spike-time-dependent synaptic plasticity. Nat. Neurosci. 3, 919-926. doi: 10.1038/78829
158. Steinberg, E. E., Keiflin, R., Boivin, J. R., Witten, I., Deisseroth, K., and Janak, P. (2013). A causal link between prediction errors, dopamine neurons and learning. Nat. Neurosci. 16, 966-973. doi: 10.1038/nn.3413
159. Sugisaki, E., Fukushima, Y., Tsukada, M., and Aihara, T. (2011). Cholinergic modulation on spike timing-dependent plasticity in hippocampal ca1 network. Neuroscience 192, 91-101. doi: 10.1016/j.neuroscience.2011.06.064
160. Suri, R. E., and Schultz, W. (2001). Temporal difference model reproduces anticipatory neural activity. Neural Comput. 13, 841-862. doi: 10.1162/089976601300014376
161. Suri, R. E., and Schultz, W. (1998). Learning of sequential movements with dopamine-like reinforcement signal in neural network model. Exp. Brain Res. 121, 350-354. doi: 10.1007/s002210050467
162. Suri, R. E., and Schultz, W. (1999). A neural network with dopamine-like reinforcement signal that learns a spatial delayed response task. Neuroscience 91, 871-890. doi: 10.1016/S0306-4522(98)00697-6
163. Sussillo, D. (2014). Neural circuits as computational dynamics systems. Curr. Opin. Neurobiol. 25, 156-163. doi: 10.1016/j.conb.2014.01.008
164. Sussillo, D., and Abbott, L. (2012). Transferring learning from external to internal weights in echo-state networks with sparse connectivity. PLOS ONE 7:e37372. doi: 10.1371/journal.pone.0037372
165. Sutton, R. (1998). Learning to predict by the method of temporal differences. Mach. Learn. 3, 9-44. doi: 10.1007/BF00115009
166. Sutton, R., and Barto, A. (1998). Reinforcement Learning. Cambridge: MIT Press.
167. Svensson, E., Grillner, S., and Parker, D. (2001). Gating and braking of short-and long-term modulatory effects by interactions between colocalized neuromodulators. J. Neurosci. 21, 5984-5992
168. Swinehart, C. D., and Abbott, L. F. (2006). Dimensional reduction for reward-based learning. Network 17, 235-252. doi: 10.1080/09548980600773215
169. Toyoizumi, T., Pfister, J.-P., Aihara, K., and Gerstner, W. (2007). Optimality model of unsupervised spike-timing dependent plasticity: synaptic memory and weight distribution. Neural Comput. 19, 639-671. doi: 10.1162/neco.2007.19.3.639
170. Tsai, H.-C., Zhang, F., Adamantidis, A., Stuber, G. D., Bonci, A., de Lecea, L., et al. (2009). Phasic firing in dopaminergic neurons is sufficient for behavioral conditioning. Science 324, 1080-1084. doi: 10.1126/science.1168878
171. Tully, K., and Bolshakov, V. (2010). Emotional enhancement of memory: how norepinephrine enables synaptic plasticity. Mol. Brain 3:15. doi: 10.1186/17566606-3-15
172. Urbanczik, R., and Senn, W. (2009). Reinforcement learning in populations of spiking neurons. Nat. Neurosci. 12, 250-252. doi: 10.1038/nn.2264
173. van der Meer, M. A. A., and Redish, A. D. (2011). Theta phase precession in rat ventral striatum links place and reward information. J. Neurosci. 31, 2843-2854. doi: 10.1523/JNEUROSCI.4869-10.2011
174. van Vreeswijk, C., and Sompolinsky, H. (1996). Chaos in neuronal networks with balanced excitatory and inhibitory activity. Science 274, 1724-1726. doi: 10.1126/science.274.5293.1724
175. Vasilaki, E., Frémaux, N., Urbanczik, R., Senn, W., and Gerstner, W. (2009). Spike-based reinforcement learning in continuous state and action space: when policy gradient methods fail. PLOS Comput. Biol. 5:e1000586. doi: 10.1371/journal.pcbi.1000586
176. Vogels, T., Rajan, K., and Abbott, L. (2005). Neural network dynamics. Annu. Rev. Neurosci. 28, 357-376. doi: 10.1146/annurev.neuro.28.061604.135637
177. Waelti, P., Dickinson, A., and Schultz, W. (2001). Dopamine responses comply with basic assumptions of formal learning theory. Nature 412, 43-48. doi: 10.1038/35083500
178. Williams, R. (1992). Simple statistical gradient-following methods for connectionist reinforcement learning. Mach. Learn. 8, 229-256. doi: 10.1007/BF00992696
179. Xie, X., and Seung, H. S. (2004). Learning in neural networks by reinforcement of irregular spiking. Phys. Rev. E 69:41909. doi: 10.1103/PhysRevE.69.041909
180. Xu, T.-X., and Yao, W.-D. (2010). D1 and d2 dopamine receptors in separate circuits cooperate to drive associative long-term potentiation in the prefrontal cortex. Proc. Natl. Acad. Sci. U.S.A. 107, 16366-16371. doi: 10.1073/pnas.1004108107
181. Yu, A. J., and Dayan, P. (2005). Uncertainty, neuromodulation, and attention. Neuron 46, 681-692. doi: 10.1016/j.neuron.2005.04.026
182. Zhang, J. C., Lau, P. M., and Bi, G. Q. (2009). Gain in sensitivity and loss in temporal contrast of stdp by dopaminergic modulation at hippocampal synapses. Proc. Natl. Aca. Sci. U.S.A. 106, 13028-13033. doi: 10.1073/pnas.0900546106