Timing and expectation of reward: A neuro-computational model of the afferents to the ventral tegmental area

Frontiers in Neurorobotics

Volumn 8, Issue JAN, 2014, Pages

  Vitay, Julien ^a   Hamker, Fred H ^a,b  

^a CHEMNITZ UNIVERSITY OF TECHNOLOGY   (Germany)

^b CHARITÉ UNIVERSITÄTSMEDIZIN BERLIN   (Germany)

Author keywords

Amygdala; Basal ganglia; Classical conditioning; Dopamine; Timing; VTA

Indexed keywords

DOPAMINE;

AMYGDALOID NUCLEUS; ARTICLE; ASSOCIATION; BASAL GANGLION; CONDITIONING; GLOBUS PALLIDUS; HABENULA; LATERAL HYPOTHALAMUS; NERVE CELL NETWORK; NEUROMODULATION; NUCLEUS ACCUMBENS; PEDUNCULOPONTINE TEGMENTAL NUCLEUS; REWARD; SENSORY NERVE; STATE DEPENDENT LEARNING; VENTRAL TEGMENTUM; VENTROMEDIAL PREFRONTAL CORTEX;

EID: 84904914233     PISSN: None     EISSN: 16625218     Source Type: Journal    
DOI: 10.3389/fnbot.2014.00004     Document Type: Article

References (166)

1. Ambroggi, F., Ishikawa, A., Fields, H. L., and Nicola, S. M. (2008). Basolateral amygdala neurons facilitate reward-seeking behavior by exciting nucleus accumbens neurons. Neuron 59, 648-661. doi: 10.1016/j.neuron.2008.07.004.
2. Apicella, P., Deffains, M., Ravel, S., and Legallet, E. (2009). Tonically active neurons in the striatum differentiate between delivery and omission of expected reward in a probabilistic task context. Eur. J. Neurosci. 30, 515-526. doi: 10.1111/j.1460-9568.2009.06872.x.
3. Balcita-Pedicino, J. J., Omelchenko, N., Bell, R., and Sesack, S. R. (2011). The inhibitory influence of the lateral habenula on midbrain dopamine cells: ultrastructural evidence for indirect mediation via the rostromedial mesopontine tegmental nucleus. J. Comp. Neurol. 519, 1143-1164. doi: 10.1002/cne.22561.
4. Balsam, P. D., Drew, M. R., and Yang, C. (2002). Timing at the start of associative learning. Learn. Motiv. 33, 141-155. doi: 10.1006/lmot.2001.1104.
5. Baxter, M. G., and Murray, E. A. (2002). The amygdala and reward. Nat. Rev. Neurosci. 3, 563-573. doi: 10.1038/nrn875.
6. Belova, M. A., Paton, J. J., Morrison, S. E., and Salzman, C. D. (2007). Expectation modulates neural responses to pleasant and aversive stimuli in primate amygdala. Neuron 55, 970-984. doi: 10.1016/j.neuron.2007.08.004.
7. Bermudez, M. A., and Schultz, W. (2010). Reward magnitude coding in primate amygdala neurons. J. Neurophysiol. 104, 3424-3432. doi: 10.1152/jn.00540.2010.
8. Bissière, S., Humeau, Y., and Lüthi, A. (2003). Dopamine gates LTP induction in lateral amygdala by suppressing feedforward inhibition. Nat. Neurosci. 6, 587-592. doi: 10.1038/nn1058.
9. Bolam, J. P., Hanley, J. J., Booth, P. A., and Bevan, M. D. (2000). Synaptic organisation of the basal ganglia. J. Anat. 196, 527-542. doi: 10.1046/j.1469-7580.2000.19640527.x.
10. Bostan, A. C., and Strick, P. L. (2010). The cerebellum and basal ganglia are interconnected. Neuropsychol. Rev. 20, 261-270. doi: 10.1007/s11065-010-9143-9.
11. Bourdy, R., and Barrot, M. (2012). A new control center for dopaminergic systems: pulling the VTA by the tail. Trends Neurosci. 35, 681-690. doi: 10.1016/j.tins.2012.06.007.
12. Brischoux, F., Chakraborty, S., Brierley, D. I., and Ungless, M. A. (2009). Phasic excitation of dopamine neurons in ventral VTA by noxious stimuli. Proc. Natl. Acad. Sci. U.S.A. 106, 4894-4899. doi: 10.1073/pnas.0811507106.
13. Bromberg-Martin, E. S., and Hikosaka, O. (2011). Lateral habenula neurons signal errors in the prediction of reward information. Nat. Neurosci. 14, 1209-1216. doi: 10.1038/nn.2902.
14. Brown, J., Bullock, D., and Grossberg, S. (1999). How the basal ganglia use parallel excitatory and inhibitory learning pathways to selectively respond to unexpected rewarding cues. J. Neurosci. 19, 10502-10511.
15. Calabresi, P., Picconi, B., Tozzi, A., and Di Filippo, M. (2007). Dopamine-mediated regulation of corticostriatal synaptic plasticity. Trends Neurosci. 30, 211-219. doi: 10.1016/j.tins.2007.03.001.
16. Cardinal, R. N., Parkinson, J. A., Hall, J., and Everitt, B. J. (2002). Emotion and motivation: the role of the amygdala, ventral striatum, and prefrontal cortex. Neurosci. Biobehav. Rev. 26, 321-352. doi: 10.1016/S0149-7634(02)00007-6.
17. Carmichael, S. T., and Price, J. L. (1995). Sensory and premotor connections of the orbital and medial prefrontal cortex of macaque monkeys. J. Comp. Neurol. 363, 642-664. doi: 10.1002/cne.903630408.
18. Carr, D. B., and Sesack, S. R. (2000). GABA-containing neurons in the rat ventral tegmental area project to the prefrontal cortex. Synapse 38, 114-123. doi: 10.1002/1098-2396(200011)38:2<114::AID-SYN2>3.0.CO;2-R.
19. Cheng, K., Saleem, K. S., and Tanaka, K. (1997). Organization of corticostriatal and corticoamygdalar projections arising from the anterior inferotemporal area TE of the macaque monkey: a Phaseolus vulgaris leucoagglutinin study. J. Neurosci. 17, 7902-7925.
20. Chorley, P., and Seth, A. K. (2011). Dopamine-signaled reward predictions generated by competitive excitation and inhibition in a spiking neural network model. Front. Comput. Neurosci. 5:21. doi: 10.3389/fncom.2011.00021.
21. Christian, K. M., and Thompson, R. F. (2003). Neural substrates of eyeblink conditioning: acquisition and retention. Learn. Mem. 10, 427-455. doi: 10.1101/lm.59603.
22. Cohen, M. X., Bour, L., Mantione, M., Figee, M., Vink, M., Tijssen, M. A. J., et al. (2012). Top-down-directed synchrony from medial frontal cortex to nucleus accumbens during reward anticipation. Hum. Brain Mapp. 33, 246-252. doi: 10.1002/hbm.21195.
23. Corbit, L. H., and Balleine, B. W. (2011). The general and outcome-specific forms of Pavlovian-instrumental transfer are differentially mediated by the nucleus accumbens core and shell. J. Neurosci. 31, 11786-11794. doi: 10.1523/JNEUROSCI.2711-11.2011.
24. Coull, J. T., Cheng, R.-K., and Meck, W. H. (2011). Neuroanatomical and neurochemical substrates of timing. Neuropsychopharmacology 36, 3-25. doi: 10.1038/npp.2010.113.
25. Darbaky, Y., Baunez, C., Arecchi, P., Legallet, E., and Apicella, P. (2005). Reward-related neuronal activity in the subthalamic nucleus of the monkey. Neuroreport 16, 1241-1244. doi: 10.1097/00001756-200508010-00022.
26. Daw, N. D., Courville, A. C., Tourtezky, D. S., and Touretzky, D. S. (2006). Representation and timing in theories of the dopamine system. Neural Comput. 18, 1637-1677. doi: 10.1162/neco.2006.18.7.1637.
27. Daw, N. D., Niv, Y., and Dayan, P. (2005). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704-1711. doi: 10.1038/nn1560.
28. Day, J. J., and Carelli, R. M. (2007). The nucleus accumbens and Pavlovian reward learning. Neuroscientist 13, 148-159. doi: 10.1177/1073858406295854.
29. Dayan, P., and Abbott, L. F. (2001). Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems. (Cambridge, MA: The MIT Press). doi: 10.1016/S0306-4522(00)00552-2.
30. Deadwyler, S. A., Hayashizaki, S., Cheer, J., and Hampson, R. E. (2004). Reward, memory and substance abuse: functional neuronal circuits in the nucleus accumbens. Neurosci. Biobehav. Rev. 27, 703-711. doi: 10.1016/j.neubiorev.2003.11.011.
31. Delgado, M. R., Li, J., Schiller, D., and Phelps, E. A. (2008). The role of the striatum in aversive learning and aversive prediction errors. Philos. Trans. R. Soc. Lond. B. Biol. Sci. 363, 3787-3800. doi: 10.1098/rstb.2008.0161.
32. Díaz-Mataix, L., Tallot, L., and Doyère, V. (2013). The amygdala: a potential player in timing CSUS intervals. Behav. Processes. doi: 10.1016/j.beproc.2013.08.007. Available online at: http://www.sciencedirect.com/science/article/pii/S0376635713001824.
33. Dormont, J. F., Condé, H., and Farin, D. (1998). The role of the pedunculopontine tegmental nucleus in relation to conditioned motor performance in the cat. I. Context-dependent and reinforcement-related single unit activity. Exp. Brain Res. 121, 401-410. doi: 10.1007/s002210050474.
34. Doyère, V., Schafe, G. E., Sigurdsson, T., and LeDoux, J. E. (2003). Long-term potentiation in freely moving rats reveals asymmetries in thalamic and cortical inputs to the lateral amygdala. Eur. J. Neurosci. 17, 2703-2715. doi: 10.1046/j.1460-9568.2003.02707.x.
35. Doyere, V., Srebro, B., and Laroche, S. (1997). Heterosynaptic LTD and depotentiation in the medial perforant path of the dentate gyrus in the freely moving rat. J. Neurophysiol. 77, 571-578.
36. Dranias, M. R., Grossberg, S., and Bullock, D. (2008). Dopaminergic and non-dopaminergic value systems in conditioning and outcome-specific revaluation. Brain Res. 1238, 239-287. doi: 10.1016/j.brainres.2008.07.013.
37. Durstewitz, D. (2004). Neural representation of interval time. Neuroreport 15, 745-749. doi: 10.1097/00001756-200404090-00001.
38. Eyny, Y. S., and Horvitz, J. C. (2003). Opposing roles of D1 and D2 receptors in appetitive conditioning. J. Neurosci. 23, 1584-1587.
39. Fiala, J. C., Grossberg, S., and Bullock, D. (1996). Metabotropic glutamate receptor activation in cerebellar Purkinje cells as substrate for adaptive timing of the classically conditioned eye-blink response. J. Neurosci. 16, 3760-3774.
40. Fields, H. L., Hjelmstad, G. O., Margolis, E. B., and Nicola, S. M. (2007). Ventral tegmental area neurons in learned appetitive behavior and positive reinforcement. Annu. Rev. Neurosci. 30, 289-316. doi: 10.1146/annurev.neuro.30.051606.094341.
41. Fino, E., Glowinski, J., and Venance, L. (2005). Bidirectional activity-dependent plasticity at corticostriatal synapses. J. Neurosci. 25, 11279-11287. doi: 10.1523/JNEUROSCI.4476-05.2005.
42. Fiorillo, C. D., Newsome, W. T., and Schultz, W. (2008). The temporal precision of reward prediction in dopamine neurons. Nat. Neurosci. 11, 966-973. doi: 10.1038/nn.2159.
43. Fiorillo, C. D., Tobler, P. N., and Schultz, W. (2003). Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299, 1898-1902. doi: 10.1126/science.1077349.
44. Fudge, J. L., and Haber, S. N. (2000). The central nucleus of the amygdala projection to dopamine subpopulations in primates. Neuroscience 97, 479-494. doi: 10.1016/S0306-4522(00)00092-0.
45. Funahashi, S., Chafee, M. V., and Goldman-Rakic, P. S. (1993). Prefrontal neuronal activity in rhesus monkeys performing a delayed anti-saccade task. Nature 365, 753-756. doi: 10.1038/365753a0.
46. Gallistel, C. R., and Gibbon, J. (2000). Time, rate, and conditioning. Psychol. Rev. 107, 289-344. doi: 10.1037/0033-295X.107.2.289.
47. Galtress, T., and Kirkpatrick, K. (2009). Reward value effects on timing in the peak procedure. Learn. Motiv. 40, 109-131. doi: 10.1016/j.lmot.2008.05.004.
48. Galtress, T., and Kirkpatrick, K. (2010). The role of the nucleus accumbens core in impulsive choice, timing, and reward processing. Behav. Neurosci. 124, 26-43. doi: 10.1037/a0018464.
49. Galtress, T., Marshall, A. T., and Kirkpatrick, K. (2012). Motivation and timing: clues for modeling the reward system. Behav. Processes 90, 142-153. doi: 10.1016/j.beproc.2012.02.014.
50. Geisler, S., Derst, C., Veh, R. W., and Zahm, D. S. (2007). Glutamatergic afferents of the ventral tegmental area in the rat. J. Neurosci. 27, 5730-5743. doi: 10.1523/JNEUROSCI.0012-07.2007.
51. Geisler, S., and Wise, R. A. (2008). Functional implications of glutamatergic projections to the ventral tegmental area. Rev. Neurosci. 19, 227-244. doi: 10.1515/REVNEURO.2008.19.4-5.227.
52. Goldman-Rakic, P. S., Lidow, M. S., Smiley, J. F., and Williams, M. S. (1992). The anatomy of dopamine in monkey and human prefrontal cortex. J. Neural Transm. Suppl. 36, 163-177. doi: 10.1007/978-3-7091-9211-5_8.
53. Goto, Y., and Grace, A. A. (2005). Dopaminergic modulation of limbic and cortical drive of nucleus accumbens in goal-directed behavior. Nat. Neurosci. 8, 805-812. doi: 10.1038/nn1471.
54. Groshek, F., Kerfoot, E., McKenna, V., Polackwich, A. S., Gallagher, M., and Holland, P. C. (2005). Amygdala central nucleus function is necessary for learning, but not expression, of conditioned auditory orienting. Behav. Neurosci. 119, 202-212. doi: 10.1037/0735-7044.119.1.202.
55. Grossberg, S., and Schmajuk, N. A. (1989). Neural dynamics of adaptive timing and temporal discrimination during associative learning. Neural Netw. 2, 79-102. doi: 10.1016/0893-6080(89)90026-9.
56. Gruber, A. J., Solla, S. A., Surmeier, D. J., and Houk, J. C. (2003). Modulation of striatal single units by expected reward: a spiny neuron model displaying dopamine-induced bistability. J. Neurophysiol. 90, 1095-1114. doi: 10.1152/jn.00618.2002.
57. Haber, S. N. (2003). The primate basal ganglia: parallel and integrative networks. J. Chem. Neuroanat. 26, 317-330. doi: 10.1016/j.jchemneu.2003.10.003.
58. Haber, S. N., Fudge, J. L., and McFarland, N. R. (2000). Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. J. Neurosci. 20, 2369-2382.
59. Haber, S. N., and Knutson, B. (2010). The reward circuit: linking primate anatomy and human imaging. Neuropsychopharmacology 35, 4-26. doi: 10.1038/npp.2009.129.
60. Hallanger, A. E., and Wainer, B. H. (1988). Ascending projections from the pedunculopontine tegmental nucleus and the adjacent mesopontine tegmentum in the rat. J. Comp. Neurol. 274, 483-515. doi: 10.1002/cne.902740403.
61. Hazy, T. E., Frank, M. J., and O'Reilly, R. C. (2010). Neural mechanisms of acquired phasic dopamine responses in learning. Neurosci. Biobehav. Rev. 34, 701-720. doi: 10.1016/j.neubiorev.2009.11.019.
62. Hikosaka, O., Sesack, S. R., Lecourtier, L., and Shepard, P. D. (2008). Habenula: crossroad between the basal ganglia and the limbic system. J. Neurosci. 28, 11825-11829. doi: 10.1523/JNEUROSCI.3463-08.2008.
63. Hochreiter, S., and Schmidhuber, J. (1997). Long short-term memory. Neural Comput. 9, 1735-1780. doi: 10.1162/neco.1997.9.8.1735.
64. Holland, P. C., and Gallagher, M. (2004). Amygdala frontal interactions and reward expectancy. Curr. Opin. Neurobiol. 14, 148-155. doi: 10.1016/j.conb.2004.03.007.
65. Hollerman, J. R., and Schultz, W. (1998). Dopamine neurons report an error in the temporal prediction of reward during learning. Nat. Neurosci. 1, 304-309. doi: 10.1038/1124.
66. Hong, S., and Hikosaka, O. (2008). The globus pallidus sends reward-related signals to the lateral habenula. Neuron 60, 720-729. doi: 10.1016/j.neuron.2008.09.035.
67. Hong, S., Jhou, T. C., Smith, M., Saleem, K. S., and Hikosaka, O. (2011). Negative reward signals from the lateral habenula to dopamine neurons are mediated by rostromedial tegmental nucleus in primates. J. Neurosci. 31, 11457-11471. doi: 10.1523/JNEUROSCI.1384-11.2011.
68. Horvitz, J. C. (2000). Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events. Neuroscience 96, 651-656. doi: 10.1016/S0306-4522(00)00019-1.
69. Humphries, M. D., Lepora, N., Wood, R., and Gurney, K. (2009). Capturing dopaminergic modulation and bimodal membrane behaviour of striatal medium spiny neurons in accurate, reduced models. Front. Comput. Neurosci. 3:26. doi: 10.3389/neuro.10.026.2009.
70. Humphries, M. D., and Prescott, T. J. (2010). The ventral basal ganglia, a selection mechanism at the crossroads of space, strategy, and reward. Prog. Neurobiol. 90, 385-417. doi: 10.1016/j.pneurobio.2009.11.003.
71. Ito, R., Robbins, T. W., McNaughton, B. L., and Everitt, B. J. (2006). Selective excitotoxic lesions of the hippocampus and basolateral amygdala have dissociable effects on appetitive cue and place conditioning based on path integration in a novel Y-maze procedure. Eur. J. Neurosci. 23, 3071-3080. doi: 10.1111/j.1460-9568.2006.04883.x.
72. Izhikevich, E. M. (2007). Solving the distal reward problem through linkage of STDP and dopamine signaling. Cereb. Cortex 17, 2443-2452. doi: 10.1093/cercor/bhl152.
73. Jhou, T. C., Fields, H. L., Baxter, M. G., Saper, C. B., and Holland, P. C. (2009). The rostromedial tegmental nucleus (RMTg), a GABAergic afferent to midbrain dopamine neurons, encodes aversive stimuli and inhibits motor responses. Neuron 61, 786-800. doi: 10.1016/j.neuron.2009.02.001.
74. Judice-Daher, D. M., and Bueno, J. L. O. (2013). Lesions of the nucleus accumbens disrupt reinforcement omission effects in rats. Behav. Brain Res. 252, 439-443. doi: 10.1016/j.bbr.2013.06.028.
75. Kirkpatrick, K. (2013). Interactions of timing and prediction error learning. Behav. Processes pii: S0376-6357(13)00180-0. doi: 10.1016/j.beproc.2013.08.005.
76. Kobayashi, Y., and Okada, K.-I. (2007). Reward prediction error computation in the pedunculopontine tegmental nucleus neurons. Ann. N. Y. Acad. Sci. 1104, 310-323. doi: 10.1196/annals.1390.003.
77. Komura, Y., Tamura, R., Uwano, T., Nishijo, H., Kaga, K., and Ono, T. (2001). Retrospective and prospective coding for predicted reward in the sensory thalamus. Nature 412, 546-549. doi: 10.1038/35087595.
78. Koo, J. W., Han, J.-S., and Kim, J. J. (2004). Selective neurotoxic lesions of basolateral and central nuclei of the amygdala produce differential effects on fear conditioning. J. Neurosci. 24, 7654-7662. doi: 10.1523/JNEUROSCI.1644-04.2004.
79. Krichmar, J. L. (2013). A neurorobotic platform to test the influence of neuromodulatory signaling on anxious and curious behavior. Front. Neurorobot. 7:1. doi: 10.3389/fnbot.2013.00001.
80. Lammel, S., Lim, B. K., Ran, C., Huang, K. W., Betley, M. J., Tye, K. M., et al. (2012). Input-specific control of reward and aversion in the ventral tegmental area. Nature 491, 212-217. doi: 10.1038/nature11527.
81. Lardeux, S., Pernaud, R., Paleressompoulle, D., and Baunez, C. (2009). Beyond the reward pathway: coding reward magnitude and error in the rat subthalamic nucleus. J. Neurophysiol. 102, 2526-2537. doi: 10.1152/jn.91009.2008.
82. Lavezzi, H. N., and Zahm, D. S. (2011). The mesopontine rostromedial tegmental nucleus: an integrative modulator of the reward system. Basal Ganglia 1, 191-200. doi: 10.1016/j.baga.2011.08.003.
83. LeDoux, J. E. (2000). Emotion circuits in the brain. Annu. Rev. Neurosci. 23, 155-184. doi: 10.1146/annurev.neuro.23.1.155.
84. Lee, H. J., Wheeler, D. S., and Holland, P. C. (2011). Interactions between amygdala central nucleus and the ventral tegmental area in the acquisition of conditioned cue-directed behavior in rats. Eur. J. Neurosci. 33, 1876-1884. doi: 10.1111/j.1460-9568.2011.07680.x.
85. Leung, L. S., and Yim, C. Y. (1993). Rhythmic delta-frequency activities in the nucleus accumbens of anesthetized and freely moving rats. Can. J. Physiol. Pharmacol. 71, 311-320. doi: 10.1139/y93-049.
86. Ljungberg, T., Apicella, P., and Schultz, W. (1992). Responses of monkey dopamine neurons during learning of behavioral reactions. J. Neurophysiol. 67, 145-163.
87. Lokwan, S. J., Overton, P. G., Berry, M. S., and Clark, D. (1999). Stimulation of the pedunculopontine tegmental nucleus in the rat produces burst firing in A9 dopaminergic neurons. Neuroscience 92, 245-254. doi: 10.1016/S0306-4522(98)00748-9.
88. Ludvig, E. A., Conover, K., and Shizgal, P. (2007). The effects of reinforcer magnitude on timing in rats. J. Exp. Anal. Behav. 87, 201-218. doi: 10.1901/jeab.2007.38-06.
89. Ludvig, E. A., Sutton, R. S., and Kehoe, E. J. (2008). Stimulus representation and the timing of reward-prediction errors in models of the dopamine system. Neural Comput. 20, 3034-3054. doi: 10.1162/neco.2008.11-07-654.
90. Ludvig, E. A., Sutton, R., Verbeek, E., and James Kehoe, E. (2009). "A computational model of hippocampal function in trace conditioning," in Advances in Neural Information Processing Systems 21, eds D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou (Vancouver), 993-1000.
91. Lustig, C., Matell, M. S., and Meck, W. H. (2005). Not "just" a coincidence: frontal-striatal interactions in working memory and interval timing. Memory 3/4, 441-448. Available online at: http://www.ncbi.nlm.nih.gov/pubmed?Db=pubmed&Cmd=Retrieve&list_uids=15952263&dopt=abstractplus
92. Luzardo, A., Ludvig, E. A., and Rivest, F. (2013). An adaptive drift-diffusion model of interval timing dynamics. Behav. Processes 95, 90-99. doi: 10.1016/j.beproc.2013.02.003.
93. Maren, S., and Quirk, G. J. (2004). Neuronal signalling of fear memory. Nat. Rev. Neurosci. 5, 844-852. doi: 10.1038/nrn1535.
94. Martin-Soelch, C., Linthicum, J., and Ernst, M. (2007). Appetitive conditioning: neural bases and implications for psychopathology. Neurosci. Biobehav. Rev. 31, 426-440. doi: 10.1016/j.neubiorev.2006.11.002.
95. Matell, M. S., and Meck, W. H. (2000). Neuropsychological mechanisms of interval timing behavior. Bioessays 22, 94-103. doi: 10.1002/(SICI)1521-1878(200001)22:1<94::AID-BIES14>3.0.CO;2-E.
96. Matell, M. S., and Meck, W. H. (2004). Cortico-striatal circuits and interval timing: coincidence detection of oscillatory processes. Cogn. Brain Res. 21, 139-170. doi: 10.1016/j.cogbrainres.2004.06.012.
97. Matsumoto, M., and Hikosaka, O. (2007). Lateral habenula as a source of negative reward signals in dopamine neurons. Nature 447, 1111-1115. doi: 10.1038/nature05860.
98. Matsumoto, M., and Hikosaka, O. (2009). Two types of dopamine neuron distinctly convey positive and negative motivational signals. Nature 459, 837-841. doi: 10.1038/nature08028.
99. McClure, S. M., Berns, G. S., and Montague, P. R. (2003). Temporal prediction errors in a passive learning task activate human striatum. Neuron 38, 339-346. doi: 10.1016/S0896-6273(03)00154-5.
100. McDannald, M., Kerfoot, E., Gallagher, M., and Holland, P. C. (2004). Amygdala central nucleus function is necessary for learning but not expression of conditioned visual orienting. Eur. J. Neurosci. 20, 240-248. doi: 10.1111/j.0953-816X.2004.03458.x.
101. McGinty, V. B., and Grace, A. A. (2009). Activity-dependent depression of medial prefrontal cortex inputs to accumbens neurons by the basolateral amygdala. Neuroscience 162, 1429-1436. doi: 10.1016/j.neuroscience.2009.05.028.
102. Meck, W. H. (2006). Neuroanatomical localization of an internal clock: a functional link between mesolimbic, nigrostriatal, and mesocortical dopaminergic systems. Brain Res. 1109, 93-107. doi: 10.1016/j.brainres.2006.06.031.
103. Mirenowicz, J., and Schultz, W. (1994). Importance of unpredictability for reward responses in primate dopamine neurons. J. Neurophysiol. 72, 1024-1027.
104. Montague, P. R., Dayan, P., and Sejnowski, T. J. (1996). A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936-1947.
105. Morris, R. W., and Bouton, M. E. (2006). Effect of unconditioned stimulus magnitude on the emergence of conditioned responding. J. Exp. Psychol. Anim. Behav. Process. 32, 371-385. doi: 10.1037/0097-7403.32.4.371.
106. Muller, J. F., Mascagni, F., and McDonald, A. J. (2007). Postsynaptic targets of somatostatin-containing interneurons in the rat basolateral amygdala. J. Comp. Neurol. 500, 513-529. doi: 10.1002/cne.21185.
107. Murray, E. A. (2007). The amygdala, reward and emotion. Trends Cogn. Sci. 11, 489-497. doi: 10.1016/j.tics.2007.08.013.
108. Nakamura, K., and Ono, T. (1986). Lateral hypothalamus neuron involvement in integration of natural and artificial rewards and cue signals. J. Neurophysiol. 55, 163-181.
109. Nicola, S. M. (2007). The nucleus accumbens as part of a basal ganglia action selection circuit. Psychopharmacology (Berl). 191, 521-550. doi: 10.1007/s00213-006-0510-4.
110. Nishijo, H., Hori, E., Tazumi, T., and Ono, T. (2008). Neural correlates to both emotion and cognitive functions in the monkey amygdala. Behav. Brain Res. 188, 14-23. doi: 10.1016/j.bbr.2007.10.013.
111. Nishijo, H., Ono, T., Uwano, T., Kondoh, T., and Torii, K. (2000). Hypothalamic and amygdalar neuronal responses to various tastant solutions during ingestive behavior in rats. J. Nutr. 130, 954S-959S. Available online at: http://jn.nutrition.org/content/130/4/954.long.
112. Niv, Y., Daw, N. D., Joel, D., and Dayan, P. (2007). Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology (Berl). 191, 507-520. doi: 10.1007/s00213-006-0502-4.
113. O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H., and Dolan, R. J. (2003). Temporal difference models and reward-related learning in the human brain. Neuron 38, 329-337. doi: 10.1016/S0896-6273(03)00169-7.
114. O'Donnell, P., and Grace, A. A. (1995). Synaptic interactions among excitatory afferents to nucleus accumbens neurons: hippocampal gating of prefrontal cortical input. J. Neurosci. 15, 3622-3639.
115. Oja, E. (1982). A simplified neuron model as a principal component analyzer. J. Math. Biol. 15, 267-273. doi: 10.1007/BF00275687.
116. Ono, T., Nishijo, H., and Uwano, T. (1995). Amygdala role in conditioned associative learning. Prog. Neurobiol. 46, 401-422. doi: 10.1016/0301-0082(95)00008-J.
117. Oprisan, S. A., and Buhusi, C. V. (2011). Modeling pharmacological clock and memory patterns of interval timing in a striatal beat-frequency model with realistic, noisy neurons. Front. Integr. Neurosci. 5:52. doi: 10.3389/fnint.2011.00052.
118. O'Reilly, R. C., and Frank, M. J. (2006). Making working memory work: a computational model of learning in the prefrontal cortex and basal ganglia. Neural Comput. 18, 283-328. doi: 10.1162/089976606775093909.
119. O'Reilly, R. C., Frank, M. J., Hazy, T. E., and Watz, B. (2007). PVLV: the primary value and learned value Pavlovian learning algorithm. Behav. Neurosci. 121, 31-49. doi: 10.1037/0735-7044.121.1.31.
120. Pan, W.-X., and Hyland, B. I. (2005). Pedunculopontine tegmental nucleus controls conditioned responses of midbrain dopamine neurons in behaving rats. J. Neurosci. 25, 4725-4732. doi: 10.1523/JNEUROSCI.0277-05.2005.
121. Pan, W.-X., Schmidt, R., Wickens, J. R., and Hyland, B. I. (2005). Dopamine cells respond to predicted events during classical conditioning: evidence for eligibility traces in the reward-learning network. J. Neurosci. 25, 6235-6242. doi: 10.1523/JNEUROSCI.1478-05.2005.
122. Pape, H.-C., and Pare, D. (2010). Plastic synaptic networks of the amygdala for the acquisition, expression, and extinction of conditioned fear. Physiol. Rev. 90, 419-463. doi: 10.1152/physrev.00037.2009.
123. Rao, R. P. N. (2010). Decision making under uncertainty: a neural model based on partially observable markov decision processes. Front. Comput. Neurosci. 4:146. doi: 10.3389/fncom.2010.00146.
124. Raybuck, J. D., and Lattal, K. M. (2013). Bridging the interval: theory and neurobiology of trace conditioning. Behav. Processes pii: S0376-6357(13)00191-5. doi: 10.1016/j.beproc.2013.08.016.
125. Redgrave, P., Gurney, K., and Reynolds, J. (2008). What is reinforced by phasic dopamine signals? Brain Res. Rev. 58, 322-339. doi: 10.1016/j.brainresrev.2007.10.007.
126. Reutimann, J., Yakovlev, V., Fusi, S., and Senn, W. (2004). Climbing neuronal activity as an event-based cortical representation of time. J. Neurosci. 24, 3295-3303. doi: 10.1523/JNEUROSCI.4098-03.2004.
127. Reynolds, J. N., and Wickens, J. R. (2002). Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw. 15, 507-521. doi: 10.1016/S0893-6080(02)00045-X.
128. Rivest, F., Kalaska, J. F., and Bengio, Y. (2010). Alternative time representation in dopamine models. J. Comput. Neurosci. 28, 107-130. doi: 10.1007/s10827-009-0191-1.
129. Rivest, F., Kalaska, J. F., and Bengio, Y. (2013). Conditioning and time representation in long short-term memory networks. Biol. Cybern. doi: 10.1007/s00422-013-0575-1. Available online at: http://link.springer.com/article/10.1007%2Fs00422-013-0575-1.
130. Robbins, T. W., and Everitt, B. J. (1996). Neurobehavioural mechanisms of reward and motivation. Curr. Opin. Neurobiol. 6, 228-236. doi: 10.1016/S0959-4388(96)80077-8.
131. Rose, J., Schmidt, R., Grabemann, M., and Güntürkün, O. (2009). Theory meets pigeons: the influence of reward-magnitude on discrimination-learning. Behav. Brain Res. 198, 125-129. doi: 10.1016/j.bbr.2008.10.038.
132. Sah, P., Faber, E. S. L., Lopez De Armentia, M., and Power, J. (2003). The amygdaloid complex: anatomy and physiology. Physiol. Rev. 83, 803-834. doi: 10.1152/physrev.00002.2003.
133. Samejima, K., and Doya, K. (2007). Multiple representations of belief states and action values in corticobasal ganglia loops. Ann. N. Y. Acad. Sci. 1104, 213-228. doi: 10.1196/annals.1390.024.
134. Schroll, H., Vitay, J., and Hamker, F. H. (2012). Working memory and response selection: a computational account of interactions among cortico-basalganglio-thalamic loops. Neural Netw. 26, 59-74. doi: 10.1016/j.neunet.2011.10.008.
135. Schultz, W., Apicella, P., and Ljungberg, T. (1993). Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J. Neurosci. 13, 900-913.
136. Schultz, W., Apicella, P., Scarnati, E., and Ljungberg, T. (1992). Neuronal activity in monkey ventral striatum related to the expectation of reward. J. Neurosci. 12, 4595-4610.
137. Schultz, W., Dayan, P., and Montague, P. R. (1997). A neural substrate of prediction and reward. Science 275, 1593-1599. doi: 10.1126/science.275.5306.1593.
138. Seamans, J. K., and Yang, C. R. (2004). The principal features and mechanisms of dopamine modulation in the prefrontal cortex. Prog. Neurobiol. 74, 1-58. doi: 10.1016/j.pneurobio.2004.10.002.
139. Semba, K., and Fibiger, H. C. (1992). Afferent connections of the laterodorsal and the pedunculopontine tegmental nuclei in the rat: a retro- and antero-grade transport and immunohistochemical study. J. Comp. Neurol. 323, 387-410. doi: 10.1002/cne.903230307.
140. Sesack, S. R., and Grace, A. A. (2010). Cortico-Basal Ganglia reward network: microcircuitry. Neuropsychopharmacology 35, 27-47. doi: 10.1038/npp.2009.93.
141. Shen, W., Flajolet, M., Greengard, P., and Surmeier, D. J. (2008). Dichotomous dopaminergic control of striatal synaptic plasticity. Science 321, 848-851. doi: 10.1126/science.1160575.
142. Simen, P., Balci, F., de Souza, L., Cohen, J. D., and Holmes, P. (2011). A model of interval timing by neural integration. J. Neurosci. 31, 9238-9253. doi: 10.1523/JNEUROSCI.3121-10.2011.
143. Singh, T., McDannald, M. A., Takahashi, Y. K., Haney, R. Z., Cooch, N. K., Lucantonio, F., et al. (2011). The role of the nucleus accumbens in knowing when to respond. Learn. Mem. 18, 85-87. doi: 10.1101/lm.2008111.
144. Smith, K. S., Tindell, A. J., Aldridge, J. W., and Berridge, K. C. (2009). Ventral pallidum roles in reward and motivation. Behav. Brain Res. 196, 155-167. doi: 10.1016/j.bbr.2008.09.038.
145. Sporns, O., and Alexander, W. H. (2002). Neuromodulation and plasticity in an autonomous robot. Neural Netw. 15, 761-774. doi: 10.1016/S0893-6080(02)00062-X.
146. Stopper, C. M., and Floresco, S. B. (2011). Contributions of the nucleus accumbens and its subregions to different aspects of risk-based decision making. Cogn. Affect. Behav. Neurosci. 11, 97-112. doi: 10.3758/s13415-010-0015-9.
147. Suri, R. E., and Schultz, W. (1999). A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task. Neuroscience 91, 871-890. doi: 10.1016/S0306-4522(98)00697-6.
148. Suri, R. E., and Schultz, W. (2001). Temporal difference model reproduces anticipatory neural activity. Neural Comput. 13, 841-862. doi: 10.1162/089976601300014376.
149. Sutton, R. S., and Barto, A. G. (1981). Toward a modern theory of adaptive networks: expectation and prediction. Psychol. Rev. 88, 135-170. doi: 10.1037/0033-295X.88.2.135.
150. Sutton, R. S., and Barto, A. G. (1998). Reinforcement Learning: An Introduction, Vol. 28. Cambridge, MA: MIT Press.
151. Tachibana, Y., and Hikosaka, O. (2012). The primate ventral pallidum encodes expected reward value and regulates motor action. Neuron 76, 826-837. doi: 10.1016/j.neuron.2012.09.030.
152. Tan, C. O., and Bullock, D. (2008). A local circuit model of learned striatal and dopamine cell responses under probabilistic schedules of reward. J. Neurosci. 28, 10062-10074. doi: 10.1523/JNEUROSCI.0259-08.2008.
153. Tanaka, K. (2000). Mechanisms of visual object recognition studied in monkeys. Spat. Vis. 13, 147-163. doi: 10.1163/156856800741171.
154. Thompson, R. F., and Steinmetz, J. E. (2009). The role of the cerebellum in classical conditioning of discrete behavioral responses. Neuroscience 162, 732-755. doi: 10.1016/j.neuroscience.2009.01.041.
155. Tindell, A. J., Berridge, K. C., and Aldridge, J. W. (2004). Ventral pallidal representation of pavlovian cues and reward: population and rate codes. J. Neurosci. 24, 1058-1069. doi: 10.1523/JNEUROSCI.1437-03.2004.
156. Tobler, P. N., Fiorillo, C. D., and Schultz, W. (2005). Adaptive coding of reward value by dopamine neurons. Science 307, 1642-1645. doi: 10.1126/science.1105370.
157. Turrigiano, G. G. (2008). The self-tuning neuron: synaptic scaling of excitatory synapses. Cell 135, 422-435. doi: 10.1016/j.cell.2008.10.008.
158. Tye, K. M., Cone, J. J., Schairer, W. W., and Janak, P. H. (2010). Amygdala neural encoding of the absence of reward during extinction. J. Neurosci. 30, 116-125. doi: 10.1523/JNEUROSCI.4240-09.2010.
159. Usuda, I., Tanaka, K., and Chiba, T. (1998). Efferent projections of the nucleus accumbens in the rat with special reference to subdivision of the nucleus: biotinylated dextran amine study. Brain Res. 797, 73-93. doi: 10.1016/S0006-8993(98)00359-X.
160. Vitay, J., and Hamker, F. H. (2010). A computational model of Basal Ganglia and its role in memory retrieval in rewarded visual memory tasks. Front. Comput. Neurosci. 4:13. doi: 10.3389/fncom.2010.00013.
161. Walker, A. G., and Steinmetz, J. E. (2008). Hippocampal lesions in rats differentially affect long- and short-trace eyeblink conditioning. Physiol. Behav. 93, 570-578. doi: 10.1016/j.physbeh.2007.10.018.
162. Winstanley, C. A., Baunez, C., Theobald, D. E. H., and Robbins, T. W. (2005). Lesions to the subthalamic nucleus decrease impulsive choice but impair autoshaping in rats: the importance of the basal ganglia in Pavlovian conditioning and impulse control. Eur. J. Neurosci. 21, 3107-3116. doi: 10.1111/j.1460-9568.2005.04143.x.
163. Wolf, J. A., Moyer, J. T., Lazarewicz, M. T., Contreras, D., Benoit-Marand, M., O'Donnell, P., et al. (2005). NMDA/AMPA ratio impacts state transitions and entrainment to oscillations in a computational model of the nucleus accumbens medium spiny projection neuron. J. Neurosci. 25, 9080-9095. doi: 10.1523/JNEUROSCI.2220-05.2005.
164. Wolf, M. E., Sun, X., Mangiavacchi, S., and Chao, S. Z. (2004). Psychomotor stimulants and neuronal plasticity. Neuropharmacology 47, 61-79. doi: 10.1016/j.neuropharm.2004.07.006.
165. Wu, G.-Y., Yao, J., Hu, B., Zhang, H.-M., Li, Y.-D., Li, X., et al. (2013). Reevaluating the role of the hippocampus in delay eyeblink conditioning. PLoS ONE 8:e71249. doi: 10.1371/journal.pone.0071249.
166. Zahm, D. S., and Heimer, L. (1990). Two transpallidal pathways originating in the rat nucleus accumbens. J. Comp. Neurol. 302, 437-446. doi: 10.1002/cne.903020302