Alternative time representation in dopamine models

Journal of Computational Neuroscience

Volumn 28, Issue 1, 2010, Pages 107-130

  Rivest, François ^a   Kalaska, John F ^a   Bengio, Yoshua ^a  

^a UNIVERSITÉ DE MONTRÉAL   (Canada)

Author keywords

Dopamine; Interval timing; Reinforcement learning; Representation learning; Reward; Trace conditioning

Indexed keywords

AMINES; BRAIN; DELAY CIRCUITS; DYNAMICS; LONG SHORT-TERM MEMORY; NEURONS; TIMING CIRCUITS;

DOPAMINE; DOPAMINERGIC NEURONS; INTERVAL-TIMING; LEARN+; NEURON ACTIVITY; REINFORCEMENT LEARNINGS; REPRESENTATION LEARNING; REWARD; TEMPORAL-DIFFERENCE ALGORITHM; TRACE CONDITIONING;

REINFORCEMENT LEARNING;

DOPAMINE;

ARTICLE; DOPAMINERGIC ACTIVITY; DOPAMINERGIC NERVE CELL; EXPERIENCE; LEARNING; LONG TERM MEMORY; NERVE CELL NETWORK; NETWORK LEARNING; PRIORITY JOURNAL; REINFORCEMENT; SHORT TERM MEMORY; STATISTICAL ANALYSIS; TASK PERFORMANCE; TEMPORAL DIFFERENCE;

EID: 77549088745     PISSN: 09295313     EISSN: None     Source Type: Journal    
DOI: 10.1007/s10827-009-0191-1     Document Type: Article

References (75)

1. Bakker, B. (2002). Reinforcement learning with long short-term memory. In T. G. Dietterich, S. Becker & Z. Ghahramani (Eds.), Neural information processing systems (pp. 1475-1482). Cambridge: MIT.
2. PD Balsam MR Drew C Yang 2002 Timing at the start of associative learning Learning and Motivation 33 141 155 10.1006/lmot.2001.1104
3. Barto, A. G. (1995). Adaptive critics and the basal ganglia. In J. C. Houk, J. L. Davis, & D. G. Beiser (Eds.), Models of information processing in the basal ganglia (pp. 215-232). The MIT.
4. M Bertin N Schweighofer K Doya 2007 Multiple model-based reinforcement learning explains dopamine neuronal activity Neural Networks 20 668 675 10.1016/j.neunet.2007.04.028 17611074 (Pubitemid 47241319)
5. AV Beylin CC Gandhi GE Wood AC Talk LD Matzel TJ Shors 2001 The role of the hippocampus in trace conditioning: temporal discontinuity or task difficulty? Neurobiology of Learning and Memory 76 447 461 10.1006/nlme.2001. 4039 1:STN:280:DC%2BD3MnotVKkug%3D%3D 11726247 (Pubitemid 34001411)
6. Braver, T. S., & Cohen, J. D. (2000). On the control of control: The role of dopamine in regulating prefrontal function and working memory. In S. Monsell & J. Driver (Eds.), Control of cognitive processes: Attention and performance XVIII (pp. 713-737). Cambridge: MIT.
7. CD Brody A Hernandez A Zainos R Romo 2003 Timing and neural encoding of somatosensory parametric working memory in macaque prefrontal cortex Cerebral Cortex 13 1196 1207 10.1093/cercor/bhg100 14576211 (Pubitemid 37322996)
8. J Brown D Bullock S Grossberg 1999 How the basal ganglia use parallel excitatory and inhibitory learning pathways to selectively respond to unexpected rewarding cues Journal of Neuroscience 19 10502 10511 1:CAS:528: DyaK1MXnslyrtbk%3D 10575046
9. CV Buhusi WH Meck 2000 Timing for the absence of a stimulus: the gap paradigm reversed Journal of Experimental Psychology: Animal Behavior Processes 26 305 322 10.1037/0097-7403.26.3.305 1:STN:280:DC%2BD3M%2FislKkuw%3D%3D 10913995
10. CV Buhusi WH Meck 2005 What makes us tick? Functional and neural mechanisms of interval timing Nature Reviews. Neuroscience 6 755 765 10.1038/nrn1764 1:CAS:528:DC%2BD2MXhtV2jsL3N 16163383 (Pubitemid 41679238)
11. Church, R. M. (2003). A concise introduction to scalar timing theory. In W. H. Meck (Ed.), Functional and neural mechanisms of interval timing (pp. 3-22). Boca Raton: CRC.
12. RE Clark LR Squire 1998 Classical conditioning and brain systems: the role of awareness Science 280 77 81 10.1126/science.280.5360.77 1:CAS:528:DyaK1cXitlWks7w%3D 9525860 (Pubitemid 28169083)
13. Daw, N. D., & Doya, K. (2006). The computational neurobiology of learning and reward. Curr. Opin. Neurobiol.
14. Daw, N. D., Courville, A. C., & Touretzky, D. S. (2003). Timing and partial observability in the dopamine system. In S. Becker, S. Thrun & K. Obermayer (Eds.), Neural information processing systems (pp. 83-90). Cambridge: MIT.
15. ND Daw AC Courville DS Touretzky 2006 Representation and timing in theories of the dopamine system Neural Computation 18 1637 1677 10.1162/neco.2006.18.7.1637 16764517 (Pubitemid 44024733)
16. JF Dormont H Conde D Farin 1998 The role of the pedunculopontine tegmental nucleus in relation to conditioned motor performance in the cat. I. Context-dependent and reinforcement-related single unit activity Experimental Brain Research 121 401 410 10.1007/s002210050474 1:STN:280:DyaK1cvhslahug%3D%3D (Pubitemid 28370634)
17. K Doya 1999 What are the computations of the cerebellum, the basal ganglia and the cerebral cortex? Neural Networks 12 961 974 10.1016/S0893- 6080(99)00046-5 12662639 (Pubitemid 29389710)
18. K Doya 2000 Complementary roles of basal ganglia and cerebellum in learning and motor control Current Opinion in Neurobiology 10 732 739 10.1016/S0959-4388(00)00153-7 1:CAS:528:DC%2BD3MXkvVegsQ%3D%3D 11240282 (Pubitemid 32052539)
19. V Dragoi JE Staddon RG Palmer CV Buhusi 2003 Interval timing as an emergent learning property Psychological Review 110 126 144 10.1037/0033-295X. 110.1.126 12529059
20. D Durstewitz 2004 Neural representation of interval time NeuroReport 15 745 749 10.1097/00001756-200404090-00001 15073507 (Pubitemid 38453450)
21. D Eck A Schmidhuber 2002 Learning the long-term structure of the blues Artificial Neural Networks-Icann 2002 2415 284 289
22. CD Fiorillo PN Tobler W Schultz 2003 Discrete coding of reward probability and uncertainty by dopamine neurons Science 299 1898 1902 10.1126/science.1077349 1:CAS:528:DC%2BD3sXitFKksr0%3D 12649484 (Pubitemid 36356659)
23. CD Fiorillo WT Newsome W Schultz 2008 The temporal precision of reward prediction in dopamine neurons Nature Neuroscience 11 966 973 10.1038/nn.2159 1:CAS:528:DC%2BD1cXptVWmu7g%3D
24. RV Florian 2007 Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity Neural Computation 19 1468 1502 10.1162/neco.2007.19.6.1468 17444757
25. M Fukuda T Ono H Nishino K Nakamura 1986 Neuronal responses in monkey lateral hypothalamus during operant feeding behavior Brain Research Bulletin 17 879 883 10.1016/0361-9230(86)90102-4 1:STN:280:DyaL2s7gt1Wlug%3D%3D 3801942 (Pubitemid 17215235)
26. M Fukuda T Ono K Nakamura R Tamura 1990 Dopamine and ACh involvement in plastic learning by hypothalamic neurons in rats Brain Research Bulletin 25 109 114 10.1016/0361-9230(90)90260-7 1:CAS:528:DyaK3cXls1GqtrY%3D 2207696 (Pubitemid 20246622)
27. S Funahashi CJ Bruce PS Goldman-Rakic 1989 Mnemonic coding of visual space in the monkey's dorsolateral prefrontal cortex Journal of Neurophysiology 61 331 349 1:STN:280:DyaL1M7ktV2rtQ%3D%3D 2918358
28. CR Gallistel J Gibbon 2000 Time, rate, and conditioning Psychological Review 107 289 344 10.1037/0033-295X.107.2.289 1:STN:280:DC%2BD3c3kvFKktg%3D%3D 10789198
29. FA Gers J Schmidhuber F Cummins 2000 Learning to forget: continual prediction with LSTM Neural Computation 12 2451 2471 10.1162/089976600300015015 1:STN:280:DC%2BD3cvnvFCnug%3D%3D 11032042
30. FA Gers NN Schraudolph J Schmidhuber 2002 Learning precise timing with LSTM recurrent networks Journal of Machine Learning Research 3 115 143 10.1162/153244303768966139
31. S Hochreiter J Schmidhuber 1997 Long short-term memory Neural Computation 9 1735 1780 10.1162/neco.1997.9.8.1735 1:STN:280:DyaK1c%2FhvVahsQ%3D%3D 9377276
32. JR Hollerman W Schultz 1998 Dopamine neurons report an error in the temporal prediction of reward during learning Nature Neuroscience 1 304 309 10.1038/1124 1:CAS:528:DyaK1MXitVCgurg%3D 10195164
33. Hopson, J. W. (2003). General learning models: Timing without a clock. In W. H. Meck (Ed.), Functional and neural mechanisms of interval timing (pp. 23-60). Boca Raton: CRC.
34. Houk, J. C., Adams, J. L., & Barto, A. G. (1995). A model of how the basal ganglia generate and use neural signals that predict reinforcement. In J. C. Houk, J. L. Davis, & D. G. Beiser (Eds.), Models of information processing in the basal ganglia (pp. 249-270). The MIT.
35. RB Ivry JE Schlerf 2008 Dedicated and intrinsic models of time perception Trends in Cognitive Sciences 12 273 280 10.1016/j.tics.2008.04.002 18539519
36. EM Izhikevich 2007 Solving the distal reward problem through linkage of STDP and dopamine signaling Cerebral Cortex 17 2443 2452 10.1093/cercor/bhl152 17220510 (Pubitemid 47517479)
37. D Joel I Weiner 2000 The connections of the dopaminergic system with the striatum in rats and primates: an analysis with respect to the functional and compartmental organization of the striatum Neuroscience 96 451 474 10.1016/S0306-4522(99)00575-8 1:CAS:528:DC%2BD3cXhslamsb4%3D 10717427 (Pubitemid 30136312)
38. UR Karmarkar DV Buonomano 2007 Timing in the absence of clocks: encoding time in neural network states Neuron 53 427 438 10.1016/j.neuron.2007.01.006 1:CAS:528:DC%2BD2sXitVekur0%3D 17270738 (Pubitemid 46161265)
39. C Kolodziejski B Porr F Worgotter 2009 On the asymptotic equivalence between differential hebbian and temporal difference learning Neural Computation 21 1173 1202 10.1162/neco.2008.04-08-750 19018698
40. Y Komura R Tamura T Uwano H Nishijo K Kaga T Ono 2001 Retrospective and prospective coding for predicted reward in the sensory thalamus Nature 412 546 549 10.1038/35087595 1:CAS:528:DC%2BD3MXlvFGrt78%3D 11484055 (Pubitemid 32743833)
41. M Laubach 2005 Who's on first? What's on second? The time course of learning in corticostriatal systems Trends in Neurosciences 28 508 511 10.1016/j.tins.2005.07.008 (Pubitemid 41316300)
42. MA Lebedev JE O'Doherty MA Nicolelis 2008 Decoding of temporal intervals from cortical ensemble activity Journal of Neurophysiology 99 166 186 10.1152/jn.00734.2007 18003881
43. MI Leon MN Shadlen 2003 Representation of time by neurons in the posterior parietal cortex of the macaque Neuron 38 317 327 10.1016/S0896- 6273(03)00185-5 1:CAS:528:DC%2BD3sXjslOktb8%3D 12718864 (Pubitemid 36555656)
44. PA Lewis 2002 Finding the timer Trends in Cognitive Sciences 6 195 196 10.1016/S1364-6613(02)01906-X 11983576 (Pubitemid 34451006)
45. T Ljungberg P Apicella W Schultz 1992 Responses of monkey dopamine neurons during learning of behavioral reactions Journal of Neurophysiology 67 145 163 1:CAS:528:DyaK38Xhs1Krsr4%3D 1552316
46. C Lucchetti L Bon 2001 Time-modulated neuronal activity in the premotor cortex of macaque monkeys Experimental Brain Research 141 254 260 10.1007/s002210100818 1:STN:280:DC%2BD3MnmslGgtA%3D%3D (Pubitemid 33020854)
47. C Lucchetti A Ulrici L Bon 2005 Dorsal premotor areas of nonhuman primate: functional flexibility in time domain European Journal of Applied Physiology 95 121 130 10.1007/s00421-005-1360-1 16047152 (Pubitemid 41829776)
48. EA Ludvig RS Sutton EJ Kehoe 2008 Stimulus representation and the timing of reward-prediction errors in models of the dopamine system Neural Computation 20 3034 3054 10.1162/neco.2008.11-07-654 18624657
49. Ludvig, E. A., Sutton, R. S., Verbeek, E., & Kehoe, E. J. (2009). A computational model of hippocampal function in trace conditioning. In D. Koller, D. Schuurmans, Y. Bengio, & L. Bottou (Eds.), Neural information processing systems (pp. 993-1000).
50. J Mirenowicz W Schultz 1994 Importance of unpredictability for reward responses in primate dopamine neurons Journal of Neurophysiology 72 1024 1027 1:STN:280:DyaK2M%2FnsVKmsA%3D%3D 7983508
51. PR Montague P Dayan TJ Sejnowski 1996 A framework for mesencephalic dopamine systems based on predictive hebbian learning Journal of Neuroscience 16 1936 1947 1:CAS:528:DyaK28Xht1GlsLc%3D 8774460
52. PR Montague SE Hyman JD Cohen 2004 Computational roles for dopamine in behavioural control Nature 431 760 767 10.1038/nature03015 1:CAS:528: DC%2BD2cXotl2kt7k%3D 15483596 (Pubitemid 39434064)
53. G Morris D Arkadir A Nevet E Vaadia H Bergman 2004 Coincident but distinct messages of midbrain dopamine and striatal tonically active neurons Neuron 43 133 143 10.1016/j.neuron.2004.06.012 1:CAS:528:DC%2BD2cXmtVyjsbs%3D 15233923 (Pubitemid 38952825)
54. G Morris A Nevet D Arkadir E Vaadia H Bergman 2006 Midbrain dopamine neurons encode decisions for future action Nature Neuroscience 9 1057 1063 10.1038/nn1743 1:CAS:528:DC%2BD28Xnt12js7c%3D 16862149 (Pubitemid 44264919)
55. RC O'Reilly MJ Frank 2006 Making working memory work: a computational model of learning in the prefrontal cortex and basal ganglia Neural Computation 18 283 328 10.1162/089976606775093909 16378516
56. S Otani H Daniel MP Roisin F Crepel 2003 Dopaminergic modulation of long-term synaptic plasticity in rat prefrontal neurons Cerebral Cortex 13 1251 1256 10.1093/cercor/bhg092 14576216 (Pubitemid 37323001)
57. WX Pan R Schmidt JR Wickens BI Hyland 2005 Dopamine cells respond to predicted events during classical conditioning: evidence for eligibility traces in the reward-learning network Journal of Neuroscience 25 6235 6242 10.1523/JNEUROSCI.1478-05.2005 1:CAS:528:DC%2BD2MXmtFKntLY%3D 15987953 (Pubitemid 40923490)
58. W Potjans A Morrison M Diesmann 2009 A spiking neural network model of an actor-critic learning agent Neural Computation 21 301 339 10.1162/neco.2008.08- 07-593 19196231
59. JN Reynolds JR Wickens 2002 Dopamine-dependent plasticity of corticostriatal synapses Neural Networks 15 507 521 10.1016/S0893-6080(02)00045- X 12371508 (Pubitemid 34947461)
60. Rivest, F., Bengio, Y., & Kalaska, J. F. (2005). Brain inspired reinforcement learning. In L. K. Saul, Y. Weiss & L. Bottou (Eds.), Neural information processing systems (pp. 1129-1136). Cambridge: The MIT.
61. PD Roberts RA Santiago G Lafferriere 2008 An implementation of reinforcement learning based on spike timing dependent plasticity Biological Cybernetics 99 517 523 10.1007/s00422-008-0265-6 18941775
62. MR Roesch DJ Calu G Schoenbaum 2007 Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards Nature Neuroscience 10 1615 1624 10.1038/nn2013 1:CAS:528:DC%2BD2sXhtlKltLbK 18026098 (Pubitemid 350175882)
63. R Romo CD Brody A Hernandez L Lemus 1999 Neuronal correlates of parametric working memory in the prefrontal cortex Nature 399 470 473 10.1038/20939 1:CAS:528:DyaK1MXjvV2jtbc%3D 10365959 (Pubitemid 29258852)
64. NP Rougier DC Noelle TS Braver JD Cohen RC O'Reilly 2005 Prefrontal cortex and flexible cognitive control: rules without symbols Proceedings of the National Academy of Sciences USA 102 7338 7343 10.1073/pnas.0502455102 1:CAS:528:DC%2BD2MXks12nsbs%3D (Pubitemid 40696380)
65. K Samejima Y Ueda K Doya M Kimura 2005 Representation of action-specific reward values in the striatum Science 310 1337 1340 10.1126/science.1115270 1:CAS:528:DC%2BD2MXht1GqsrrP 16311337 (Pubitemid 41698867)
66. W Schultz P Apicella E Scarnati T Ljungberg 1992 Neuronal activity in monkey ventral striatum related to the expectation of reward Journal of Neuroscience 12 4595 4610 1:STN:280:DyaK3s7gtVGnuw%3D%3D 1464759
67. W Schultz P Apicella T Ljungberg 1993 Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task Journal of Neuroscience 13 900 913 1:STN:280: DyaK3s7nvF2gtA%3D%3D 8441015
68. Schultz, W., Apicella, P., Romo, R., & Scarnati, E. (1995). Context-dependent activity in primate striatum reflecting past and future behavioral events. In J. C. Houk, J. L. Davis, & D. G. Beiser (Eds.), Models of information processing in the basal ganglia (pp. 11-27). The MIT.
69. W Schultz P Dayan PR Montague 1997 A neural substrate of prediction and reward Science 275 1593 1599 10.1126/science.275.5306.1593 1:CAS:528: DyaK2sXhvFSntro%3D 9054347 (Pubitemid 27120526)
70. RE Suri W Schultz 1998 Learning of sequential movements by neural network model with dopamine-like reinforcement signal Experimental Brain Research 121 350 354 10.1007/s002210050467 1:STN:280:DyaK1cvhslahsA%3D%3D (Pubitemid 28371966)
71. RE Suri W Schultz 1999 A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task Neuroscience 91 871 890 10.1016/S0306-4522(98)00697-6 1:CAS:528:DyaK1MXjtFChs7g%3D 10391468 (Pubitemid 29225621)
72. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge: The MIT.
73. G Thibaudeau O Potvin K Allen FY Dore S Goulet 2007 Dorsal, ventral, and complete excitotoxic lesions of the hippocampus in rats failed to impair appetitive trace conditioning Behavioural Brain Research 185 9 20 10.1016/j.bbr.2007.07.004 17707920 (Pubitemid 47552571)
74. Todd, M. T., Niv, Y., & Cohen, J. D. (2009). Learning to use working memory in partially observable environments through dopaminergic reinforcement. In D. Koller, D. Schuurmans, Y. Bengio & L. Bottou (Eds.), Neural information processing systems (pp. 1689-1696). Cambridge: The MIT.
75. Wickens, J., & Kotter, R. (1995). Cellular models of reinforcement. In J. C. Houk, J. L. Davis & D. G. Beiser (Eds.), Models of information processing in the basal ganglia (pp. 187-214). Cambridge: The MIT.