Interactions of timing and prediction error learning

Behavioural Processes

Volumn 101, Issue , 2014, Pages 135-145

  Kirkpatrick, Kimberly ^a  

^a KANSAS STATE UNIVERSITY   (United States)

Author keywords

Computational modeling; Motivation; Prediction error learning; Timing

Indexed keywords

COMPUTATIONAL MODELING; MOTIVATION; PREDICTION ERROR LEARNING; TIMING;

ANIMALS; BRAIN; CONDITIONING, CLASSICAL; HUMANS; LEARNING; MODELS, NEUROLOGICAL; MOTIVATION; NEURAL PATHWAYS; REWARD;

EID: 84894227087     PISSN: 03766357     EISSN: 18728308     Source Type: Journal    
DOI: 10.1016/j.beproc.2013.08.005     Document Type: Review

References (129)

1. Balci F., Ludvig E.A., Abner R., Zhuang X., Poon P., Brunner D. Motivational effects on interval timing in dopamine transporter (DAT) knockdown mice. Brain Research 2010, 1325:89-99.
2. Balci F., Ludvig E.A., Brunner D. Within-session modulation of anticipatory timing: when to start responding. Behavioural Processes 2010, 85:204-206.
3. Balci F., Wiener M., Çavdaroǧlu B., Coslett H.B. Epistasis effects of dopamine genes on interval timing and reward magnitude in humans. Neuropsychologia 2013, 51:293-308.
4. Balsam P., Drew M., Yang C. Timing at the start of associative learning. Learning and Motivation 2002, 33:141-155.
5. Balsam P., Drew M.R., Gallistel C.R. Time and associative learning. Comparative Cognition and Behavior Reviews 2010, 5:1-22.
6. Balsam P., Gallistel C.R. Temporal maps and informativeness in associative learning. Trends in Neurosciences 2009, 32:73-78.
7. Balsam P., Sanchez-Castillo H., Taylor K., Van Volkinburg H., Ward R.D. Timing and anticipation: conceptual and methodological approaches. European Journal of Neuroscience 2009, 30:1749-1755.
8. Barnet R.C., Grahame N.J., Miller R.R. Temporal encoding as a determinant of blocking. Journal of Experimental Psychology: Animal Behavior Processes 1993, 19:327-341.
9. Basten U., Biele G., Heekeren H.R., Fiebach C.J. How the brain integrates costs and benefits during decision making. Proceedings of the National Academy of Sciences of the United States of America 2010, 107:21767-22177.
10. Baxter M.G., Murray E.A. The amygdala and reward. Nature Reviews 2002, 3:563-573.
11. Bayer H.M., Glimcher P.W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 2005, 47:129-141.
12. Bezzina G., Body S., Cheung T.H., Hampson C.L., Deakin J.F.W., Anderson I.M., Szabadi E., Bradshaw C.M. Effect of quinolinic acid-induced lesions of the nucleus accumbens core on performance on a progressive ratio schedule of reinforcement: implications for inter-temporal choice. Psychopharmacology 2008, 197:339-350.
13. Bezzina G., Cheung T.H.C., Asgari K., Hampson C.L., Body S., Bradshaw C.M., Szabadi E., Deakin J.F.W., Anderson I.M. Effects of quinolinic acid-induced lesions of the nucleus accumbens core on inter-temporal choice: a quantitative analysis. Psychopharmacology (Berlin) 2007, 195:71-84.
14. Bhattacharjee Y. Neuroscience: a timely debate about the brain. Science (Washington, D.C., 1883-) 2006, 311:596-598.
15. Blundell P., Hall G., Killcross A.S. Lesions of the basolateral amygdala disrupt selective aspects of reinforcer representation in rats. Journal of Neuroscience 2001, 21:9018-9026.
16. Buhusi C.V., Meck W.H. What makes us tick? Functional and neural mechanisms of interval timing. Nature Reviews Neuroscience 2005, 6:755-765.
17. Cardinal R.N., Pennicott D.R., Sugathapala C.L., Robbins T.W., Everitt B.J. Impulsive choice induced in rats by lesions of the nucleus accumbens core. Science 2001, 292:2499-2501.
18. Church R.M., Broadbent H.A. Alternative representations of time, number, and rate. Cognition 1990, 37:55-81.
19. Church R.M., Kirkpatrick K. Theories of conditioning and timing. Handbook of Contemporary Learning Theories 2001, Lawrence Erlbaum Associates, Hillsdale, NJ. R.R. Mowrer, S.B. Klein (Eds.).
20. Church R.M., Lacourse D.M., Crystal J.D. Temporal search as a function of the variability of interfood intervals. Journal of Experimental Psychology: Animal Behavior Processes 1998, 24:291-315.
21. Cohen J.D., McClure S.M., Yu A.J. Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration. Philosophical Transactions of the Royal Society B 2007, 362:933-942.
22. Coull J.T., Cheng R.-K., Meck W.H. Neuroanatomical and neurochemical substrates of timing. Neuropsychopharmacology 2011, 36:3-25.
23. Coull J.T., Nazarian B., Vidal F. Timing, storage, and comparison of stimulus duration engage discrete anatomical components of a perceptual timing network. Journal of Cognitive Neuroscience 2008, 20:2185-2197.
24. Coull J.T., Nobre A. Dissociating explicit timing from temporal expectation with fMRI. Current Opinion in Neurobiology 2008, 18:137-144.
25. Coull J.T., Vidal F., Nazarian B., Macar F. Functional anatomy of the attentional modulation of time estimation. Science (Washington, D.C., 1883-) 2004, 303:1506-1508.
26. da Costa Araujo S., Body S., Valencia Torres L., Olarte Sanchez C.M., Bak V.K., Deakin J.F.W., Anderson I.M., Bradshaw C.M., Szabadi E. Choice between reinforcer delays versus choice between reinforcer magnitudes: differential Fos expression in the orbital prefrontal cortex and nucleus accumbens core. Behavioural Brain Research 2010, 213:269-277.
27. Davis M., Schlesinger L.S., Sorenson C.A. Temporal specificity of fear conditioning: effects of different conditioned stimulus-unconditioned stimulus intervals on the fear-potentiated startle effect. Journal of Experimental Psychology: Animal Behavior Processes 1989, 15:295-310.
28. Daw N.D., O'Doherty J.P., Dayan P., Seymour B., Dolan R.J. Cortical substrates for exploratory decisions in humans. Nature (London) 2006, 441:876-879.
29. Doughty A.H., Richards J.B. Effects of reinforcer magnitude on responding under differential-reinforcement-of-low-rate schedules of rats and pigeons. Journal of the Experimental Analysis of Behavior 2002, 78:17-30.
30. Drew M.R., Zupan B., Cooke A., Couvillon P.A., Balsam P.D. Temporal control of conditioned responding in goldfish. Journal of Experimental Psychology: Animal Behavior Processes 2005, 31:31-39.
31. Fairhurst S., Gallistel C.R., Gibbon J. Temporal landmarks: proximity prevails. Animal Cognition 2003, 6:113-120.
32. Frank M.J., Claus E.D. Anatomy of a decision: striato-orbitofrontal interactions in reinforcement learning, decision making, and reversal. Psychological Review 2006, 113:300-326.
33. Gaioni S.J. Blocking and nonsimultaneous compounds: comparison of responding during compound conditioning and testing. Pavlovian Journal of Biological Science 1982, January-March:16-29.
34. Gallistel C.R., Gibbon J. Time, rate, and conditioning. Psychological Review 2000, 107:289-344.
35. Galtress T., Kirkpatrick K. Reward value effects on timing in the peak procedure. Learning and Motivation 2009, 40:109-131.
36. Galtress T., Kirkpatrick K. Reward magnitude effects on temporal discrimination. Learning and Motivation 2010, 41:108-124.
37. Galtress T., Kirkpatrick K. The role of the nucleus accumbens core in impulsive choice, timing, and reward processing. Behavioral Neuroscience 2010, 124:26-43.
38. Galtress T., Marshall A.T., Kirkpatrick K. Motivation and timing: clues for modeling the reward system. Behavioural Processes 2012, 90:142-153.
39. Gibbon J., Balsam P. Spreading association in time. Autoshaping and Conditioning Theory 1981, Academic Press, New York. C.M. Locurto, H.S. Terrace, J. Gibbon (Eds.).
40. Gibbon J., Church R.M. Sources of variance in an information processing theory of timing. Animal Cognition 1984, Elrbaum, Hillsdale, NJ. H.L. Roitblat, T.G. Bever, H.S. Terrace (Eds.).
41. Gibbon J., Church R.M., Meck W.H. Scalar timing in memory. Timing and Time Perception (Annals of the New York Academy of Sciences) 1984, New York Academy of Sciences, New York. J. Gibbon, L. Allan (Eds.).
42. Gottfried J.A., O'Doherty J., Dolan R.J. Encoding predictive reward value in human amygdala and orbitofrontal cortex. Science (Washington, D.C., 1883-) 2003, 301:1104-1107.
43. Grace R.C., Nevin J.A. Response strength and temporal control in fixed-interval schedules. Animal Learning and Behavior 2000, 28:313-331.
44. Gregorios-Pippas L., Tobler P.N., Schultz W. Short-term temporal discounting of reward value in human ventral striatum. Journal of Neurophysiology 2009, 101:1507-1523.
45. Grossberg S., Schmajuk N.A. Neural dynamics of adaptive timing and temporal discrimination during associative learning. Neural Network 1989, 2:79-102.
46. Guilhardi P., Yi L., Church R.M. A modular theory of learning and performance. Psychonomic Bulletin and Review 2007, 14:543-559.
47. Hancock R.A. Tests of the conditioned reinforcement value of sequential stimuli in pigeons. Animal Learning and Behavior 1982, 10:46-54.
48. Hazy T.E., Frank M.J., O'Reilly R.C. Neural mechanisms of acquired phasic dopamine responses in learning. Neuroscience and Biobehavioral Reviews 2010, 34:701-720.
49. Holland P.C. Effects of interstimulus and intertrial intervals on appetitive conditioning of rats. Animal Learning and Behavior 2000, 28:121-135.
50. Holland P.C., Gallagher M. Amygdala-frontal interactions and reward expectancy. Current Opinion in Neurobiology 2004, 14:148-155.
51. Jennings D., Kirkpatrick K. Interval duration effects on blocking in appetitive conditioning. Behavioural Processes 2006, 71:318-329.
52. Jennings D.J., Alonso E., Mondragón E., Bonardi C. Temporal uncertainty during overshadowing: a temporal difference account. Computational Neuroscience for Advancing Artificial Intelligence: Models, Methods and Applications 2011, Medical Information Science Reference, Hershey, PA. E. Alonso, E. Mondragón (Eds.).
53. Jennings D.J., Bonardi C., Kirkpatrick K. Overshadowing and stimulus duration. Journal of Experimental Psychology: Animal Behavior Processes 2007, 33:464-475.
54. Jozefowiez J., Staddon J.E.R., Cerutti D.T. The behavioral economics of choice and interval timing. Psychological Review 2009, 116:519-539.
55. Kable J.W., Glimcher P.W. The neural correlates of subjective value during intertemporal choice. Nature Neuroscience 2007, 10:1625-1633.
56. Kable J.W., Glimcher P.W. The neurobiology of decision: consensus and controversy. Neuron 2009, 63:733-745.
57. Kacelnik A., Brunner D. Timing and foraging: Gibbon's scalar expectancy theory and optimal patch exploitation. Learning and Motivation 2002, 33:177-195.
58. Kamin L.J. Attention-like processes in classical conditioning. Miami Symposium on the Prediction of Behavior: Aversive Stimulation 1968, University of Miami Press, Coral Gables, FL. M.R. Jones (Ed.).
59. Kamin L.J. Predictability, surprise, attention, and conditioning. Punishment and Aversive Behavior 1969, Appleton-Century-Crofts, New York. B.A. Campbell, R.M. Church (Eds.).
60. Kehoe E.J. CS-US contiguity and CS intensity in conditioning of the rabbit's nictitating membrane response to serial compound stimuli. Journal of Experimental Psychology: Animal Behavior Processes 1983, 9:307-319.
61. Kehoe E.J., Schreurs B.G., Amodei N. Blocking acquisition of the rabbit's nictitating membrane response to serial conditioned stimuli. Learning and Motivation 1981, 12:92-108.
62. Kehoe E.J., Schreurs B.G., Graham P. Temporal primacy overrides prior training in serial compound conditioning of the rabbit's nictitating membrane response. Animal Learning and Behavior 1987, 15:455-464.
63. Killeen P.R., Fetterman J.G. A behavioral theory of timing. Psychological Review 1988, 95:274-295.
64. Kirkpatrick K. Packet theory of conditioning and timing. Behavioural Processes 2002, 57:89-106.
65. Kirkpatrick K., Church R.M. Are separate theories of conditioning and timing necessary?. Behavioural Processes 1998, 44:163-182.
66. Kirkpatrick K., Church R.M. Independent effects of stimulus and cycle duration in conditioning: the role of timing processes. Animal Learning and Behavior 2000, 28:373-388.
67. Kirkpatrick K., Church R.M. Tracking of the expected time to reinforcement in temporal conditioning procedures. Learning and Behavior 2003, 31:3-21.
68. Kirkpatrick K., Church R.M. Temporal learning in random control procedures. Journal of Experimental Psychology: Animal Behavior Processes 2004, 30:213-228.
69. Kohler E.A., Ayres J.J.B. The Kamin blocking effect with variable-duration CSs. Animal Learning and Behavior 1979, 7:347-350.
70. Kohler E.A., Ayres J.J.B. Blocking with serial and simultaneous compounds in a trace conditioning procedure. Animal Learning and Behavior 1982, 10:277-287.
71. Komura Y., Tamura R., Uwano T., Nishijo H., Kaga K., Ono T. Retrospective and prospective coding for predicted reward in the sensory thalamus. Nature 2001, 412:546-549.
72. Kringlebach M.L., Rolls E.T. The functional neuroanatomy of the human orbitofrontal cortex: evidence from neuroimaging and neuropsychology. Progress in Neurobiology 2004, 72:341-372.
73. Lattal K.M. Trial and intertrial durations in Pavlovian conditioning: issues of learning and performance. Journal of Experimental Psychology: Animal Behavior Processes 1999, 25:433-450.
74. Lebedev M.A., O'Doherty J.E., Nicolelis M.A. Decoding of temporal intervals from cortical ensemble activity. Journal of Neurophysiology 2008, 99:166-186.
75. Leon M.I., Shadlen M.N. Representation of time by neurons in the posterior parietal cortex of the macaque. Neuron 2003, 38:317-327.
76. Ludvig E.A., Balci F., Spetch M.L. Reward magnitude and timing in pigeons. Behavioural Processes 2011, 86:359-363.
77. Ludvig E.A., Bellamare M.G., Pearson K.G. A primer on reinforcement learning in the brain. Computational Neuroscience for Advancing Artificial Intelligence: Models, Methods and Applications 2011, IGI Global, Hershey, PA. psychological, computational, P. neural, E. In:, E. Alonso, Mondragón (Eds.).
78. Ludvig E.A., Conover K., Shizgal P. The effects of reinforcer magnitude on timing in rats. Journal of the Experimental Analysis of Behavior 2007, 87:201-218.
79. Ludvig E.A., Sutton R.S., Kehoe E.J. Stimulus representations and the timing of reward-prediction errors in models of the dopamine system. Neural Computation 2008, 20:3034-3054.
80. Ludvig E.A., Sutton R.S., Kehoe E.J. Evaluating the TD model of classical conditioning. Learning and Behavior 2012, 40:305-319.
81. Luzardo A., Ludvig E.A., Rivest F. An adaptive drift-diffusion model of interval timing dynamics. Behavioural Processes 2013, 95:90-99.
82. Machado A. Learning the temporal dynamics of behavior. Psychological Review 1997, 104:241-265.
83. Maia T.V. Reinforcement learning, conditioning, and the brain: successes and challenges. Cognitive, Affective, and Behavioral Neuroscience 2009, 9:343-364.
84. Maleske R.T., Frey P.W. Blocking in eyelid conditioning: effect of changing the CS-US interval and introducing an intertrial stimulus. Animal Learning and Behavior 1979, 7:452-456.
85. Matell M.S., Meck W.H. Cortico-striatal circuits and interval timing: coincidence detection of oscillatory processes. Cognitive Brain Research 2004, 21:139-170.
86. Matell M.S., Meck W.H., Nicolelis M.A.L. Interval timing and the encoding of signal duration by ensembles of cortical and striatal neurons. Behavioral Neuroscience 2003, 117:760-773.
87. McClure S.M., Laibson D.I., Loewenstein G., Cohen J.D. Separate neural systems value immediate and delayed monetary rewards. Science (Washington, D.C., 1883-) 2004, 306:503-507.
88. McMillan N., Roberts W.A. The effects of cue competition on timing in pigeons. Behavioural Processes 2010, 84:581-590.
89. Meck W.H. Attentional bias between modalities. Annals of the New York Academy of Sciences 1984, 423:278-291.
90. Meck W.H. Neuropharmacology of timing and time perception. Cognitive Brain Research 1996, 3:227-242.
91. Meck W.H. Neuroanatomical localization of an internal clock: a functional link between mesolimbic, nigrostriatal, and mesocortical dopaminergic systems. Brain Research 2006, 1109:93-107.
92. Meck W.H., Penney T.B., Pouthas V. Cortico-striatal representation of time in animals and humans. Current Opinion in Neurobiology 2008, 18:145-152.
93. Miall C. The storage of time intervals using oscillating neurons. Neural Computation 1989, 1:359-371.
94. Montague P.R., Dayan P., Sejnowski T.J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. Journal of Neuroscience 1996, 16:1936-1947.
95. Morillon B., Kell C.A., Giraud A.L. Three stages and four neural systems in time estimation. Journal of Neuroscience 2009, 29:14803-14811.
96. Nenadic I., Gaser C., Volz H.P., Rammsayer T., Hager F., Sauer H. Processing of temporal information and the basal ganglia: new evidence from fMRI. Experimental Brain Research 2003, 148:238-246.
97. Niv Y. Reinforcement learning in the brain. Journal of Mathematical Psychology 2009, 53:139-154.
98. O'Neill M.O., Schultz W. Coding of reward risk by orbitofrontal neurons is mostly distinct from coding of reward value. Neuron 2010, 68:789-800.
99. Ohyama T., Mauk M.D. Latent acquisition of timed responses in cerebellar cortex. The Journal of Neuroscience 2001, 21:682-690.
100. Olausson P., Jentsch J.D., Tronson N., Neve R.L., Nestler E.J., Taylor J.R. δFoSB in the nucleus accumbens regulates food-reinforced instrumental behavior and motivation. The Journal of Neuroscience 2006, 26:9196-9204.
101. Pavlov I.P. Conditioned Reflexes 1927, Dover, New York.
102. Peters J., Büchel C. Overlapping and distinct neural systems code for subjective value during intertemporal and risky decision making. The Journal of Neuroscience 2009, 29:15727-15734.
103. Peters J., Büchel C. Neural representations of subjective reward value. Behavioural Brain Research 2010, 213:135-141.
104. Peters J., Büchel C. The neural mechanisms of inter-temporal decision-making: understanding variability. Trends in Cognitive Sciences 2011, 15:227-239.
105. Pothuizen H.H., Jongen-Relo A.L., Feldon J., Yee B.K. Double dissociation of the effects of selective nucleus accumbens core and shell lesions on impulsive-choice behaviour and salience learning in rats. European Journal of Neuroscience 2005, 22:2605-2616.
106. Rao S.M., Mayer A.R., Harrington D.L. The evolution of brain activation during temporal processing. Nature Neuroscience 2001, 4:317-323.
107. Rescorla R.A., Wagner A.R. A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. Classical Conditioning II: Current Research and Theory 1972, Appleton-Century-Crofts, New York. A.H.B.W.F. Prokasy (Ed.).
108. Reutimann J., Yakovlev V., Fusi S., Senn W. Climbing neuronal activity as an event-based cortical representation of time. Journal of Neuroscience 2004, 24:3295-3303.
109. Rivest F., Bengio Y. Adaptive Drift Diffusion Process to Learn Time Intervals 2011, Cornell University Library.
110. Robbins T.W., Everitt B.J. Neurobehavioural mechanisms of reward and motivation. Current Opinion in Neurobiology 1996, 6:228-236.
111. Roberts S. Isolation of an internal clock. Journal of Experimental Psychology: Animal Behavior Processes 1981, 7:242-268.
112. Roesch M.R., Calu D.J., Schoenbaum G. Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nature Neuroscience 2007, 10:1615-1624.
113. Schoenbaum G., Setlow B., Saddoris M.P., Gallagher M. Encoding predicted outcome and acquired value in orbitofrontal cortex during cue sampling depends upon input from basolateral amygdala. Neuron 2003, 39:855-867.
114. Schreurs B.G., Westbrook R.F. The effects of changes in the CS-US interval during compound conditioning upon an otherwise blocked element. Quarterly Journal of Experimental Psychology B 1982, 34:19-30.
115. Schultz W. Predictive reward signal of dopamine neurons. Journal of Neurophysiology 1998, 80:1-27.
116. Schultz W. Behavioral theories and the neurophysiology of reward. Annual Review of Psychology 2006, 57:87-115.
117. Schultz W., Dayan P., Montague P.R. A neural substrate of prediction and reward. Science (Washington, D.C., 1883-) 1997, 275:1593-1599.
118. Shiflett M.W., Balleine B.W. At the limbic-motor interface: disconnection of basolateral amygdala from nucleus accumbens core and shell reveals dissociable components of incentive motivation. European Journal of Neuroscience 2010, 32:1735-1743.
119. Simen P., Balci F., deSouza L., Cohen J.D., Holmes P. A model of interval timing by neural integration. The Journal of Neuroscience 2011, 31:9238-9253.
120. Staddon J.E.R., Higa J.J. Multiple time scales in simple habituation. Psychological Review 1996, 103:720-733.
121. Sutton R.S., Barto A.G. Toward a modern theory of adaptive networks: expectation and prediction. Psychological Review 1981, 88:135-170.
122. Sutton R.S., Barto A.G. Time derivative models of Pavlovian reinforcement. Learning and Computational Neuroscience: Foundations of Adaptive Networks 1990, MIT Press, Cambridge, MA. M.R. Gabriel, J.W. Moore (Eds.).
123. Tobler P.N., Fiorillo C.D., Schultz W. Adaptive coding of reward value by dopamine neurons. Science (Washington, D.C., 1883-) 2005, 307:1642-1645.
124. Waelti P., Dickinson A., Schultz W. Dopamine responses comply with basic assumptions of formal learning theory. Nature 2001, 412:43-48.
125. Ward R.D., Kellendonk C., Simpson E.H., Lipatova O., Drew M.R., Fairhurst S., Kandel E.R., Balsam P.D. Impaired timing precision produced by striatal D2 receptor overexpression is mediated by cognitive and motivational deficits. Behavioral Neuroscience 2009, 123:720-730.
126. Ward R.D., Odum A.L. Effects of prefeeding, intercomponent-interval food, and extinction on temporal discrimination and pacemaker rate. Behavioural Processes 2006, 71:297-306.
127. Winstanley C.A., Theobald D.E., Dalley J.W., Robbins T.W. Interactions between serotonin and dopamine in the control of impulsive choice in rats: therapeutic implications for impulse control disorders. Neuropsychopharmacology 2005, 30:669-682.
128. Winstanley C.A., Theobald D.E.H., Cardinal R.N., Robbins T.W. Contrasting roles of basolateral amygdala and orbitofrontal cortex in impulsive choice. The Journal of Neuroscience 2004, 24:4718-4722.
129. Zhang M., Balmadrid C., Kelley A. Nucleus accumbens opiod, GABAergic and dopaminergic modualtion of palatable food motivation: contrasting effects revealed by a progressive ratio study in the rat. Behavioral Neuroscience 2003, 117:202-211.