Dialogues on prediction errors

Trends in Cognitive Sciences

Volumn 12, Issue 7, 2008, Pages 265-272

  Niv, Yael ^a   Schoenbaum, Geoffrey ^b  

^a PRINCETON UNIVERSITY   (United States)

^b UNIVERSITY OF MARYLAND SCHOOL OF MEDICINE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ERRORS; NEURONS; REINFORCEMENT; REINFORCEMENT LEARNING;

ACTIVE AREAS; COGNITIVE NEUROSCIENCES; DOPAMINERGIC (DA) NEURONS; ELSEVIER (CO); ERROR SIGNALS; HABITUAL BEHAVIOR; MIDBRAIN; NEURAL CIRCUITS; NEURAL CORRELATES; PRE-FRONTAL CORTEX (PFC); PREDICTION ERRORS; STRIATUM; TRANSITION (JEL CLASSIFICATIONS:E52 ,E41 ,E31);

FORECASTING;

DOPAMINE;

ANALYTICAL ERROR; BEHAVIOR; BEHAVIORAL RESEARCH; DOPAMINERGIC NERVE CELL; DOPAMINERGIC SYSTEM; HUMAN; LEARNING; NEUROSCIENCE; NONHUMAN; OUTCOMES RESEARCH; PREDICTION; PROBABILITY; REINFORCEMENT; REVIEW; STATISTICAL ANALYSIS; TRAINING;

ARTIFICIAL INTELLIGENCE; ASSOCIATION LEARNING; BRAIN; DISCRIMINATION LEARNING; DOPAMINE; HUMANS; MODELS, NEUROLOGICAL; MODELS, PSYCHOLOGICAL; PROBABILITY LEARNING; PSYCHOLOGICAL THEORY; REINFORCEMENT (PSYCHOLOGY);

EID: 45949092119     PISSN: 13646613     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.tics.2008.03.006     Document Type: Review

References (61)

1. Schultz W., et al. A neural substrate for prediction and reward. Science 275 (1997) 1593-1599
2. Montague P.R., et al. A framework for mesencephalic dopamine systems based on predictive hebbian learning. J. Neurosci. 16 (1996) 1936-1947
3. Feierstein C.E., et al. Representation of spatial goals in rat orbitofrontal cortex. Neuron 51 (2006) 495-507
4. Belova M.A., et al. Expectation modulates neural responses to pleasant and aversive stimuli in primate amygdala. Neuron 55 (2007) 970-984
5. O'Doherty J., et al. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304 (2004) 452-454
6. Tobler P.N., et al. Human neural learning depends on reward prediction errors in the blocking paradigm. J. Neurophysiol. 95 (2006) 301-310
7. Matsumoto M., et al. Medial prefrontal cell activity signaling prediction errors of action values. Nat. Neurosci. 10 (2007) 647-656
8. Schultz W., and Dickinson A. Neuronal coding of prediction errors. Annu. Rev. Neurosci. 23 (2000) 473-500
9. Nobre A.C., et al. Orbitofrontal cortex is activated during breaches of expectation in tasks of visual attention. Nat. Neurosci. 2 (1999) 11-12
10. Rescorla R.A., and Wagner A.R. A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. In: Black A.H., and Prokasy W.F. (Eds). Classical Conditioning II: Current Research and Theory (1972), Appleton-Century-Crofts 64-99
11. Kamin L.J. Predictability, suprise, attention, and conditioning. In: Campbell B.A., and Church R.M. (Eds). Punishment and Aversive Behavior (1969), Appleton-Century-Crofts 242-259
12. Reynolds G.S. Attention in the pigeon. J. Exp. Anal. Behav. 4 (1961) 203-208
13. Konorski J. Conditioned Reflexes and Neuron Organization (1948), Cambridge University Press
14. Rescorla R.A., and LoLordo V.M. Inhibition of avoidance behavior. J. Comp. Physiol. Psychol. 59 (1968) 406-412
15. Rescorla R.A. Reduction in effectiveness of reinforcement after prior extinction conditioning. Learn. Motiv. 1 (1970) 372-381
16. Kremer E.F. Rescorla-Wagner model - losses in associative strength in compound conditioned stimuli. J. Exp. Psychol. Anim. Behav. Process. 4 (1978) 22-36
17. Sutton R.S., and Barto A.G. Time-derivative models of Pavlovian reinforcement. In: Gabriel M., and Moore J. (Eds). Learning and Computational Neuroscience: Foundations of Adaptive Networks (1990), MIT Press 497-537
18. Sutton R.S., and Barto A.G. Reinforcement Learning: An introduction (1998), MIT Press
19. Niv Y., and Montague P.R. Theoretical and empirical studies of learning. In: Glimcher P.W., et al. (Ed). Neuroeconomics: Decision-Making and the Brain (2008), Academic Press 329-349
20. Matsumoto M., and Hikosaka O. Lateral habenula as a source of negative reward signals in dopamine neurons. Nature 447 (2007) 1111-1115
21. Fiorillo C.D., et al. Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299 (2003) 1898-1902
22. Tobler P.N., et al. Coding of predicted reward omission by dopamine neurons in a conditioned inhibition paradigm. J. Neurosci. 23 (2003) 10402-10410
23. Roesch M.R., et al. Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat. Neurosci. 10 (2007) 1615-1624
24. Redgrave P., et al. Is the short-latency dopamine response too short to signal reward error?. Trends Neurosci. 22 (1999) 146-151
25. Berridge K.C. The debate over dopamine's role in reward: the case for incentive salience. Psychopharmacology (Berl.) 191 (2007) 391-431
26. Horvitz J.C. Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events. Neuroscience 96 (2000) 651-656
27. Ungless M.A., et al. Uniform inhibition of dopamine neurons in the ventral tegmental area by aversive stimuli. Science 303 (2004) 2040-2042
28. Coizet V., et al. Nociceptive responses of midbrain dopaminergic neurons are modulated by the superior colliculus in the rat. Neuroscience 139 (2006) 1479-1493
29. Mirenowicz J., and Schultz W. Preferential activation of midbrain dopamine neurons by appetitive rather than aversive stimuli. Nature 379 (1996) 449-451
30. Daw N.D., et al. Opponent interactions between serotonin and dopamine. Neural Netw. 15 (2002) 603-616
31. Bayer H.M., et al. Statistics of midbrain dopamine neuron spike trains in the awake primate. J. Neurophysiol. 98 (2007) 1428-1439
32. Daw N.D., et al. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8 (2005) 1704-1711
33. O'Doherty J.P., et al. Temporal difference learning model accounts for responses in human ventral striatum and orbitofrontal cortex during Pavlovian appetitive learning. Neuron 38 (2003) 329-337
34. Ramus S.J., and Eichenbaum H. Neural correlates of olfactory recognition memory in the rat orbitofrontal cortex. J. Neurosci. 20 (2000) 8199-8208
35. Tremblay L., and Schultz W. Reward-related neuronal activity during go-nogo task performance in primate orbitofrontal cortex. J. Neurophysiol. 83 (2000) 1864-1876
36. Logothetis N.K., et al. Neurophysiological investigation of the basis of the fMRI signal. Nature 412 (2001) 150-157
37. Logothetis N.K., and Wandell B.A. Interpreting the BOLD signal. Annu. Rev. Physiol. 66 (2004) 735-769
38. Attwell D., and Iadecola C. The neural basis of functional imaging signals. Trends Neurosci. 25 (2002) 621-625
39. Krimer L.S., et al. Dopaminergic regulation of cerebral cortical microcirculation. Nat. Neurosci. 1 (1998) 286-289
40. Costa R.M., et al. Rapid alterations in corticostriatal ensemble coordination during acute dopamine-dependent motor dysfunction. Neuron 52 (2006) 359-369
41. Jensen J., et al. Separate brain regions code for salience versus valence during reward prediction in humans. Hum. Brain Mapp. 28 (2007) 294-302
42. Seymour B., et al. Temporal difference models describe higher order learning in humans. Nature 429 (2004) 664-667
43. Menon M., et al. Temporal difference modeling of the blood-oxygen level dependent response during aversive conditioning in humans: effects of dopaminergic modulation. Biol. Psychiatry 62 (2007) 765-772
44. Carelli R.M. Nucleus accumbens cell firing during goal-directed behaviors for cocaine vs. 'natural' reinforcement. Physiol. Behav. 76 (2002) 379-387
45. Carelli R.M., and Wondolowski J. Selective encoding of cocaine versus natural rewards by nucleus accumbens neurons is not related to chronic drug exposure. J. Neurosci. 23 (2003) 11214-11223
46. Roitman M.F., et al. Nucleus accumbens neurons are innately tuned for rewarding and aversive taste stimuli, encode their predictors, and are linked to motor output. Neuron 45 (2005) 587-597
47. Nicola S.M., et al. Cue-evoked firing of nucleus accumbens neurons encodes motivational significance during a discriminative task. J. Neurophysiol. 91 (2004) 1840-1865
48. Setlow B., et al. Neural encoding in ventral striatum during olfactory discrimination learning. Neuron 38 (2003) 625-636
49. Gallagher M., et al. The amygdala central nucleus and appetitive Pavlovian conditioning: lesions impair one class of conditioned behavior. J. Neurosci. 10 (1990) 1906-1911
50. Holland P.C., and Gallagher M. Amygdala central nucleus lesions disrupt increments, but not decrements, in conditioned stimulus processing. Behav. Neurosci. 107 (1993) 246-253
51. Han J.S., et al. The role of an amygdalo-nigrostriatal pathway in associative learning. J. Neurosci. 17 (1997) 3913-3919
52. Kobayashi Y., and Okada K. Reward prediction error computation in the pedunculopontine tegmental nucleus neurons. Ann. N. Y. Acad. Sci. 1104 (2007) 310-323
53. Ji H., and Shepard P.D. Lateral habenula stimulation inhibits rat midbrain dopamine neurons through a GABA(A) receptor-mediated mechanism. J. Neurosci. 27 (2007) 6923-6930
54. Christoph G.R., et al. Stimulation of the lateral habenula inhibits dopamine containing neurons in the substantia nigra and ventral tegmental area of the rat. J. Neurosci. 6 (1986) 613-619
55. Barto A.G. Adaptive critics and the basal ganglia. In: Houk J.C., et al. (Ed). Models of Information Processing in the Basal Ganglia (1995), MIT Press 215-232
56. Sutton R.S. Learning to predict by the method of temporal difference. Mach. Learn. 3 (1988) 9-44
57. Houk J.C., et al. A model of how basal ganglia generate and use neural signals that predict reinforcement. In: Houk J.C., et al. (Ed). Models of Information Processing in the Basal Ganglia (1995), MIT Press
58. Joel D., et al. Actor-critic models of the basal ganglia: new anatomical and computational perspectives. Neural Netw. 15 (2002) 535-547
59. Morris G., et al. Midbrain dopamine neurons encode decisions for future action. Nat. Neurosci. 9 (2006) 1057-1063
60. Dickinson A., and Balleine B.W. Motivational control of goal-directed action. Anim. Learn. Behav. 22 (1994) 1-18
61. Dayan P., and Balleine B.W. Reward, motivation, and reinforcement learning. Neuron 36 (2002) 285-298