A dual role for prediction error in associative learning

Cerebral Cortex

Volumn 19, Issue 5, 2009, Pages 1175-1185

  Den Ouden, Hanneke E M ^a   Friston, Karl J ^a   Daw, Nathaniel D ^b   McIntosh, Anthony R ^c   Stephan, Klaas E ^a,d  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

^b NEW YORK UNIVERSITY   (United States)

^c UNIVERSITY OF TORONTO   (Canada)

^d UNIVERSITY OF ZURICH   (Switzerland)

Author keywords

Associative learning; Cross modal; Dynamic causal modeling; Effective connectivity; FMRI; Rescorla Wagner model

Indexed keywords

ADULT; ARTICLE; AUDITORY SYSTEM; BRAIN FUNCTION; CONTROLLED STUDY; ERROR; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HUMAN; HUMAN EXPERIMENT; MALE; MODEL; NERVE CELL PLASTICITY; NEUROPHYSIOLOGY; PERCEPTION; PREDICTION; PRIORITY JOURNAL; PUTAMEN; STATE DEPENDENT LEARNING; STIMULUS; TASK PERFORMANCE; VISUAL CORTEX; VISUAL STIMULATION; VISUAL SYSTEM; AUDITORY STIMULATION; BIOLOGICAL MODEL; COGNITION; CONDITIONING; LEARNING; NUCLEAR MAGNETIC RESONANCE IMAGING; PHOTOSTIMULATION; PHYSIOLOGY;

ACOUSTIC STIMULATION; ADULT; ASSOCIATION LEARNING; COGNITION; CONDITIONING (PSYCHOLOGY); FEMALE; HUMANS; MAGNETIC RESONANCE IMAGING; MALE; MODELS, NEUROLOGICAL; NEURONAL PLASTICITY; PHOTIC STIMULATION; YOUNG ADULT;

EID: 67649185192     PISSN: 10473211     EISSN: 14602199     Source Type: Journal    
DOI: 10.1093/cercor/bhn161     Document Type: Article

References (61)

1. Anderson JL, Hutton C, Ashburner J, Turner R, Friston K. 2001. Modeling geometric deformations in EPI time series. Neuroimage. 13: 903-919.
2. Aron AR, Shohamy D, Clark J, Myers C, Gluck MA, Poldrack RA. 2004. Human midbrain sensitivity to cognitive feedback and uncertainty during classification learning. J Neurophysiol. 92:1144-1152.
3. Baier B, Kleinschmidt A, Muller NG. 2006. Cross-modal processing in early visual and auditory cortices depends on expected statistical relationship of multisensory information. J Neurosci. 26: 12260-12265.
4. Baldeweg T. 2006. Repetition effects to sounds: evidence for predictive coding in the auditory system. Trends Cogn Sci. 10:93-94.
5. Blakemore SJ, Wolpert DM, Frith CD. 1998. Central cancellation of selfproduced tickle sensation. Nat Neurosci. 1:635-640.
6. Brogden WJ. 1939. Sensory preconditioning. J Exp Psychol. 25:323-332.
7. Büchel C, Coull JT, Friston KJ. 1999. The predictive value of changes in effective connectivity for human learning. Science. 283:1538-1541.
8. Corlett PR, Aitken MR, Dickinson A, Shanks DR, Honey GD, Honey RA, Robbins TW, Bullmore ET, Fletcher PC. 2004. Prediction error during retrospective revaluation of causal associations in humans: fMRI evidence in favor of an associative model of learning. Neuron. 44:877-888.
9. Csepe V, Karmos G, Molnar M. 1987. Evoked potential correlates of stimulus deviance during wakefulness and sleep in cat-animal model of mismatch negativity. Electroencephalogr Clin Neurophysiol. 66:571-578.
10. David O, Kiebel SJ, Harrison LM, Mattout J, Kilner JM, Friston KJ. 2006. Dynamic causal modeling of evoked responses in EEG and MEG. Neuroimage. 30:1255-1272.
11. Daw ND, Niv Y, Dayan P. 2005. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci. 8:1704-1711.
12. Eickhoff SB, Stephan KE, Mohlberg H, Grefkes C, Fink GR, Amunts K, Zilles K. 2005. A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data. Neuroimage. 25:1325-1335.
13. Falchier A, Clavagnier S, Barone P, Kennedy H. 2002. Anatomical evidence of multimodal integration in primate striate cortex. J Neurosci. 22:5749-5759.
14. Fletcher PC, Anderson JM, Shanks DR, Honey R, Carpenter TA, Donovan T, Papadakis N, Bullmore ET. 2001. Responses of human frontal cortex to surprising events are predicted by formal associative learning theory. Nat Neurosci. 4:1043-1048.
15. Friston K. 2003. Learning and inference in the brain. Neural Netw. 16: 1325-1352.
16. Friston K. 2005. A theory of cortical responses. Philos Trans R Soc Lond B Biol Sci. 360:815-836.
17. Friston K, Kilner J, Harrison L. 2006. A free energy principle for the brain. J Physiol Paris. 100:70-87.
18. Friston KJ, Glaser DE, Henson RN, Kiebel S, Phillips C, Ashburner J. 2002. Classical and Bayesian inference in neuroimaging: applications. Neuroimage. 16:484-512.
19. Friston KJ, Harrison L, Penny W. 2003. Dynamic causal modelling. Neuroimage. 19:1273-1302.
20. Garrido MI, Kilner JM, Kiebel SJ, Stephan KE, Friston KJ. 2007. Dynamic causal modelling of evoked potentials: a reproducibility study. Neuroimage. 36:571-580.
21. Gewirtz JC, Davis M. 2000. Using Pavlovian higher-order conditioning paradigms to investigate the neural substrates of emotional learning and memory. Learn Mem. 7:257-266.
22. Gläscher J, Büchel C. 2005. Formal learning theory dissociates brain regions with different temporal integration. Neuron. 47:295-306.
23. Hebb DO. 1949. The organisation of behaviour. New York: John Wiley. Jensen J, Smith AJ, Willeit M, Crawley AP, Mikulis DJ, Vitcu I, Kapur S. 2007. Separate brain regions code for salience vs. valence during reward prediction in humans. Hum Brain Mapp. 28:294-302.
24. Mackintosh NJ. 1975. A theory of attention: variations in the associability of stimulus with reinforcement. Psychol Rev. 82:276-298.
25. Maldjian JA, Laurienti PJ, Kraft RA, Burdette JH. 2003. An automated method for neuroanatomic and cytoarchitectonic atlas-based interrogation of fMRI data sets. Neuroimage. 19:1233-1239.
26. McClure SM, Berns GS, Montague PR. 2003. Temporal prediction errors in a passive learning task activate human striatum. Neuron. 38: 339-346.
27. McIntosh AR, Cabeza RE, Lobaugh NJ. 1998. Analysis of neural interactions explains the activation of occipital cortex by an auditory stimulus. J Neurophysiol. 80:2790-2796.
28. McLaren IP, Kaye H, Mackintosh NJ. 1989. An associative theory of the representation of stimuli: applications to perceptual learning and latent inhibition. In: Morris RGM, editor. Parallel distributed processing: implications for psychology and neurobiology. Oxford: Clarendon Press. p. 102-120.
29. Menon M, Jensen J, Vitcu I, Graff-Guerrero A, Crawley A, Smith MA, Kapur S. 2007. Temporal difference modeling of the blood-oxygen level dependent response during aversive conditioning in humans: effects of dopaminergic modulation. Biol Psychiatry. 62:765-772
30. Montague PR, Berns GS. 2002. Neural economics and the biological substrates of valuation. Neuron. 36:265-284.
31. Murray SO, Kersten D, Olshausen BA, Schrater P, Woods DL. 2002. Shape perception reduces activity in human primary visual cortex. Proc Natl Acad Sci USA. 99:15164-15169.
32. Naatanen R, Gaillard AW, Mantysalo S. 1978. Early selective-attention effect on evoked potential reinterpreted. Acta Psychol (Amst). 42: 313-329.
33. Naatanen R, Tervaniemi M, Sussman E, Paavilainen P, Winkler I. 2001. Primitive intelligence" in the auditory cortex. Trends Neurosci. 24: 283-288.
34. O'Doherty J, Dayan P, Schultz J, Deichmann R, Friston K, Dolan RJ. 2004. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science. 304:452-454.
35. O'Doherty JP, Dayan P, Friston K, Critchley H, Dolan RJ. 2003. Temporal difference models and reward-related learning in the human brain. Neuron. 38:329-337.
36. Pearce JM, Bouton ME. 2001. Theories of associative learning in animals. Annu Rev Psychol. 52:111-139.
37. Pearce JM, Hall G. 1980. A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychol Rev. 87:532-552.
38. Penny WD, Stephan KE, Mechelli A, Friston KJ. 2004. Comparing dynamic causal models. Neuroimage. 22:1157-1172.
39. Pessiglione M, Seymour B, Flandin G, Dolan RJ, Frith CD. 2006. Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature. 442:1042-1045.
40. Pincze Z, Lakatos P, Rajkai C, Ulbert I, Karmos G. 2002. Effect of deviant probability and interstimulus/interdeviant interval on the auditory N1 and mismatch negativity in the cat auditory cortex. Brain Res Cogn Brain Res. 13:249-253.
41. Rao RP, Ballard DH. 1999. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat Neurosci. 2:79-87.
42. Rescorla RA, Wagner AR. 1972. A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. In: Black AH, Prokasy WF, editors. Classical conditioning II: current research and theory. New York: Appleton Century Crofts. p. 64-99.
43. Rockland KS, Ojima H. 2003. Multisensory convergence in calcarine visual areas in macaque monkey. Int J Psychophysiol. 50:19-26.
44. Schultz W. 1998. Predictive reward signal of dopamine neurons. J Neurophysiol. 80:1-27.
45. Schultz W, Dayan P, Montague PR. 1997. A neural substrate of prediction and reward. Science. 275:1593-1599.
46. Schultz W, Dickinson A. 2000. Neuronal coding of prediction errors. Annu Rev Neurosci. 23:473-500.
47. Seymour B, O'Doherty JP, Dayan P, Koltzenburg M, Jones AK, Dolan RJ, Friston KJ, Frackowiak RS. 2004. Temporal difference models describe higher-order learning in humans. Nature. 429:664-667.
48. Shanks DR. 1995. The psychology of associative learning. Cambridge, UK: Cambridge University Press.
49. Shergill SS, Samson G, Bays PM, Frith CD, Wolpert DM. 2005. Evidence for sensory prediction deficits in schizophrenia. Am J Psychiatry. 162:2384-2386.
50. Stephan KE. 2004. On the role of general system theory for functional neuroimaging. J Anat. 205:443-470.
51. Stephan KE, Harrison LM, Kiebel SJ, David O, Penny WD, Friston KJ. 2007. Dynamic causal models of neural system dynamics: current state and future extensions. J Biosci. 32:129-144.
52. Stephan KE, Marshall JC, Penny WD, Friston KJ, Fink GR. 2007. Interhemispheric integration of visual processing during task-driven lateralization. J Neurosci. 27:3512-3522.
53. Summerfield C, Egner T, Greene M, Koechlin E, Mangels J, Hirsch J. 2006. Predictive codes for forthcoming perception in the frontal cortex. Science. 314:1311-1314.
54. Sutton RS, Barto AG. 1998. Reinforcement learning: an introduction. Cambridge (MA): MIT Press.
55. Tobler PN, O'Doherty JP, Dolan RJ, Schultz W. 2007. Reward value coding distinct from risk attitude-related uncertainty coding in human reward systems. J Neurophysiol. 97:1621-1632.
56. Turner DC, Aitken MR, Shanks DR, Sahakian BJ, Robbins TW, Schwarzbauer C, Fletcher PC. 2004. The role of the lateral frontal cortex in causal associative learning: exploring preventative and super-learning. Cereb Cortex. 14:872-880. von Kriegstein K, Giraud AL. 2006. Implicit multisensory associations influence voice recognition. PLoS Biol. 4:e326.
57. Wasserman EA, Elek SM, Chatlosh DL, Baker AG. 1993. Rating causal relations: Role of probability in judgments of response ôutcome contingency. J Exp Psychol Learn Mem Cogn. 19:174-188.
58. Watkins S, Shams L, Tanaka S, Haynes JD, Rees G. 2006. Sound alters activity in human V1 in association with illusory visual perception. Neuroimage. 31:1247-1256.
59. Wittmann BC, Bunzeck N, Dolan RJ, Duzel E. 2007. Anticipation of novelty recruits reward system and hippocampus while promoting recollection. Neuroimage. 38:194-202.
60. Wolpert DM, Ghahramani Z, Jordan MI. 1995. An internal model for sensorimotor integration. Science. 269:1880-1882.
61. Zink CF, Pagnoni G, Chappelow J, Martin-Skurski M, Berns GS. 2006. Human striatal activation reflects degree of stimulus saliency. Neuroimage. 29:977-983.