Reinforcement learning signals predict future decisions

Journal of Neuroscience

Volumn 27, Issue 2, 2007, Pages 371-378

  Cohen, Michael X ^a,b   Ranganath, Charan ^b  

^a UNIVERSITY OF BONN   (Germany)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Decision making; Dopamine; ERN; Event related potential; Reinforcement learning; Reward prediction error

Indexed keywords

ADULT; ARTICLE; BEHAVIOR; COMPUTER MODEL; CONTROLLED STUDY; DECISION MAKING; EVENT RELATED POTENTIAL; FEEDBACK SYSTEM; FEMALE; HUMAN; HUMAN EXPERIMENT; LEARNING; MALE; MOTOR CORTEX; NORMAL HUMAN; OUTCOME ASSESSMENT; PREDICTION; PRIORITY JOURNAL; REINFORCEMENT; REWARD; SIGNAL TRANSDUCTION;

ADULT; DECISION MAKING; EVOKED POTENTIALS; FEMALE; FORECASTING; HUMANS; LEARNING; MALE; PREDICTIVE VALUE OF TESTS; REINFORCEMENT (PSYCHOLOGY); SIGNAL TRANSDUCTION;

EID: 33846225079     PISSN: 02706474     EISSN: 02706474     Source Type: Journal    
DOI: 10.1523/JNEUROSCI.4421-06.2007     Document Type: Article

References (68)

1. Alexander GE, Crutcher MD (1990) Functional architecture of basal ganglia circuits: neural substrates of parallel processing. Trends Neurosci 13:266-271.
2. Alexander GE, DeLong MR, Strick PL (1986) Parallel organization of functionally segregated circuits linking basal ganglia and cortex. Annu Rev Neurosci 9:357-381.
3. Barraclough DJ, Conroy ML, Lee D (2004) Prefrontal cortex and decision making in a mixed-strategy game. Nat Neurosci 7:404-410.
4. Barto AG (1995) Reinforcement learning. In: Handbook of brain theory and neural networks (Arbib MA, ed), pp 804-809. Cambridge, MA: MIT.
5. Bayer HM, Glimcher PW (2005) Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47:129-141.
6. Braver TS, Brown JW (2003) Principles of pleasure prediction: specifying the neural dynamics of human reward learning. Neuron 38:150-152.
7. Camerer CF (2003) Behavioral game theory: experiments in strategic interaction. Princeton: Princeton UP.
8. Carrillo-de-la-Pena MT, Lastra-Barreira C, Galdo-Alvarez S (2006) Limb (hand vs. foot) and response conflict have similar effects on event-related potentials (ERPs) recorded during motor imagery and overt execution. Eur J Neurosci 24:635-643.
9. Cohen MX (2006) Individual differences and the neural representations of reward expectation and reward prediction error. Soc Cogn Affect Neurosci, in press.
10. Cohen MX, Ranganath C (2005) Behavioral and neural predictors of upcoming decisions. Cogn Affect Behav Neurosci 5:117-126.
11. Dayan P, Balleine BW (2002) Reward, motivation, and reinforcement learning. Neuron 36:285-298.
12. Debener S, Ullsperger M, Siegel M, Fiehler K, von Cramon DY, Engel AK (2005) Trial-by-trial coupling of concurrent electroencephalogram and functional magnetic resonance imaging identifies the dynamics of performance monitoring. J Neurosci 25:11730-11737.
13. Donkers FC, Nieuwenhuis S, van Boxtel GJ (2005) Mediofrontal negativities in the absence of responding. Brain Res Cogn Brain Res 25:777-787.
14. Eblen F, Graybiel AM (1995) Highly restricted origin of prefrontal cortical inputs to striosomes in the macaque monkey. J Neurosci 15:5999-6013.
15. Egelman DM, Person C, Montague PR (1998) A computational role for dopamine delivery in human decision-making. J Cogn Neurosci 10:623-630.
16. Fiehler K, Ullsperger M, von Cramon DY (2005) Electrophysiological correlates of error correction. Psychophysiology 42:72-82.
17. Fiorillo CD, Tobler PN, Schultz W (2003) Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299:1898-1902.
18. Frank MJ (2005) Dynamic dopamine modulation in the basal ganglia: a neurocomputational account of cognitive deficits in medicated and nonmedicated Parkinsonism. J Cogn Neurosci 17:51-72.
19. Frank MJ, Woroch BS, Curran T (2005) Error-related negativity predicts reinforcement learning and conflict biases. Neuron 47:495-501.
20. Galdo-Alvarez S, Carrillo-de-la-Pena MT (2004) ERP evidence of MI activation without motor response execution. NeuroReport 15:2067-2070.
21. Garavan H, Ross TJ, Murphy K, Roche RA, Stein EA (2002) Dissociable executive functions in the dynamic control of behavior: inhibition, error detection, and correction. NeuroImage 17:1820-1829.
22. Gariano RF, Groves PM (1988) Burst firing induced in midbrain dopamine neurons by stimulation of the medial prefrontal and anterior cingulate cortices. Brain Res 462:194-198.
23. Gehring WJ, Fencsik DE (2001) Functions of the medial frontal cortex in the processing of conflict and errors. J Neurosci 21:9430-9437.
24. Gehring WJ, Willoughby AR (2002) The medial frontal cortex and the rapid processing of monetary gains and losses. Science 295:2279-2282.
25. Gehring WJ, Goss B, Coles MG, Meyer DE, Donchin E (1993) A neural system for error detection and compensation. Psychol Sci 4:385-390.
26. Gold JI, Shadlen MN (2000) Representation of a perceptual decision in developing oculomotor commands. Nature 404:390-394.
27. Gurney K, Prescott TJ, Redgrave P (2001) A computational model of action selection in the basal ganglia. I. A new functional anatomy. Biol Cybern 84:401-410.
28. Hajcak G, Holroyd CB, Moser JS, Simons RF (2005) Brain potentials associated with expected and unexpected good and bad outcomes. Psychophysiology 42:161-170.
29. Herrmann MJ, Rommler J, Ehlis AC, Heidrich A, Fallgatter AJ (2004) Source localization (LORETA) of the error-related-negativity (ERN/Ne) and positivity (Pe). Brain Res Cogn Brain Res 20:294-299.
30. Hewig J, Trippe R, Hecht H, Coles MG, Holroyd CB, Miltner WH (2006) Decision-making in blackjack: an electrophysiological analysis. Cereb Cortex, in press.
31. Holroyd CB, Coles MG (2002) The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychol Rev 109:679-709.
32. Holroyd CB, Coles MG, Nieuwenhuis S (2002) Medial prefrontal cortex and error potentials. Science 296:1610-1611; author reply 1610-1611.
33. Holroyd CB, Nieuwenhuis S, Yeung N, Cohen JD (2003) Errors in reward prediction are reflected in the event-related brain potential. NeuroReport 14:2481-2484.
34. Joel D, Weiner I (2000) The connections of the dopaminergic system with the striatum in rats and primates: an analysis with respect to the functional and compartmental organization of the striatum. Neuroscience 96:451-474.
35. Jung TP, Makeig S, Humphries C, Lee TW, McKeown MJ, Iragui V, Sejnowski TJ (2000) Removing electroencephalographic artifacts by blind source separation. Psychophysiology 37:163-178.
36. Kennerley SW, Walton ME, Behrens TE, Buckley MJ, Rushworth MF (2006) Optimal decision making and the anterior cingulate cortex. Nat Neurosci 9:940-947.
37. Lagarias JC, Reeds JA, Wright MH, Wright PE (1998) Convergence properties of the Nelder-Mead simplex method in low dimensions. SIAM J Optim 9:112-147.
38. Y, Ma Z, Lu W (2006) Automatic removal of the eye blink artifact from EEG using an ICA-based template matching approach. Physiol Meas 27:425-436.
39. Luce DP (1999) Individual choice behavior. New York: Wiley.
40. Montague PR, Berns GS (2002) Neural economics and the biological substrates of valuation. Neuron 36:265-284.
41. Montague PR, Hyman SE, Cohen JD (2004) Computational roles for dopamine in behavioural control. Nature 431:760-767.
42. Mookherjee D, Sopher B (1994) Learning behavior in an experimental matching pennies game. Games Econ Behav 7:62-91.
43. Mordkoff JT, Gianaros PJ (2000) Detecting the onset of the lateralized readiness potential: a comparison of available methods and procedures. Psychophysiology 37:347-360.
44. Nieuwenhuis S, Ridderinkhof KR, Blom J, Band GP, Kok A (2001) Error-related brain potentials are differentially related to awareness of response errors: evidence from an antisaccade task. Psychophysiology 38:752-760.
45. Nieuwenhuis S, Ridderinkhof KR, Talsma D, Coles MG, Holroyd CB, Kok A, van der Molen MW (2002) A computational account of altered error processing in older age: dopamine and the error-related negativity. Cogn Affect Behav Neurosci 2:19-36.
46. Nieuwenhuis S, Holroyd CB, Mol N, Coles MG (2004) Reinforcement-related brain potentials from medial frontal cortex: origins and functional significance. Neurosci Biobehav Rev 28:441-448.
47. Onn SP, Wang XB (2005) Differential modulation of anterior cingulate cortical activity by afferents from ventral tegmental area and mediodorsal thalamus. Eur J Neurosci 21:2975-2992.
48. Orieux G, Francois C, Feger J, Hirsch EC (2002) Consequences of dopaminergic denervation on the metabolic activity of the cortical neurons projecting to the subthalamic nucleus in the rat. J Neurosci 22:8762-8770.
49. Pascual-Marqui RD, Michel CM, Lehmann D (1994) Low resolution electromagnetic tomography: a new method for localizing electrical activity in the brain. Int J Psychophysiol 18:49-65.
50. Ridderinkhof KR, Nieuwenhuis S, Bashore TR (2003) Errors are foreshadowed in brain potentials associated with action monitoring in cingulate cortex in humans. Neurosci Lett 348:1-4.
51. Ridderinkhof KR, Ullsperger M, Crone EA, Nieuwenhuis S (2004a) The role of the medial frontal cortex in cognitive control. Science 306:443-447.
52. Ridderinkhof KR, van den Wildenberg WP, Segalowitz SJ, Carter CS (2004b) Neurocognitive mechanisms of cognitive control: the role of prefrontal cortex in action selection, response inhibition, performance monitoring, and reward-based learning. Brain Cogn 56:129-140.
53. Ruchsow M, Grothe J, Spitzer M, Kiefer M (2002) Human anterior cingulate cortex is activated by negative feedback: evidence from event-related potentials in a guessing task. Neurosci Lett 325:203-206.
54. Samejima K, Ueda Y, Doya K, Kimura M (2005) Representation of action-specific reward values in the striatum. Science 310:1337-1340.
55. Schall JD (1995) Neural basis of saccade target selection. Rev Neurosci 6:63-85.
56. Schall JD (2005) Decision making. Curr Biol 15:R9-R11.
57. Schultz W (2004) Neural coding of basic reward terms of animal learning theory, game theory, microeconomics and behavioural ecology. Curr Opin Neurobiol 14:139-147.
58. Schultz W, Dayan P, Montague PR (1997) A neural substrate of prediction and reward. Science 275:1593-1599.
59. Sugrue LP, Corrado GS, Newsome WT (2004) Matching behavior and the representation of value in the parietal cortex. Science 304:1782-1787.
60. Sutton RS (1992) Introduction: the challenge of reinforcement learning. Mach Learn 8:225-227.
61. Sutton RS, Barto AG (1998) Reinforcement learning. Cambridge, MA: MIT.
62. Taylor SF, Martis B, Fitzgerald KD, Welsh RC, Abelson JL, Liberzon I, Himle JA, Gehring WJ (2006) Medial frontal cortex activity and loss-related responses to errors. J Neurosci 26:4063-4070.
63. Waelti P, Dickinson A, Schultz W (2001) Dopamine responses comply with basic assumptions of formal learning theory. Nature 412:43-48.
64. Wang C, Ulbert I, Schomer DL, Marinkovic K, Halgren E (2005) Responses of human anterior cingulate cortex microdomains to error detection, conflict monitoring, stimulus-response mapping, familiarity, and orienting. J Neurosci 25:604-613.
65. Williams SM, Goldman-Rakic PS (1998) Widespread origin of the primate mesofrontal dopamine system. Cereb Cortex 8:321-345.
66. Yasuda A, Sato A, Miyawaki K, Kumano H, Kuboki T (2004) Error-related negativity reflects detection of negative reward prediction error. NeuroReport 15:2561-2565.
67. Yeung N, Sanfey AG (2004) Independent coding of reward magnitude and valence in the human brain. J Neurosci 24:6258-6264.
68. Yeung N, Holroyd CB, Cohen JD (2005) ERP correlates of feedback and reward processing in the presence and absence of response choice. Cereb Cortex 15:535-544.