FMRI and EEG predictors of dynamic decision parameters during human reinforcement learning

Journal of Neuroscience

Volumn 35, Issue 2, 2015, Pages 485-494

  Frank, Michael J ^a,b   Gagne, Chris ^a   Nyhus, Erika ^a,c   Masters, Sean ^a   Wiecki, Thomas V ^a   Cavanagh, James F ^a,d   Badre, David ^a  

^a BROWN UNIVERSITY   (United States)

^b BROWN UNIVERSITY   (United States)

^c BOWDOIN COLLEGE   (United States)

^d UNIVERSITY OF NEW MEXICO   (United States)

Author keywords

Basal ganglia; Decision making; Drift diffusion model; Prefrontal cortex; Subthalamic nucleus

Indexed keywords

ADULT; ARTICLE; BASAL GANGLION; BAYESIAN LEARNING; BEHAVIOR; CONFLICT; DECISION MAKING; DRIFT DIFFUSION MODEL; ELECTROENCEPHALOGRAPHY; EVOKED MUSCLE RESPONSE; EYE MOVEMENT; EYELID REFLEX; FEMALE; FRONTAL CORTEX; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HUMAN; MALE; MEMORY; PRIORITY JOURNAL; PSYCHOPHYSIOLOGY; REINFORCEMENT; REWARD; STATISTICAL MODEL; SUBTHALAMIC NUCLEUS; VISUAL FEEDBACK; VISUAL STIMULATION; ADOLESCENT; BIOLOGICAL MODEL; CONDITIONING; FRONTAL LOBE; NUCLEAR MAGNETIC RESONANCE IMAGING; PHYSIOLOGY; THETA RHYTHM;

ADOLESCENT; ADULT; CONDITIONING (PSYCHOLOGY); DECISION MAKING; ELECTROENCEPHALOGRAPHY; FEMALE; FRONTAL LOBE; HUMANS; MAGNETIC RESONANCE IMAGING; MALE; MODELS, NEUROLOGICAL; REINFORCEMENT (PSYCHOLOGY); SUBTHALAMIC NUCLEUS; THETA RHYTHM;

EID: 84920972979     PISSN: 02706474     EISSN: 15292401     Source Type: Journal    
DOI: 10.1523/JNEUROSCI.2036-14.2015     Document Type: Article

References (47)

1. Allen PJ, Polizzi G, Krakow K, Fish DR, Lemieux L (1998) Identification of EEG events in theMRscanner: the problem of pulse artifact and a method for its subtraction. Neuroimage 8:229-239.CrossRef Medline
2. Allen PJ, Josephs O, Turner R (2000) A method for removing imaging artifact from continuous EEG recorded during functional MRI. Neuroimage 12:230-239.CrossRef Medline
3. Aron AR, Behrens TE, Smith S, Frank MJ, Poldrack RA (2007) Triangulating a cognitive control network using diffusion-weighted magnetic resonance imaging (MRI) and functional MRI. J Neurosci 27:3743–3752.CrossRef Medline
4. Brett M, Anton JL, Valabregue R, Poline JB (2002) Region of interest analysis using an SPM toolbox. Japan: Sendai.
5. Cavanagh JF, Frank MJ (2014) Frontal theta as a mechanism for cognitive control. Trends Cogn Sci 18:414–421.CrossRef Medline
6. Cavanagh JF, Shackman AJ (2014) Frontal midline theta reflects anxiety andcognitive control: meta-analytic evidence. J Physiol Paris S0928-4257(14)00014-X.
7. Cavanagh JF, Wiecki TV, Cohen MX, Figueroa CM, Samanta J, Sherman SJ, Frank MJ (2011) Subthalamic nucleus stimulation reverses mediofrontal influence over decision threshold. Nat Neurosci 14:1462–1467.CrossRef Medline
8. Cavanagh JF, Eisenberg I, Guitart-Masip M, Huys Q, Frank MJ (2013) Frontal theta overrides pavlovian learning biases. J Neurosci 33:8541-8548.CrossRef Medline
9. Cavanagh JF, Wiecki TV, Kochar A, Frank MJ (2014) Eye tracking and pupillometry are indicators of dissociable latent decision processes. J Exp Psychol Gen 143:1476-1488.CrossRef Medline
10. Cisek P, Puskas GA, El-Murr S (2009) Decisions in changing conditions: the urgency-gating model. J Neurosci 29:11560-11571.CrossRef Medline
11. Dale AM (1999) Optimal experimental design for event-related fMRI. Hum Brain Mapp 8:109-114.CrossRef Medline
12. Delorme A, Makeig S (2004) EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J Neurosci Methods 134:9-21.CrossRef Medline
13. Doll BB, Jacobs WJ, Sanfey AG, Frank MJ (2009) Instructional control of reinforcement learning: a behavioral and neurocomputational investigation. Brain Res 1299:74-94.CrossRef Medline
14. Doll BB, Hutchison KE, Frank MJ (2011) Dopaminergic genes predict individual differences in susceptibility to confirmation bias. J Neurosci 31: 6188–6198.CrossRef Medline
15. Erev I, Barron G (2005) On adaptation, maximization, and reinforcement learning among cognitive strategies. Psychol Rev 112:912–931.CrossRef Medline
16. Forstmann BU, Keuken MC, Jahfari S, Bazin PL, Neumann J, Schäfer A, Anwander A, Turner R (2012) Cortico-subthalamic white matter tract strength predicts interindividual efficacy in stopping a motor response. Neuroimage 60:370-375.CrossRef Medline
17. Frank MJ (2006) Hold your horses: a dynamic computational role for the subthalamic nucleus in decision making. Neural Netw 19:1120-1136.CrossRef Medline
18. Frank MJ, Samanta J, Moustafa AA, Sherman SJ (2007a) Hold your horses: impulsivity, deep brain stimulation, and medication in parkinsonism. Science 318:1309-1312.CrossRef Medline
19. Frank MJ, Moustafa AA, Haughey HM, Curran T, Hutchison KE (2007b) Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning. Proc Natl Acad SciU S A 104:16311–16316.CrossRef Medline
20. Goense JB, Logothetis NK (2008) Neurophysiology of the BOLD fMRI signal in awake monkeys. Curr Biol 18:631-640.CrossRef Medline
21. Green N, Bogacz R, Huebl J, Beyer AK, Ku¨hn AA, Heekeren HR (2013) Reduction of influence of task difficulty on perceptual decision making by STN deep brain stimulation. Curr Biol 23:1681–1684.CrossRef Medline
22. Hare TA, Schultz W, Camerer CF, O’Doherty JP, Rangel A (2011) Transformation of stimulus value signals into motor commands during simple choice. Proc Natl Acad Sci U S A 108:18120-18125.CrossRef Medline
23. Haruno M, Kawato M (2006) Heterarchical reinforcement-learning model for integration of multiple cortico-striatal loops: fMRI examination in stimulus-action-reward association learning. Neural Netw 19:1242–1254.CrossRef Medline
24. Isoda M, Hikosaka O (2007) Switching from automatic to controlled action by monkey medial frontal cortex. Nat Neurosci 10:240-248.CrossRef Medline
25. Jocham G, Klein TA, Ullsperger M (2011) Dopamine-mediated reinforcement learning signals in the striatum and ventromedial prefrontal cortex underlie value-based choices. J Neurosci 31:1606-1613.CrossRef Medline
26. Kayser J, Tenke CE (2006) Principal components analysis of Laplacian waveforms as a generic method for identifying ERP generator patterns: I. evaluation with auditory oddball tasks. Clin Neurophysiol 117:369-380.CrossRef Medline
27. Krajbich I, Armel C, Rangel A (2010) Visual fixations and the computation and comparison of value in simple choice. Nat Neurosci 13:1292–1298.CrossRef Medline
28. Kruschke JK (2011) Doing Bayesian data analysis: a tutorial with Rand BUGS. Burlington, MA: Academic.
29. Lim SL, O’Doherty JP, Rangel A (2011) The decision value computations in the VMPFC and striatum use a relative value code that is guided by visual attention. J Neurosci 31:13214-13223.CrossRef Medline
30. Luck SJ (2005) An introduction to the event-related potential technique. Cambridge, MA:MIT.
31. Matzke D, Wagenmakers EJ (2009) Psychological interpretation of the ex- Gaussian and shifted Wald parameters: a diffusion model analysis. Psychon Bull Rev 16:798–817.CrossRef Medline
32. Niessing J, Ebisch B, Schmidt KE, Niessing M, Singer W, Galuske RA (2005) Hemodynamic signals correlate tightly with synchronized gamma oscillations. Science 309:948-951.CrossRef Medline
33. Pessiglione M, Seymour B, Flandin G, Dolan RJ, Frith CD (2006) Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature 442:1042–1045.CrossRef Medline
34. Ratcliff R, Frank MJ (2012) Reinforcement-based decision making in corticostriatal circuits: mutual constraints by neurocomputational and diffusion models. Neural Comput 24:1186-1229.CrossRef Medline
35. Ratcliff R, McKoon G (2008) The diffusion decision model: theory and data for two-choice decision tasks. Neural Comput 20:873–922.CrossRef Medline
36. Samejima K, Ueda Y, Doya K, Kimura M (2005) Representation of actionspecific reward values in the striatum. Science 310:1337–1340.CrossRef Medline
37. Spiegelhalter DJ, Best NG, Carlin BP, van der Linde A (2002) Bayesian measures of model complexity and fit. J Royal Stat Soc Series B 64:83-639. CrossRef
38. Srinivasan R, Nunez PL, Tucker DM, Silberstein RB, Cadusch PJ (1996) Spatial sampling and filtering of EEG with spline Laplacians to estimate cortical potentials. Brain Topogr 8:355–366.CrossRef Medline
39. Stewart N, Chater N, Brown GD (2006) Decision by sampling. Cogn Psychol 53:1–26.CrossRef Medline
40. Ullsperger M, von Cramon DY (2001) Subprocesses of performance monitoring: a dissociation of error processing and response competition revealed by event-related fMRI and ERPs. Neuroimage 14:1387–1401.CrossRef Medline
41. Usami K, Matsumoto R, Kunieda T, Shimotake A, Matsuhashi M, Miyamoto S, Fukuyama H, Takahashi R, Ikeda A (2013) Pre-SMA actively engages in conflict processing in human: a combined study of epicortical ERPs and direct cortical stimulation. Neuropsychologia 51:1011–1017.CrossRef Medline
42. Voytek B, D’Esposito M, Crone N, Knight RT (2013) A method for eventrelated phase/amplitude coupling. Neuroimage 64:416–424.CrossRef Medline
43. Wagenmakers EJ, Lodewyckx T, Kuriyal H, Grasman R (2010) Bayesian hypothesis testing for psychologists: a tutorial on the savage-dickey method. Cogn Psychol 60:158-189.CrossRef Medline
44. Wiecki TV, Frank MJ (2013) A computational model of inhibitory control in frontal cortex and basal ganglia. Psychol Rev 120:329-355.CrossRef Medline
45. Wiecki TV, Sofer I, Frank MJ (2013) HDDM: hierarchical Bayesian estimation of the drift-diffusion model in python. Front Neuroinform 7:14.CrossRef Medline
46. Zaghloul KA, Weidemann CT, Lega BC, Jaggi JL, Baltuch GH, Kahana MJ (2012) Neuronal activity in the human subthalamic nucleus encodes decision conflict during action selection. J Neurosci 32:2453–2460.CrossRef Medline
47. Zavala BA, Tan H, Little S, Ashkan K, Hariz M, Foltynie T, Zrinzo L, Zaghloul KA, Brown P (2014) Midline frontal cortex low-frequency activity drives subthalamic nucleus oscillations during conflict. J Neurosci 34: 7322–7333.CrossRef Medline