Linking across levels of computation in model-based cognitive neuroscience

An Introduction to Model-Based Cognitive Neuroscience

Volumn , Issue , 2015, Pages 159-177

  Frank, Michael J ^a  

^a BROWN UNIVERSITY   (United States)

Author keywords

Algorithms; Basal ganglia; Computational models; Decision making; Dopamine; Neural networks; Prefrontal cortex; Reinforcement learning

Indexed keywords

EID: 84944582652     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1007/978-1-4939-2236-9_8     Document Type: Book

References (79)

1. Alexander GE, Crutcher MD (1990) Functional architecture of basal ganglia circuits: neural substrates of parallel processing. Trends Neurosci 13(7):266-271
2. Anderson JR (1991) The adaptive nature of human categorization. Psychol Rev 98(3):409-429
3. Aron AR, Behrens TE, Smith S, Frank MJ, Poldrack R (2007) Triangulating a cognitive control network using diffusion-weighted magnetic resonance imaging (MRI) and functional MRI. J Neurosci 27(14):3743-3752
4. Badre D, Frank MJ (2012) Mechanisms of hierarchical reinforcement learning in corticostriatal circuits 2: Evidence from fMRI. Cerebral Cortex 22:527-536
5. Badre D, Doll BB, Long NM, Frank MJ (2012) Rostrolateral prefrontal cortex and individual differences in uncertainty-driven exploration. Neuron 73:595-607
6. Bódi N, Kéri S, Nagy H, Moustafa A, Myers CE, Daw N, Dibó G, Takáts A, Bereczki D, Gluck MA (2009). Reward-learning and the novelty-seeking personality: a between- and within-subjects study of the effects of dopamine agonists on young Parkinson’s patients. Brain 132:2385-2395
7. Bogacz R, Gurney K (2007) The basal ganglia and cortex implement optimal decision making between alternative actions. Neural Comput 19(2):442-477
8. Bogacz R, Larsen T (2011) Integration of reinforcement learning and optimal decision-making theories of the basal ganglia. Neural Comput 23(4):817-851
9. Bogacz R, Wagenmaker EJ, Forstmann BU, Nieuwenhuis S (2010) The neural basis of the speed-accuracy tradeoff. Trends Neurosci 33(1):10-16
10. Botvinick MM, Niv Y, Barto AC (2009) Hierarchically organized behavior and its neural foundations: a reinforcement learning perspective. Cognition 113(3):262-280
11. Braver TS, Cohen JD (2000) On the control of control: the role of dopamine in regulating pre- frontal function and working memory. In: Monsell S, Driver J (eds) Control of cognitive processes: attention and performance XVIII. MIT Press, Cambridge, pp 713-737
12. Brittain JS, Watkins KE, Joundi RA, Ray NJ, Holland P, Green AL, Aziz TJ, Jenkinson N (2012) A role for the subthalamic nucleus in response inhibition during conflict. J Neurosci 32(39):13396-13401
13. Cavanagh JF, Frank MJ, Klein TJ, Allen JJB (2010) Frontal theta links prediction error to behavioral adaptation in reinforcement learning. Neuroimage 49(4):3198-3209
14. Cavanagh JF, Figueroa CM, Cohen MX, Frank MJ (2011a) Frontal theta reflects uncertainty and unexpectedness during exploration and exploitation. Cereb Cortex 22(11):2575-2586
15. Cavanagh JF, Wiecki TV, Cohen MX, Figueroa CM, Samanta J, Sherman SJ, Frank MJ (2011b) Subthalamic nucleus stimulation reverses mediofrontal influence over decision threshold. Nat Neurosci 14(11):1462-1467
16. Collins AGE, Frank MJ (2012) How much of reinforcement learning is working memory, not reinforcement learning? A behavioral, computational, and neurogenetic analysis. Eur J Neurosci 35(7):1024-1035
17. Collins AGE, Frank MJ (2013) Cognitive control over learning: creating, clustering and generalizing task-set structure. Psychol Rev 120(1):190-229
18. Collins AGE, Frank MJ (2013) Opponent Actor Learning (OpAL): modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive. Psychol Rev 121:337-366
19. Coulthard EJ, Bogacz R, Javed S, Mooney LK, Murphy G, Keeley S, Whone AL (2012). Distinct roles of dopamine and subthalamic nucleus in learning and probabilistic decision making. Brain 135:3721-3734
20. Dayan P, Sejnowksi T (1996) Exploration bonuses and dual control. Mach Learn 25:5-22
21. Doll BB, Jacobs WJ, Sanfey AG, Frank MJ (2009) Instructional control of reinforcement learning: a behavioral and neurocomputational investigation. Brain Res 1299:74-94
22. Doll BB, Hutchison KE, Frank MJ (2011) Dopaminergic genes predict individual differences in susceptibility to confirmation bias. J Neurosci 31(16):6188-6198
23. Ford KA, Everling S (2009) Neural activity in primate caudate nucleus associated with proand antisaccades. J Neurophysiol 102(4):2334-2341
24. Forstmann BU, Dutilh G, Brown S, Neumann J, von Cramon DY, Ridderinkhof KR, Wagenmakers EJ (2008a) Striatum and pre-SMA facilitate decision-making under time pressure. Proc Natl Acad Sci USA 105(45):17538-17542
25. Forstmann BU, Jahfari S, Scholte HS, Wolfensteller U, van den Wildenberg WP, Ridderinkhof KR (2008b) Function and structure of the right inferior frontal cortex predict individual differences in response inhibition: a model-based approach. J Neurosci 28(39):9790-9796
26. Frank MJ (2005) Dynamic dopamine modulation in the basal ganglia: a neurocomputational account of cognitive deficits in medicated and nonmedicated Parkinsonism. J Cogn Neurosci 17(1):51-72
27. Frank MJ (2006) Hold your horses: a dynamic computational role for the subthalamic nucleus in decision making. Neural Netw 19(8):1120-1136
28. Frank MJ (2011) Computational models of motivated action selection in corticostriatal circuits. Curr Opin Neurobiol 2:381-386
29. Frank MJ, Badre D (2012) Mechanisms of hierarchical reinforcement learning in corticostriatal circuits 1: computational analysis. Cereb Cortex 22(3):509-526
30. Frank MJ, Claus ED (2006) Anatomy of a decision: striato-orbitofrontal interactions in reinforcement learning, decision making, and reversal. Psychol Rev 113(2):300-326
31. Frank MJ, Fossella JA (2011) Neurogenetics and pharmacology of learning, motivation, and cognition. Neuropsychopharmacology 36:133-152
32. Frank MJ, Hutchison K (2009) Genetic contributions to avoidance-based decisions: striatal D2 receptor polymorphisms. Neuroscience 164(1):131-140
33. Frank MJ, O’Reilly RC (2006) A mechanistic account of striatal dopamine function in human cognition: psychopharmacological studies with cabergoline and haloperidol. Behav Neurosci 120(3):497-517
34. Frank MJ, Loughry B, O’Reilly RC (2001) Interactions between frontal cortex and basal ganglia in working memory: a computational model. Cogn Affect Behav Neurosci 1(2):137-160
35. Frank MJ, Seeberger LC, O’Reilly RC (2004) By carrot or by stick: cognitive reinforcement learning in parkinsonism. Science 306(5703):1940-1943
36. Frank MJ, Santamaria A, O’Reilly R, Willcutt E (2007a) Testing computational models of dopamine and noradrenaline dysfunction in attention deficit/hyperactivity disorder. Neuropsychopharmacology 32(7):1583-1599
37. Frank MJ, D’Lauro C, Curran T (2007b) Cross-task individual differences in error processing: neural, electrophysiological, and genetic components. Cogn Affect Behav Neurosci 7(4):297-308
38. Frank MJ, Moustafa AA, Haughey H, Curran T, Hutchison K (2007c) Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning. Proc Natl Acad Sci 104(41):16311-16316
39. Frank MJ, Samanta J, Moustafa AA, Sherman SJ (2007d) Hold your horses: impulsivity, deep brain stimulation and medication in Parkinsonism. Science 318:1309-1312
40. Frank MJ, Doll BB, Oas-Terpstra J, Moreno F (2009). Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation. Nat Neurosci 12(8):1062-1068
41. Frank MJ, Gagne C, Nyhus E, Masters S, Wiecki TV, Cavanagh JF, Badre D (2015) fMRI and EEG Predictors of dynamic decision parameters during human reinforcement learning. J Neurosci 35:484-494
42. Frazier P, Yu AJ (2008) Sequential hypothesis testing under stochastic deadlines. Adv Neural Inf Process Syst 20:465-472. (MIT Press, Cambridge)
43. Gurney K, Prescott TJ, Redgrave P (2001) A computational model of action selection in the basal ganglia. II. Analysis and simulation of behaviour. Biol Cybern 84(6):411-423
44. Hikida T, Kimura K, Wada N, Funabiki K, Nakanishi S (2010) Distinct roles of synaptic transmission in direct and indirect striatal pathways to reward and aversive behavior. Neuron 66(6):896-907
45. Isoda M, Hikosaka O (2007) Switching from automatic to controlled action by monkey medial frontal cortex. Nat Neurosci 10(2):240-248
46. Isoda M, Hikosaka O (2008) Role for subthalamic nucleus neurons in switching from automatic to controlled eye movement. J Neurosci 28(28):7209-7218
47. Jahfari S, Verbruggen F, Frank MJ, Waldorp LJ, Colzato L, Ridderinkhof KR, Forstmann BU (2012) How preparation changes the need for top-down control of the basal ganglia when inhibiting premature actions. J Neurosci 32(32):10870-10878
48. Koechlin E, Summerfield C (2007) An information theoretical approach to the prefrontal executive function. Trends Cogn Sci 11(6):229-235
49. Kravitz AV, Freeze BS, Parker PRL, Kay K, Thwin MT, Deisseroth K, Kreitzer AC (2010) Regulation of parkinsonian motor behaviours by optogenetic control of basal ganglia circuitry. Nature 466(7306):622-626
50. Kravitz AV, Tye LD, Kreitzer AC (2012) Distinct roles for direct and indirect pathway striatal neurons in reinforcement. Nat Neurosci 15:816-818
51. Lau B, Glimcher PW (2008) Value representations in the primate striatum during matching behavior. Neuron 58(3):451-463
52. Lo CC, Wang XJ (2006) Cortico-basal ganglia circuit mechanism for a decision threshold in reaction time tasks. Nat Neurosci 9(7):956-963
53. Maia TV, Frank MJ (2011) From reinforcement learning models to psychiatric and neurological disorders. Nat Neurosci 2:154-162
54. Mink JW (1996) The basal ganglia: focused selection and inhibition of competing motor programs. Prog Neurobiol 50(4):381-425
55. Munafò MR, Stothart G, Flint J (2009) Bias in genetic association studies and impact factor. Mol Psychiatry 14:119-120
56. O’Reilly RC, McClelland JL (1994) Hippocampal conjunctive encoding, storage, and recall: avoiding a trade-off. Hippocampus 4(6):661-682
57. Pessiglione M, Seymour B, Flandin G, Dolan RJ, Frith CD (2006) Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature 442(7106):1042-1045
58. Ratcliff R, Frank MJ (2012) Reinforcement-based decision making in corticostriatal circuits: mutual constraints by neurocomputational and diffusion models. Neural Comput 24:1186-1229
59. Ratcliff R, McKoon G (2008) The diffusion decision model: theory and data for two-choice decision tasks. Neural Comput 20(4):873-922
60. Reynolds JR, O’Reilly RC (2009) Developing PFC representations using reinforcement learning. Cognition 113(3):281-292
61. Samejima K, Ueda Y, Doya K, Kimura M (2005) Representation of action-specific reward values in the striatum. Science 310(5752):1337-1340
62. Sanborn AN, Griffiths TL, Navarro DJ (2010) Rational approximations to rational models: alternative algorithms for category learning. Psychol Rev 117(4):1144-1167
63. Schultz W (2002) Getting formal with dopamine and reward. Neuron 36(2):241-263
64. Shen W, Flajolet M, Greengard P, Surmeier DJ (2008) Dichotomous dopaminergic control of striatal synaptic plasticity. Science 321(5890):848-851
65. Strauss GP, Frank MJ, Waltz JA, Kasanova Z, Herbener ES, Gold JM (2011) Deficits in positive reinforcement learning and uncertainty-driven exploration are associated with distinct aspects of negative symptoms in schizophrenia. Biol Psychiatry 69:424-431
66. Tai LH, Lee AM, Benavidez N, Bonci A, Wilbrecht L (2012) Transient stimulation of distinct subpopulations of striatal neurons mimics changes in action value. Nat Neurosci 15:1281-1289
67. Tan KR, Yvon C, Turiault M, Mirabekov JJ, Doehner J, Labouèbe G, Deisseroth K, Tye KM, Lüscher C (2012) GABA neurons of the VTA drive conditioned place aversion. Neuron 73:1173-1183
68. Tsai HC, Zhang F, Adamantidis A, Stuber GD, Bonci A, de Lecea L, Deisseroth K (2009) Phasic firing in dopaminergic neurons is sufficient for behavioral conditioning. Science 324:1080-1084
69. Voon V, Pessiglione M, Brezing C, Gallea C, Fernandez HH, Dolan RJ, Hallett M (2010) Mechanisms underlying dopamine-mediated reward bias in compulsive behaviors. Neuron 65(1):135-142
70. Wang XJ (2012) Neural dynamics and circuit mechanisms of decision-making. Curr Opin Neurobiol 22:1039-1046
71. Watanabe M, Munoz DP (2009) Neural correlates of conflict resolution between automatic and volitional actions by basal ganglia. Eur J Neurosci 30(11):2165-2176
72. Watkins CJCH, Dayan P (1992) Q-Learning. Mach Learn 8:279-292
73. Wiecki TV, Frank MJ (2010) Neurocomputational models of motor and cognitive deficits in Parkinson’s disease. Prog Brain Res 183:275-297
74. Wiecki TV, Frank MJ (in press). A computational model of inhibitory control in frontal cortex and basal ganglia. Psychological Review.
75. Wiecki TV, Riedinger K, Meyerhofer A, Schmidt W, Frank MJ (2009) A neurocomputational account of catalepsy sensitization induced by D2 receptor blockade in rats: context dependency, extinction, and renewal. Psychopharmacology (Berl) 204:265-277
76. Wiecki TV, Sofer I, Frank MJ (2012). Hierarchical Bayesian parameter estimation of Drift Diffusion Models (Version0.4RC1) [software]. http://ski.clps.brown.edu/hddm_docs/.
77. Wiecki TV, Sofer I, Frank MJ (2013) HDDM: Hierarchical Bayesian estimation of the Drift-Diffusion Model in Python. Fron Neuroinformatics 7:1-10
78. Wong KF, Wang XJ (2006) A recurrent network mechanism of time integration in perceptual decisions. J Neurosci 26(4):1314-1328
79. Zaghloul K, Weidemann CT, Lega BC, Jaggi JL, Baltuch GH, Kahana MJ (2012) Neuronal activity in the human subthalamic nucleus encodes decision conflict during action selection. J Neurosci 32(7):2453-2460