How much of reinforcement learning is working memory, not reinforcement learning? A behavioral, computational, and neurogenetic analysis

European Journal of Neuroscience

Volumn 35, Issue 7, 2012, Pages 1024-1035

  Collins, Anne G E ^a   Frank, Michael J ^a  

^a BROWN UNIVERSITY   (United States)

Author keywords

Basal ganglia; Capacity; Human; Prefrontal cortex; Striatum

Indexed keywords

CATECHOL METHYLTRANSFERASE; G PROTEIN COUPLED RECEPTOR; G PROTEIN COUPLED RECEPTOR 6; UNCLASSIFIED DRUG;

ADULT; ARTICLE; BASAL GANGLION; BEHAVIOR; CAUCASIAN; COMT GENE; CONTROLLED STUDY; ETHNICITY; FEMALE; GENE; GENE DOSAGE; GENOTYPE; GPR6 GENE; HOMOZYGOTE; HUMAN; HUMAN EXPERIMENT; LEARNING; MALE; MATHEMATICAL MODEL; PREFRONTAL CORTEX; PRIORITY JOURNAL; REINFORCEMENT; SINGLE NUCLEOTIDE POLYMORPHISM; TASK PERFORMANCE; WORKING MEMORY;

ADOLESCENT; ADULT; CATECHOL O-METHYLTRANSFERASE; COMPUTATIONAL BIOLOGY; FEMALE; HUMANS; LEARNING; MALE; MEMORY, SHORT-TERM; MODELS, NEUROLOGICAL; PHOTIC STIMULATION; POLYMORPHISM, SINGLE NUCLEOTIDE; PSYCHOMOTOR PERFORMANCE; RECEPTORS, G-PROTEIN-COUPLED; REINFORCEMENT (PSYCHOLOGY); YOUNG ADULT;

EID: 84859317336     PISSN: 0953816X     EISSN: 14609568     Source Type: Journal    
DOI: 10.1111/j.1460-9568.2011.07980.x     Document Type: Article

References (70)

1. Baier, B., Karnath, H.-O., Dieterich, M., Birklein, F., Heinze, C. & Muller, N.G. (2010) Keeping memory clear and stable - the contribution of human basal ganglia and prefrontal cortex to working memory. J. Neurosci., 30, 9788-9792.
2. Bayer, H.M. & Glimcher, P.W. (2005) Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron, 47, 129-141.
3. Bays, P.M. & Husain, M. (2008) Dynamic shifts of limited working memory resources in human vision. Science, 321, 851-854. doi: .
4. Behrens, T.E.J., Woolrich, M.W., Walton, M.E. & Rushworth, M.F.S. (2007) Learning the value of information in an uncertain world. Nat. Neurosci., 10, 1214-1221.
5. Blasi, G., Mattay, V.S., Bertolino, A., Elvevåg, B., Callicott, J.H., Das, S., Kolachana, B.S., Egan, M.F., Goldberg, T.E. & Weinberger, D.R. (2005) Effect of catechol-O-methyltransferase val158met genotype on attentional control. J. Neurosci., 25, 5038-5045.
6. Botvinick, M., Niv, Y. & Barto, A. (2009) Hierarchically organized behavior and its neural foundations: A reinforcement-learning perspective. Cognition, 113, 262-280.
7. Brovelli, A., Laksiri, N., Nazarian, B., Meunier, M. & Boussaoud, D. (2008) Understanding the neural computations of arbitrary visuomotor learning through fMRI and associative learning theory. Cereb. Cortex, 18, 1485-1495.
8. Bruder, G.E., Keilp, J.G., Xu, H., Shikhman, M., Schori, E., Gorman, J.M. & Gilliam, T.C. (2005) Catechol-O-methyltransferase (COMT) genotypes and working memory: associations with differing cognitive operations. Biol. Psychiatry, 58, 901-907.
9. Cavanagh, J.F., Frank, M.J., Klein, T.J. & Allen, J.J.B. (2010) Frontal theta links prediction errors to behavioral adaptation in reinforcement learning. NeuroImage, 49, 3198-3209.
10. Cools, R., Sheridan, M., Jacobs, E. & D'Esposito, M. (2007) Impulsive personality predicts dopamine-dependent changes in frontostriatal activity during component processes of working memory. J. Neurosci., 27, 5506-5514.
11. Corrado, G. & Doya, K. (2007) Understanding neural coding through the model-based analysis of decision making. J. Neurosci., 27, 8178-8180.
12. Cowan, N. (2010) The magical mystery four: how is working memory capacity limited, and why? Curr. Dir. Psychol. Sci., 19, 51-57.
13. Dahlin, E., Neely, A.S., Larsson, A., Bäckman, L. & Nyberg, L. (2008) Transfer of learning after updating training mediated by the striatum. Science, 320, 1510-1512.
14. Daw, N.D. (2011) Trial-by-trial data analysis using computational models. In Phelps, E.A., Robbins, T.W. & Delgado, M. (Eds), Affect, Learning and Decision Making, Attention and Performance XXIII. Oxford University Press, New York, pp. 3-38.
15. Daw, N.D. & Doya, K. (2006) The computational neurobiology of learning and reward. Curr. Opin. Neurobiol., 16, 199-204.
16. Daw, N.D., Niv, Y. & Dayan, P. (2005) Uncertainty-based competition between prefrontal and dorso-lateral striatal systems for behavioral control. Nat. Neurosci., 8, 1704-1711.
17. Daw, N.D., Gershman, S.J., Seymour, B., Dayan, P. & Dolan, R.J. (2011) Model-based influences on humans' choices and striatal prediction errors. Neuron, 69, 1204-1215.
18. Doll, B.B., Hutchison, K.E. & Frank, M.J. (2011) Dopaminergic genes predict individual differences in susceptibility to confirmation bias. J. Neurosci., 31, 6188-6198.
19. Doya, K. (2002) Metalearning and neuromodulation. Neural Netw., 15, 495-506.
20. Durstewitz, D. & Seamans, J.K. (2008) The dual-state theory of prefrontal cortex dopamine function with relevance to catechol-o-methyltransferase genotypes and schizophrenia. Biol. Psychiatry, 64, 739-749.
21. Durstewitz, D., Vittoz, N.M., Floresco, S.B. & Seamans, J.K. (2010) Abrupt transitions between prefrontal neural ensemble states accompany behavioral transitions during rule learning. Neuron, 66, 438-448.
22. Egan, M.F., Goldberg, T.E., Kolachana, B.S., Callicott, J.H., Mazzanti, C.M., Straub, R.E., Goldman, D. & Weinberger, D.R. (2001) Effect of COMT Val108/158 Met genotype on frontal lobe function and risk for schizophrenia. Proc. Natl. Acad. Sci. USA, 98, 6917-6922.
23. Ernst, C., Sequeira, A., Klempan, T., Ernst, N., ffrench-Mullen, J. & Turecki, G. (2007) Confirmation of region-specific patterns of gene expression in the human brain. Neurogenetics, 8, 219-224.
24. Evans, J.S.B.T. (2003) In two minds: dual-process accounts of reasoning. Trends Cogn. Sci., 7, 454-459.
25. Frank, M.J. (2005) Dynamic dopamine modulation in the basal ganglia: a neurocomputational account of cognitive deficits in medicated and nonmedicated Parkinsonism. J. Cogn. Neurosci., 17, 51-72.
26. Frank, M.J. & Badre, D. (2011) Mechanisms of hierarchical reinforcement learning in corticostriatal circuits 1: computational analysis. Cereb. Cortex, doi:.
27. Frank, M.J. & Fossella, J.A. (2010) Neurogenetics and pharmacology of learning, motivation, and cognition. Neuropsychopharmacology, 36, 133-152.
28. Frank, M.J., Loughry, B. & O'Reilly, R.C. (2001) Interactions between frontal cortex and basal ganglia in working memory: a computational model. Cogn. Affect. Behav. Neurosci., 1, 137-160.
29. Frank, M.J., Moustafa, A.A., Haughey, H.M., Curran, T. & Hutchison, K.E. (2007) Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning. Proc. Natl. Acad. Sci. USA, 104, 16311-16316.
30. Frank, M., Doll, B., Oas-Terpstra, J. & Moreno, F. (2009a) The neurogenetics of exploration and exploitation Supplementary Material. Nature Neuroscience, 12, 1062.
31. Frank, M.J., Doll, B.B., Oas-Terpstra, J. & Moreno, F. (2009b) Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation. Nat. Neurosci., 12, 1062-1068.
32. de Frias, C.M., Marklund, P., Eriksson, E., Larsson, A., Oman, L., Annerbrink, K., Bäckman, L., Nilsson, L.-G. & Nyberg, L. (2010) Influence of COMT gene polymorphism on fMRI-assessed sustained and transient activity during a working memory task. J. Cogn. Neurosci., 22, 1614-1622.
33. Gogos, J.A., Morgan, M., Luine, V., Santha, M., Ogawa, S., Pfaff, D. & Karayiorgou, M. (1998) Catechol-O-methyltransferase-deficient mice exhibit sexually dimorphic changes in catecholamine levels and behavior. Proc. Natl. Acad. Sci. USA, 95, 9991-9996.
34. Goldberg, T.E. & Weinberger, D.R. (2004) Genes and the parsing of cognitive processes. Trends Cogn. Sci., 8, 325-335.
35. Green, A., Munafò, M., DeYoung, C., Fossella, J., Fan, J. & Gray, J. (2008) using genetic data in cognitive neuroscience: from growing pains to genuine insights. Nat. Rev. Neurosci., 9, 710-720.
36. Gruber, A.J., Dayan, P., Gutkin, B.S. & Solla, S.A. (2006) Dopamine modulation in the basal ganglia locks the gate to working memory. J. Comput. Neurosci., 20, 153-166.
37. Hampton, A.N., Bossaerts, P. & O'Doherty, J.P. (2006) The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. J. Neurosci., 26, 8360-8367.
38. Huotari, M., Gogos, J.A., Karayiorgou, M., Koponen, O., Forsberg, M., Raasmaja, A., Hyttinen, J. & Männistö, P.T. (2002) Brain catecholamine metabolism in catechol-O-methyltransferase (COMT)-deficient mice. Eur. J. Neurosci., 15, 246-256.
39. Imamizu, H., Sugimoto, N., Osu, R., Tsutsui, K., Sugiyama, K., Wada, Y. & Kawato, M. (2007) Explicit contextual information selectively contributes to predictive switching of internal models. Exp. Brain Res., 181, 395-408.
40. Jocham, G., Klein, T.A. & Ullsperger, M. (2011) Dopamine-mediated reinforcement learning signals in the striatum and ventromedial prefrontal cortex underlie value-based choices. J. Neurosci., 31, 1606-1613.
41. Kahnt, T., Park, S.Q., Cohen, M.X., Beck, A., Heinz, A. & Wrase, J. (2009) Dorsal striatal-midbrain connectivity in humans predicts how reinforcements are used to guide decisions. J. Cogn. Neurosci., 21, 1332-1345.
42. Lau, B. & Glimcher, P.W. (2007) Action and outcome encoding in the primate caudate nucleus. J. Neurosci., 27, 14502-14514. doi: .
43. Lengyel, M. & Dayan, P. (2007) Hippocampal contributions to control: the third way. NIPS, 20, 889-896.
44. Lobo, M.K., Cui, Y., Ostlund, S.B., Balleine, B.W. & Yang, X.W. (2007) Genetic control of instrumental conditioning by striatopallidal neuron-specific S1P receptor Gpr6. Nat. Neurosci., 10, 1395-1397.
45. Matsumoto, M., Weickert, C.S., Akil, M., Lipska, B.K., Hyde, T.M., Herman, M.M., Kleinman, J.E. & Weinberger, D.R. (2003) Catechol O-methyltransferase mRNA expression in human and rat brain: evidence for a role in cortical neuronal function. Neuroscience, 116, 127-137.
46. Mier, D., Kirsch, P. & Meyer-Lindenberg, A. (2010) Neural substrates of pleiotropic action of genetic variation in COMT: a meta-analysis. Mol. Psychiatry, 15, 918-927.
47. Montague, P.R., Dayan, P. & Sejnowski, T.J. (1996) A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci., 16, 1936-1947.
48. Moustafa, A.A., Sherman, S.J. & Frank, M.J. (2008) A dopaminergic basis for working memory, learning and attentional shifting in Parkinsonism. Neuropsychologia, 46, 3144-3156.
49. O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K. & Dolan, R.J. (2004) Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science, 304, 452-454.
50. O'Doherty, J.P., Hampton, A. & Kim, H. (2007) Model-based fMRI and its application to reward learning and decision making. Ann. NY Acad. Sci., 1104, 35-53.
51. O'Reilly, R.C. & Frank, M.J. (2006) Making working memory work: a computational model of learning in the prefrontal cortex and basal ganglia. Neural Comput., 18, 283-328.
52. Pessiglione, M., Seymour, B., Flandin, G., Dolan, R.J. & Frith, C.D. (2006) Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature, 442, 1042-1045.
53. Ribas-Fernandes, J.J.F., Solway, A., Diuk, C., McGuire, J.T., Barto, A.G., Niv, Y. & Botvinick, M.M. (2011) A neural signature of hierarchical reinforcement learning. Neuron, 71, 370-379.
54. Roth, R.B., Hevezi, P., Lee, J., Willhite, D., Lechner, S.M., Foster, A.C. & Zlotnik, A. (2006) Gene expression analyses reveal molecular relationships among 20 regions of the human CNS. Neurogenetics, 7, 67-80.
55. Samejima, K., Ueda, Y., Doya, K. & Kimura, M. (2005) Representation of action-specific reward values in the striatum. Science, 310, 1337-1340.
56. Schönberg, T., Daw, N.D., Joel, D. & O'Doherty, J.P. (2007) Reinforcement learning signals in the human striatum distinguish learners from nonlearners during reward-based decision making. J. Neurosci., 27, 12860-12867.
57. Schultz, W. (1997) A neural substrate of prediction and reward. Science, 275, 1593-1599.
58. Sesack, S.R., Hawrylak, V.A., Matus, C., Guido, M.A. & Levey, A.I. (1998) Dopamine axon varicosities in the prelimbic division of the rat prefrontal cortex exhibit sparse immunoreactivity for the dopamine transporter. J. Neurosci., 18, 2697-2708.
59. Seymour, B., O'Doherty, J., Dayan, P., Koltzenburg, M., Jones, A., Dolan, R., Friston, K. & Frackowiak, R. (2004) Temporal difference models describe higher-order learning in humans. Nature, 429, 664-667.
60. Slifstein, M., Kolachana, B., Simpson, E.H., Tabares, P., Cheng, B., Duvall, M., Frankle, W.G., Weinberger, D.R., Laruelle, M. & Abi-Dargham, A. (2008) COMT genotype predicts cortical-limbic D1 receptor availability measured with [11C]NNC112 and PET. Mol. Psychiatry, 13, 821-827.
61. Sloman, S.A. & Hagmayer, Y. (2006) The causal psycho-logic of choice. Trends Cogn. Sci., 10, 407-412.
62. Stephan, K.E., Penny, W.D., Daunizeau, J., Moran, R.J. & Friston, K.J. (2009) Bayesian model selection for group studies. NeuroImage, 46, 1004-1017.
63. Sutton, R. & Barto, A. (1998) Reinforcement Learning, Vol. 9. MIT Press, Cambridge, Massachussetts.
64. Tan, H.-Y., Chen, Q., Goldberg, T.E., Mattay, V.S., Meyer-Lindenberg, A., Weinberger, D.R. & Callicott, J.H. (2007) Catechol-O-methyltransferase Val158Met modulation of prefrontal-parietal-striatal brain systems during arithmetic and temporal transformations in working memory. J. Neurosci., 27, 13393-13401.
65. Tanaka, S.C., Doya, K., Okada, G., Ueda, K., Okamoto, Y. & Yamawaki, S. (2004) Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nat. Neurosci., 7, 887-893.
66. Tanaka, S.C., Samejima, K., Okada, G., Ueda, K., Okamoto, Y., Yamawaki, S. & Doya, K. (2006) Brain mechanism of reward prediction under predictable and unpredictable environmental dynamics. Neural Netw., 19, 1233-1241.
67. Todd, M.T., Niv, Y. & Cohen, J.D. (2008) Learning to use working memory in partially observable environments through dopaminergic reinforcement. in Koller, D. (Ed.), Twenty-First Annual Conference on Neural Information Processing Systems (NIPS) 2008. Vancouver: Canada.
68. Tunbridge, E.M., Bannerman, D.M., Sharp, T. & Harrison, P.J. (2004) Catechol-o-methyltransferase inhibition improves set-shifting performance and elevates stimulated dopamine release in the rat pre-frontal cortex. J. Neurosci., 24, 5331-5335.
69. Zhang, W. & Luck, S.J. (2008) Discrete fixed-resolution representations in visual working memory. Nature, 453, 233-235.
70. Zhang, W. & Luck, S.J. (2011) The number and quality of representations in working memory. Psychol. Sci., 22, 1434-1441. doi: .