Reinforcement learning in multidimensional environments relies on attention mechanisms

Journal of Neuroscience

Volumn 35, Issue 21, 2015, Pages 8145-8157

  Niv, Yael ^a   Daniel, Reka ^a   Geana, Andra ^a   Gershman, Samuel J ^b   Leong, Yuan Chang ^c   Radulescu, Angela ^a   Wilson, Robert C ^d  

^a PRINCETON UNIVERSITY   (United States)

^b MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^c STANFORD UNIVERSITY   (United States)

^d University of Arizona   (United States)

Author keywords

Attention; fMRI; Frontoparietal network; Model comparison; Reinforcement learning; Representation learning

Indexed keywords

ADULT; ARTICLE; ATTENTION; BAYESIAN LEARNING; CLINICAL ARTICLE; DORSOLATERAL PREFRONTAL CORTEX; FEMALE; FRONTOPARIETAL CORTEX; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HUMAN; INTRAPARIETAL SULCUS; MALE; MATHEMATICAL MODEL; PRECUNEUS; PRIORITY JOURNAL; REINFORCEMENT LEARNING; REPRESENTATION LEARNING; YOUNG ADULT; ADOLESCENT; CONTROLLED STUDY; DECISION MAKING; ENVIRONMENT; LEARNING; PHOTOSTIMULATION; PHYSIOLOGY; PROCEDURES; RANDOMIZED CONTROLLED TRIAL; REINFORCEMENT;

ADOLESCENT; ADULT; ATTENTION; CHOICE BEHAVIOR; ENVIRONMENT; FEMALE; HUMANS; LEARNING; MALE; PHOTIC STIMULATION; REINFORCEMENT (PSYCHOLOGY); YOUNG ADULT;

EID: 84930260511     PISSN: 02706474     EISSN: 15292401     Source Type: Journal    
DOI: 10.1523/JNEUROSCI.2978-14.2015     Document Type: Article

References (83)

1. Akaishi R, Umeda K, Nagase A, Sakai K (2014) Autonomous mechanism of internal choice estimate underlies decision inertia. Neuron 81:195–206.
2. Ashby FG, Maddox WT (2005) Human category learning. Annu Rev Psychol 56:149–178.
3. Baldauf D, Desimone R (2014) Neural mechanisms of object-based attention. Science 344:424–427.
4. Bar-Gad I, Havazelet-Heimer G, Goldberg JA, Ruppin E, Bergman H (2000) Reinforcement-driven dimensionality reduction-a model for information processing in the basal ganglia. J Basic Clin Physiol Pharmacol 11: 305–320.
5. Barto AG (1995) Adaptive critic and the basal ganglia. In: Models of information processing in the basal ganglia (Houk JC, Davis JL, Beiser DG, eds), pp 215–232. Cambridge, MA: MIT.
6. Birrell JM, Brown VJ (2000) Medial frontal cortex mediates perceptual attentional set shifting in the rat. J Neurosci 20:4320–4324.
7. Bishara AJ, Kruschke JK, Stout JC, Bechara A, McCabe DP, Busemeyer JR (2010) Sequential learning models for the wisconsin card sort task: assessing processes in substance dependent individuals. J Math Psychol 54:5–13.
8. Bower G, Trabasso T (1963) Reversals prior to solution in concept identification. J Exp Psychol 66:409–418.
9. Buchsbaum BR, Greer S, Chang WL, Berman KF (2005) Meta-analysis of neuroimaging studies of the Wisconsin card-sorting task and component processes. Hum Brain Mapp 25:35–45.
10. Cañas F, Jones M (2010) Attention and reinforcement learning: constructing representations from indirect feedback. Paper presented at CogSci 2010: The Annual Meeting of the Cognitive Science Society, Portland, OR, August.
11. Chica AB, Bartolomeo P, Valero-Cabré A (2011) Dorsal and ventral parietal contributions to spatial orienting in the human brain. J Neurosci 31: 8143–8149.
12. Chudasama Y, Robbins TW (2004) Dopaminergic modulation of visual attention and working memory in the rodent prefrontal cortex. Neuropsychopharmacology 29:1628–1636.
13. Cohen Y, Schneidman E (2013) High-order feature-based mixture models of classification learning predict individual learning curves and enable personalized teaching. Proc Natl Acad Sci U S A 110:684–689.
14. Corbetta M, Shulman GL (2002) Control of goal-directed and stimulusdriven attention in the brain. Nat Rev Neurosci 3:201–215.
15. Corbetta M, Akbudak E, Conturo TE, Snyder AZ, Ollinger JM, Drury HA, Linenweber MR, Petersen SE, Raichle ME, Van Essen DC, Shulman GL (1998) A common network of functional areas for attention and eye movements. Neuron 21:761–773.
16. Crofts HS, Dalley JW, Collins P, Van Denderen JC, Everitt BJ, Robbins TW, Roberts AC (2001) Differential effects of 6-OHDA lesions of the frontal cortex and caudate nucleus on the ability to acquire an attentional set. Cereb Cortex 11:1015–1026.
17. Culham JC, Kanwisher NG (2001) Neuroimaging of cognitive functions in human parietal cortex. Curr Opin Neurobiol 11:157–163.
18. Dalley JW, Cardinal RN, Robbins TW (2004) Prefrontal executive and cognitive functions in rodents: neural and neurochemical substrates. Neurosci Biobehav Rev 28:771–784.
19. Daw ND (2011) Trial by trial data analysis using computational models. In: Decision making, affect, and learning: attention and performance (Delgado MR, Phelps EA, Robbins TW, eds), p xxiii. Oxford: Oxford UP.
20. Dias R, Robbins TW, Roberts AC (1996a) Dissociation in prefrontal cortex of affective and attentional shifts. Nature 380:69–72.
21. Dias R, Robbins TW, Roberts AC (1996b) Primate analogue of the Wisconsin Card Sorting Test: effects of excitotoxic lesions of the prefrontal cortex in the marmoset. Behav Neurosci 110:872–886.
22. Dias R, Robbins TW, Roberts AC (1997) Dissociable forms of inhibitory control within prefrontal cortex with an analog of the Wisconsin Card Sort Test: restriction to novel situations and independence from “online” processing. J Neurosci 17:9285–9297.
23. Diuk C, Barto AG, Botvinick MB, Niv Y (2010) Hierarchical reinforcement learning: an fMRI study of learning in a two-level gambling task. Soc Neurosci Abstr 36:907.14.
24. Dove A, Pollmann S, Schubert T, Wiggins CJ, von Cramon DY (2000) Prefrontal cortex activation in task switching: an event-related FMRI study. Brain Res Cogn Brain Res 9:103–109.
25. Doya K, Samejima K, Katagiri K, Kawato M (2002) Multiple model-based reinforcement learning. Neural Comput 14:1347–1369.
26. Fletcher PJ, Tenn CC, Rizos Z, Lovic V, Kapur S (2005) Sensitization to amphetamine, but not PCP, impairs attentional set shifting: reversal by a D1 receptor agonist injected into the medial prefrontal cortex. Psychopharmacology (Berl), 183:190–200.
27. Floresco SB, Block AE, Tse MT (2008) Inactivation of the medial prefrontal cortex of the rat impairs strategy set-shifting, but not reversal learning, using a novel, automated procedure. Behav Brain Res 190:85–96.
28. Frank MJ, Badre D (2012) Mechanisms of hierarchical reinforcement learning in corticostriatal circuits 1: computational analysis. Cereb Cortex 22: 509–526.
29. Gershman SJ, Niv Y (2010) Learning latent structure: carving nature at its joints. Curr Opin Neurobiol 20:251–256.
30. Gluck MA, Shohamy D, Myers C (2002) How do people solve the “weather prediction” task? Individual variability in strategies for probabilistic category learning. Learn Mem 9:408–418.
31. Grinband J, Wager TD, Lindquist M, Ferrera VP, Hirsch J (2008) Detection of time-varying signals in event-related FMRI designs. Neuroimage 43: 509–520.
32. Hare TA, O’Doherty J, Camerer CF, Schultz W, Rangel A (2008) Dissociating the role of the orbitofrontal cortex and the striatum in the computation of goal values and prediction errors. J Neurosci 28:5623–5630.
33. Holland PC, Gallagher M (1999) Amygdala circuitry in attentional and representational processes. Trends Cogn Sci 3:65–73.
34. Jones M, Cañas F (2010) Integrating reinforcement learning with models of representation learning. Paper presented at CogSci 2010: The Annual Meeting of the Cognitive Science Society, Portland, OR, August.
35. Kruschke J (2006) Learned attention. Paper presented at ICDL 2006: International Conference on Development and Learning—Dynamics of Development and Learning, Bloomington, IN, June.
36. Kruschke JK (1992) Alcove: an exemplar-based connectionist model of category learning. Psychol Rev 99:22–44.
37. Lau B, Glimcher PW (2005) Dynamic response-by-response models of matching behavior in rhesus monkeys. J Exp Anal Behav 84:555–579.
38. Lawrence V, Houghton S, Douglas G, Durkin K, Whiting K, Tannock R (2004) Executive function and adhd: a comparison of children’s performance during neuropsychological testing and real-world activities. J Atten Disord 7:137–149.
39. Levine M (1959) A model of hypothesis behavior in discrimination learning set. Psychol Rev 66:353–366.
40. Levine M (1969) Neo-noncontinuity theory. In: The psychology of learning and motivation (Bower GH, Spence JT, eds), pp 101–134. New York: Academic.
41. Liu T, Hospadaruk L, Zhu DC, Gardner JL (2011) Feature-specific attentional priority signals in human cortex. J Neurosci 31:4484–4495.
42. Love BC, Medin DL, Gureckis TM (2004) Sustain: a network model of category learning. Psychol Rev 111:309–332.
43. Lyvers MF, Maltzman I (1991) Selective effects of alcohol on Wisconsin Card Sorting Test performance. Br J Addict 86:399–407.
44. Mack ML, Preston AR, Love BC (2013) Decoding the brain’s algorithm for categorization from its neural implementation. Curr Biol 23:2023–2027.
45. McClure SM, Berns GS, Montague PR (2003) Temporal prediction errors in a passive learning task activate human striatum. Neuron 38:339–346.
46. Miller R, Wickens JR (1991) Corticostriatal cell assemblies in selective attention and in representation of predictable and controllable events. Concepts Neurosci 2:65–95.
47. Milner B (1963) Effects of different brain lesions on card sorting. Arch Neurol 9:100–110.
48. Monchi O, Petrides M, Doyon J, Postuma RB, Worsley K, Dagher A (2004) Neural bases of set-shifting deficits in Parkinson’s disease. J Neurosci 24:702–710.
49. Montague PR, Dayan P, Sejnowski TJ (1996) A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J Neurosci 16:1936–1947.
50. Niv Y (2009) Reinforcement learning in the brain. J Math Psychol 53:139– 154.
51. Niv Y, Schoenbaum G (2008) Dialogues on prediction errors. Trends Cogn Sci 12:265–272.
52. Niv Y, Edlund JA, Dayan P, O’Doherty JP (2012) Neural prediction errors reveal a risk-sensitive reinforcement-learning process in the human brain. J Neurosci 32:551–562.
53. Nosofsky RM (1986) Attention, similarity, and the identificationcategorization relationship. J Exp Psychol Gen 115:39–61.
54. Nosofsky RM, Palmeri TJ (1996) Learning to classify integral-dimension stimuli. Psychon Bull Rev 3:222–226.
55. Nosofsky RM, Palmeri TJ, McKinley SC (1994a) Rule-plus-exception model of classification learning. Psychol Rev 101:53–79.
56. Nosofsky RM, Gluck MA, Palmeri TJ, McKinley SC, Glauthier P (1994b) Comparing models of rule-based classification learning: a replication and extension of Shepard, Hovland, and Jenkins (1961). Mem Cognit 22:352– 369.
57. O’Doherty J, Dayan P, Schultz J, Deichmann R, Friston K, Dolan RJ (2004) Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304:452–454.
58. O’Doherty JP, Dayan P, Friston K, Critchley H, Dolan RJ (2003) Temporal difference learning model accounts for responses in human ventral striatum and orbitofrontal cortex during Pavlovian appetitive learning. Neuron 38:329–337.
59. Ornstein TJ, Iddon JL, Baldacchino AM, Sahakian BJ, London M, Everitt BJ, Robbins TW (2000) Profiles of cognitive dysfunction in chronic amphetamine and heroin abusers. Neuropsychopharmacology 23:113–126.
60. Owen AM, Roberts AC, Polkey CE, Sahakian BJ, Robbins TW (1991) Extradimensional versus intra-dimensional set shifting performance following frontal lobe excisions, temporal lobe excisions or amygdalohippocampectomy in man. Neuropsychologia 29:993–1006.
61. Owen AM, Roberts AC, Hodges JR, Summers BA, Polkey CE, Robbins TW (1993) Contrasting mechanisms of impaired attentional set-shifting in patients with frontal lobe damage or Parkinson’s disease. Brain 116:1159– 1175.
62. Price AL (2009) Distinguishing the contributions of implicit and explicit processes to performance of the weather prediction task. Mem Cognit 37:210–222.
63. Rehder B, Hoffman AB (2005a) Eyetracking and selective attention in category learning. Cogn Psychol 51:1–41.
64. Rehder B, Hoffman AB (2005b) Thirty-something categorization results explained: selective attention, eyetracking, and models of category learning. J Exp Psychol Learn Mem Cogn 31:811–829.
65. Rescorla RA, Wagner AR (1972) A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. In Classical conditioning II: current research and theory (Black AH, Prokasy WF, eds), pp 64–99. New York: Appleton-Century-Crofts.
66. Reynolds JN, Hyland BI, Wickens JR (2001) A cellular mechanism of reward-related learning. Nature 413:67–70.
67. Rodriguez PF, Aron AR, Poldrack RA (2006) Ventral-striatal/nucleusaccumbens sensitivity to prediction errors during classification learning. Hum Brain Mapp 27:306–313.
68. Schönberg T, Daw ND, Joel D, O’Doherty JP (2007) Reinforcement learning signals in the human striatum distinguish learners from nonlearners during reward-based decision making. J Neurosci 27:12860–12867.
69. Schultz W, Dayan P, Montague PR (1997) A neural substrate of prediction and reward. Science 275:1593–1599.
70. Seymour B, Daw ND, Roiser JP, Dayan P, Dolan R (2012) Serotonin selectively modulates reward value in human decision-making. J Neurosci 32:5833–5842.
71. Shepard RN (1987) Toward a universal law of generalization for psychological science. Science 237:1317–1323.
72. Shepard RN, Hovland CI, Jenkins HM (1961) Learning and memorization of classifications. Psychol Monogr 75:1–42.
73. Shohamy D, Myers CE, Onlaor S, Gluck MA (2004) Role of the basal ganglia in category learning: how do patients with parkinson’s disease learn? Behav Neurosci 118:676–686.
74. Smith JD, Minda JP, Washburn DA (2004) Category learning in rhesus monkeys: a study of the Shepard, Hovland, and Jenkins (1961) tasks. J Exp Psychol Gen 133:398–414.
75. Soto FA, Gershman SJ, Niv Y (2014) Explaining compound generalization in associative and causal learning through rational principles of dimensional generalization. Psychol Rev 121:526–558.
76. Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. Cambridge, MA: MIT.
77. Trabasso T, Bower G (1964) Presolution reversal and dimensional shifts in concept identification. J Exp Psychol 67:398–399.
78. van Spaendonck KP, Berger HJ, Horstink MW, Borm GF, Cools AR (1995) Card sorting performance in Parkinson’s Disease: a comparison between acquisition and shifting performance. J Clin Exp Neuropsychol 17:918– 925.
79. Wei P, Müller HJ, Pollmann S, Zhou X (2011) Neural correlates of binding features withinor cross-dimensions in visual conjunction search: an FMRI study. Neuroimage 57:235–241.
80. Wickens JR, Reynolds JN, Hyland BI (2003) Neural mechanisms of rewardrelated motor learning. Curr Opin Neurobiol 13:685–690.
81. Wickens JR, Kötter R (1995) Cellular models of reinforcement. In: Models of information processing in the basal ganglia (Houk JC, Davis JL, Beiser DG, eds), pp 187–214. Cambridge, MA: MIT.
82. Wilson RC, Niv Y (2011) Inferring relevance in a changing world. Front Hum Neurosci 5:189.
83. Zhou H, Desimone R (2011) Feature-based attention in the frontal eye field and area v4 during visual search. Neuron 70:1205–1217.