Goals and habits in the brain

Neuron

Volumn 80, Issue 2, 2013, Pages 312-325

  Dolan, Ray J ^a   Dayan, Peter ^b  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

^b UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ANIMAL BEHAVIOR; BRAIN FUNCTION; COGNITION; CONDITIONING; DECISION MAKING; DRUG DEPENDENCE; HABIT; HUMAN; LEARNING; MENTAL DISEASE; MOTIVATION; NEUROSCIENCE; NONHUMAN; OBSESSIVE COMPULSIVE DISORDER; PRIORITY JOURNAL; PROSPECTIVE MEMORY; REFLEX; REINFORCEMENT; RETROSPECTIVE MEMORY; REVIEW; RODENT; STIMULUS RESPONSE; TASK PERFORMANCE;

ANIMALS; BRAIN; CONDITIONING, OPERANT; GOALS; HABITS; HUMANS; MODELS, PSYCHOLOGICAL;

EID: 84885802926     PISSN: 08966273     EISSN: 10974199     Source Type: Journal    
DOI: 10.1016/j.neuron.2013.09.007     Document Type: Review

References (164)

1. Adams C.D., Dickinson A. Instrumental responding following reinforcer devaluation. Q.J. Exp. Psychol. B 1981, 33:109-121.
2. Balleine B.W. Neural bases of food-seeking: affect, arousal and reward in corticostriatolimbic circuits. Physiol. Behav. 2005, 86:717-730.
3. Balleine B.W., Dickinson A. Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology 1998, 37:407-419.
4. Balleine B.W., O'Doherty J.P. Human and rodent homologies in action control: corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacology 2010, 35:48-69.
5. Barto A.G., Sutton R.S., Anderson C.W. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Trans. Syst. Man Cybern. 1983, 13:835-846.
6. Belin D., Jonkman S., Dickinson A., Robbins T.W., Everitt B.J. Parallel and interactive learning processes within the basal ganglia: relevance for the understanding of addiction. Behav. Brain Res. 2009, 199:89-102.
7. Bellman R. Dynamic Programming 1957, Princeton University Press, Princeton.
8. Berns G.S., McClure S.M., Pagnoni G., Montague P.R. Predictability modulates human brain response to reward. J.Neurosci. 2001, 21:2793-2798.
9. Berridge K.C. Reward learning: reinforcement incentives and expectations. Psychology of Learning and Motivation 2001, 223-278. Academic Press, San Diego. D.L. Medin (Ed.).
10. Biele G., Rieskamp J., Krugel L.K., Heekeren H.R. The neural basis of following advice. PLoS Biol. 2011, 9:e1001089.
11. Blodgett H.C. The effect of the introduction of reward upon the maze performance of rats. University of California Publications in Psychology 1929, 4:113-134.
12. Botvinick M.M. Hierarchical reinforcement learning and decision making. Curr. Opin. Neurobiol. 2012, 22:956-962.
13. Bradfield L.A., Balleine B.W. Hierarchical and binary associations compete for behavioral control during instrumental biconditional discrimination. J.Exp. Psychol. Anim. Behav. Process. 2013, 39:2-13.
14. Burgess N. Spatial cognition and the brain. Ann. N Y Acad. Sci. 2008, 1124:77-97.
15. Camille N., Tsuchida A., Fellows L.K. Double dissociation of stimulus-value and action-value learning in humans with orbitofrontal or anterior cingulate cortex damage. J.Neurosci. 2011, 31:15048-15052.
16. Cheng K. A purely geometric module in the rat's spatial representation. Cognition 1986, 23:149-178.
17. Chersi F., Pezzulo G. Using hippocampal-striatal loops for spatial navigation and goal-directed decision-making. Cogn. Process. 2012, 13(Suppl 1):S125-S129.
18. Collins A.G., Frank M.J. How much of reinforcement learning is working memory, not reinforcement learning? A behavioral, computational, and neurogenetic analysis. Eur. J. Neurosci. 2012, 35:1024-1035.
19. Cools R. Dopaminergic control of the striatum for high-level cognition. Curr. Opin. Neurobiol. 2011, 21:402-407.
20. Cools R., D'Esposito M. Inverted-U-shaped dopamine actions on human working memory and cognitive control. Biol. Psychiatry 2011, 69:e113-e125.
21. Corbit L.H., Balleine B.W. The role of prelimbic cortex in instrumental conditioning. Behav. Brain Res. 2003, 146:145-157.
22. Corbit L.H., Balleine B.W. Double dissociation of basolateral and central amygdala lesions on the general and outcome-specific forms of pavlovian-instrumental transfer. J.Neurosci. 2005, 25:962-970.
23. Coutureau E., Killcross S. Inactivation of the infralimbic prefrontal cortex reinstates goal-directed responding in overtrained rats. Behav. Brain Res. 2003, 146:167-174.
24. Daw N.D., Niv Y., Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 2005, 8:1704-1711.
25. Daw N.D., Niv Y., Dayan P. Actions, policies, values and the basal ganglia. Recent Breakthroughs in Basal Ganglia Research 2006, 91-96. Nova Science Publishers, New York. E. Bezard (Ed.).
26. Daw N.D., O'Doherty J.P., Dayan P., Seymour B., Dolan R.J. Cortical substrates for exploratory decisions in humans. Nature 2006, 441:876-879.
27. Daw N.D., Gershman S.J., Seymour B., Dayan P., Dolan R.J. Model-based influences on humans' choices and striatal prediction errors. Neuron 2011, 69:1204-1215.
28. Dayan P. Improving generalization for temporal difference learning: the successor representation. Neural Comput. 1993, 5:613-624.
29. Dayan P. Goal-directed control and its antipodes. Neural Netw. 2009, 22:213-219.
30. Dayan P. How to set the switches on this thing. Curr. Opin. Neurobiol. 2012, 22:1068-1074.
31. Dayan P., Daw N.D. Decision theory, reinforcement learning, and the brain. Cogn. Affect. Behav. Neurosci. 2008, 8:429-453.
32. Dayan P., Niv Y., Seymour B., Daw N.D. The misbehavior of value and the discipline of the will. Neural Netw. 2006, 19:1153-1160.
33. de Wit S., Barker R.A., Dickinson A.D., Cools R. Habitual versus goal-directed action control in Parkinson disease. J.Cogn. Neurosci. 2011, 23:1218-1229.
34. de Wit S., Standing H.R., Devito E.E., Robinson O.J., Ridderinkhof K.R., Robbins T.W., Sahakian B.J. Reliance on habits at the expense of goal-directed control following dopamine precursor depletion. Psychopharmacology (Berl.) 2012, 219:621-631.
35. de Wit S., Watson P., Harsay H.A., Cohen M.X., van de Vijver I., Ridderinkhof K.R. Corticostriatal connectivity underlies individual differences in the balance between habitual and goal-directed action control. J.Neurosci. 2012, 32:12066-12075.
36. Dickinson A. Actions and habits: the development of behavioural and autonomy. Philos. Trans. R. Soc. Lond. B Biol. Sci. 1985, 308:67-78.
37. Dickinson A., Balleine B.W. Motivational control of goal-directed action. Anim. Learn. Behav. 1994, 22:1-18.
38. Dickinson A., Balleine B. The role of learning in the operation of motivational systems. Learning, Motivation, and Emotion 2002, 497-533. John Wiley & Sons, New York. H. Pashler, C.R. Gallistel (Eds.).
39. Dickinson A., Charnock D.J. Contingency effects with maintained instrumental reinforcement. Q.J. Exp. Psychol. B 1985, 37:397-416.
40. Dickinson A., Dawson G.R. Pavlovian processes in the motivational control of instrumental performance. Q.J. Exp. Psychol. B 1987, 39:201-213.
41. Dickinson A., Dawson G.R. The role of the instrumental contingency in the motivational control of performance. Q.J. Exp. Psychol. B 1987, 39:77-93.
42. Dickinson A., Nicholas D.J., Adams C.D. The effect of the instrumental training contingency on susceptibility to reinforcer devaluation. Q.J. Exp. Psychol. B 1983, 35:35-51.
43. Doll B.B., Jacobs W.J., Sanfey A.G., Frank M.J. Instructional control of reinforcement learning: a behavioral and neurocomputational investigation. Brain Res. 2009, 1299:74-94.
44. Doll B.B., Hutchison K.E., Frank M.J. Dopaminergic genes predict individual differences in susceptibility to confirmation bias. J.Neurosci. 2011, 31:6188-6198.
45. Doll B.B., Simon D.A., Daw N.D. The ubiquity of model-based reinforcement learning. Curr. Opin. Neurobiol. 2012, 22:1075-1081.
46. Doya K. What are the computations of the cerebellum, the basal ganglia and the cerebral cortex?. Neural Netw. 1999, 12:961-974.
47. Doya K., Samejima K., Katagiri K., Kawato M. Multiple model-based reinforcement learning. Neural Comput. 2002, 14:1347-1369.
48. Dragoi G., Buzsáki G. Temporal encoding of place sequences by hippocampal cell assemblies. Neuron 2006, 50:145-157.
49. Everitt B.J., Belin D., Economidou D., Pelloux Y., Dalley J.W., Robbins T.W. Review. Neural mechanisms underlying the vulnerability to develop compulsive drug-seeking habits and addiction. Philos. Trans. R. Soc. Lond. B Biol. Sci. 2008, 363:3125-3135.
50. Faure A., Haberland U., Condé F., El Massioui N. Lesion to the nigrostriatal dopamine system disrupts stimulus-response habit formation. J.Neurosci. 2005, 25:2771-2780.
51. Fellows L.K. The neurology of value. Neurobiology of Sensation and Reward 2011, CRC Press, Boca Raton. J.A. Gottfried (Ed.).
52. Fermin A., Yoshida T., Ito M., Yoshimoto J., Doya K. Evidence for model-based action planning in a sequential finger movement task. J.Mot. Behav. 2010, 42:371-379.
53. FitzGerald T.H.B., Friston K.J., Dolan R.J. Action-specific valuesignals in reward-related regions of the human brain. J.Neurosci. 2012, 32. 16417-23a.
54. Foster D.J., Wilson M.A. Reverse replay of behavioural sequences in hippocampal place cells during the awake state. Nature 2006, 440:680-683.
55. Foster D.J., Wilson M.A. Hippocampal theta sequences. Hippocampus 2007, 17:1093-1099.
56. Frank M.J. Dynamic dopamine modulation in the basal ganglia: a neurocomputational account of cognitive deficits in medicated and nonmedicated Parkinsonism. J.Cogn. Neurosci. 2005, 17:51-72.
57. Frank M.J., Badre D. Mechanisms of hierarchical reinforcement learning in corticostriatal circuits 1: computational analysis. Cereb. Cortex 2012, 22:509-526.
58. Frank M.J., Loughry B., O'Reilly R.C. Interactions between frontal cortex and basal ganglia in working memory: a computational model. Cogn. Affect. Behav. Neurosci. 2001, 1:137-160.
59. Frank M.J., Seeberger L.C., O'reilly R.C. By carrot or by stick: cognitive reinforcement learning in parkinsonism. Science 2004, 306:1940-1943.
60. Gallistel C.R. The Organization of Learning 1990, MIT Press, Cambridge.
61. Gershman S.J., Markman A.B., Otto A.R. Retrospective revaluation in sequential decision making: a tale of two systems. J.Exp. Psychol. Gen. 2012, Published online December 10, 2012. 10.1037/a0030844.
62. Gillan C.M., Papmeyer M., Morein-Zamir S., Sahakian B.J., Fineberg N.A., Robbins T.W., de Wit S. Disruption in the balance between goal-directed behavior and habit learning in obsessive-compulsive disorder. Am. J. Psychiatry 2011, 168:718-726.
63. Gläscher J., Daw N., Dayan P., O'Doherty J.P. States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 2010, 66:585-595.
64. Gottfried J.A., Dolan R.J. Human orbitofrontal cortex mediates extinction learning while accessing conditioned representations of value. Nat. Neurosci. 2004, 7:1144-1152.
65. Graybiel A.M. Habits, rituals, and the evaluative brain. Annu. Rev. Neurosci. 2008, 31:359-387.
66. Groenewegen H.J., Berendse H.W., Wolters J.G., Lohman A.H. The anatomical relationship of the prefrontal cortex with the striatopallidal system, the thalamus and the amygdala: evidence for a parallel organization. Prog. Brain Res. 1990, 85:95-116. discussion 116-118.
67. Haber S.N., Knutson B. The reward circuit: linking primate anatomy and human imaging. Neuropsychopharmacology 2010, 35:4-26.
68. Haber S.N., Fudge J.L., McFarland N.R. Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. J.Neurosci. 2000, 20:2369-2382.
69. Hampton A.N., Bossaerts P., O'Doherty J.P. The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. J.Neurosci. 2006, 26:8360-8367.
70. Haruno M., Kawato M. Heterarchical reinforcement-learning model for integration of multiple cortico-striatal loops: fMRI examination in stimulus-action-reward association learning. Neural Netw. 2006, 19:1242-1254.
71. Haruno M., Kuroda T., Doya K., Toyama K., Kimura M., Samejima K., Imamizu H., Kawato M. A neural correlate of reward-based behavioral learning in caudate nucleus: a functional magnetic resonance imaging study of a stochastic decision task. J.Neurosci. 2004, 24:1660-1665.
72. Hassabis D., Kumaran D., Vann S.D., Maguire E.A. Patients with hippocampal amnesia cannot imagine new experiences. Proc. Natl. Acad. Sci. USA 2007, 104:1726-1731.
73. Hikosaka O., Nakahara H., Rand M.K., Sakai K., Lu X., Nakamura K., Miyachi S., Doya K. Parallel neural networks for learning sequential procedures. Trends Neurosci. 1999, 22:464-471.
74. Hitchcott P.K., Quinn J.J., Taylor J.R. Bidirectional modulation of goal-directed actions by prefrontal cortical dopamine. Cereb. Cortex 2007, 17:2820-2827.
75. Holland P.C. Relations between Pavlovian-instrumental transfer and reinforcer devaluation. J.Exp. Psychol. Anim. Behav. Process. 2004, 30:104-117.
76. Hu D., Amsel A. A simple test of the vicarious trial-and-error hypothesis of hippocampal function. Proc. Natl. Acad. Sci. USA 1995, 92:5506-5509.
77. Hurley K.M., Herbert H., Moga M.M., Saper C.B. Efferent projections of the infralimbic cortex of the rat. J.Comp. Neurol. 1991, 308:249-276.
78. Huys Q.J., Eshel N., O'Nions E., Sheridan L., Dayan P., Roiser J.P. Bonsai trees in your head: how the pavlovian system sculpts goal-directed choices by pruning decision trees. PLoS Comput. Biol. 2012, 8:e1002410.
79. Isoda M., Hikosaka O. Cortico-basal ganglia mechanisms for overcoming innate, habitual and motivational behaviors. Eur. J. Neurosci. 2011, 33:2058-2069.
80. James W. The Principles of Psychology 1890, Holt, New York.
81. Joel D., Weiner I. The connections of the dopaminergic system with the striatum in rats and primates: an analysis with respect to the functional and compartmental organization of the striatum. Neuroscience 2000, 96:451-474.
82. Johnson A., Redish A.D. Hippocampal replay contributes to within session learning in a temporal difference reinforcement learning model. Neural Netw. 2005, 18:1163-1171.
83. Johnson A., Redish A.D. Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. J.Neurosci. 2007, 27:12176-12189.
84. Kahneman D. Thinking, Fast and Slow 2011, Farrar, Straus and Giroux, New York. First Edition.
85. Keramati M., Dezfouli A., Piray P. Speed/accuracy trade-off between the habitual and the goal-directed processes. PLoS Comput. Biol. 2011, 7:e1002055.
86. Killcross S., Coutureau E. Coordination of actions and habits in the medial prefrontal cortex of rats. Cereb. Cortex 2003, 13:400-408.
87. Kimble G.A., Perlmuter L.C. The problem of volition. Psychol. Rev. 1970, 77:361-384.
88. Koechlin E., Hyafil A. Anterior prefrontal function and the limits of human decision-making. Science 2007, 318:594-598.
89. Koechlin E., Ody C., Kouneiher F. The architecture of cognitive control in the human prefrontal cortex. Science 2003, 302:1181-1185.
90. Krieckhaus E.E., Wolf G. Acquisition of sodium by rats: interaction of innate mechanisms and latent learning. J.Comp. Physiol. Psychol. 1968, 65:197-201.
91. Kruse J.M., Overmier J.B., Konz W.A., Rokke E. Pavlovian conditioned stimulus effects upon instrumental choice behavior are reinforcer specific. Learn. Mem. 1983, 14:165-181.
92. Lee A.K., Wilson M.A. Memory of sequential experience in the hippocampus during slow wave sleep. Neuron 2002, 36:1183-1194.
93. Lengyel M., Dayan P. Hippocampal contributions to control: the third way. Adv. Neural Inf. Process. Syst. 2008, 20:889-896.
94. Li J., Daw N.D. Signals in human striatum are appropriate for policy update rather than value prediction. J.Neurosci. 2011, 31:5504-5511.
95. Liljeholm M., Tricomi E., O'Doherty J.P., Balleine B.W. Neural correlates of instrumental contingency learning: differential effects of action-reward conjunction and disjunction. J.Neurosci. 2011, 31:2474-2480.
96. Louie K., Wilson M.A. Temporally structured replay of awake hippocampal ensemble activity during rapid eye movement sleep. Neuron 2001, 29:145-156.
97. Mackintosh N.J. Conditioning and Associative Learning 1983, Oxford University Press, Oxford.
98. Maia T.V. Reinforcement learning, conditioning, and the brain: Successes and challenges. Cogn. Affect. Behav. Neurosci. 2009, 9:343-364.
99. Maia T.V., Cooney R.E., Peterson B.S. The neural bases of obsessive-compulsive disorder in children and adults. Dev. Psychopathol. 2008, 20:1251-1283.
100. McClelland J.L., McNaughton B.L., O'Reilly R.C. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychol. Rev. 1995, 102:419-457.
101. McDannald M.A., Lucantonio F., Burke K.A., Niv Y., Schoenbaum G. Ventral striatum and orbitofrontal cortex are both required for model-based, but not model-free, reinforcement learning. J.Neurosci. 2011, 31:2700-2705.
102. McGeorge A.J., Faull R.L. The organization of the projection from the cerebral cortex to the striatum in the rat. Neuroscience 1989, 29:503-537.
103. Montague P.R., Dayan P., Sejnowski T.J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J.Neurosci. 1996, 16:1936-1947.
104. Moran R.J., Symmonds M., Stephan K.E., Friston K.J., Dolan R.J. An in vivo assay of synaptic function mediating human cognition. Curr. Biol. 2011, 21:1320-1325.
105. Muenzinger K.F. Vicarious trial and error at a point of choice. I. A general survey of its relation to learning efficiency. J.Genet. Psychol. 1938, 53:75-86.
106. Niv Y. Reinforcement learning in the brain. J.Math. Psychol. 2009, 53:139-154.
107. Niv Y., Daw N.D., Joel D., Dayan P. Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology (Berl.) 2007, 191:507-520.
108. O'Doherty J.P. Reward representations and reward-related learning in the human brain: insights from neuroimaging. Curr. Opin. Neurobiol. 2004, 14:769-776.
109. O'Doherty J.P. Contributions of the ventromedial prefrontal cortex to goal-directed action selection. Ann. N Y Acad. Sci. 2011, 1239:118-129.
110. O'Doherty J.P., Dayan P., Friston K., Critchley H., Dolan R.J. Temporal difference models and reward-related learning in the human brain. Neuron 2003, 38:329-337.
111. O'Keefe J., Nadel L. The Hippocampus as a Cognitive Map 1978, Oxford University Press, Oxford.
112. O'Reilly R.C., Frank M.J. Making working memory work: a computational model of learning in the prefrontal cortex and basal ganglia. Neural Comput. 2006, 18:283-328.
113. Ostlund S.B., Balleine B.W. Lesions of medial prefrontal cortex disrupt the acquisition but not the expression of goal-directed learning. J.Neurosci. 2005, 25:7763-7770.
114. Ostlund S.B., Maidment N.T. Dopamine receptor blockade attenuates the general incentive motivational effects of noncontingently delivered rewards and reward-paired cues without affecting their ability to bias action selection. Neuropsychopharmacology 2012, 37:508-519.
115. Otto A.R., Gershman S.J., Markman A.B., Daw N.D. The curse of planning: dissecting multiple reinforcement-learning systems by taxing the central executive. Psychol. Sci. 2013, 24:751-761.
116. Packard M.G., McGaugh J.L. Inactivation of hippocampus or caudate nucleus with lidocaine differentially affects expression of place and response learning. Neurobiol. Learn. Mem. 1996, 65:65-72.
117. Padoa-Schioppa C., Assad J.A. Neurons in the orbitofrontal cortex encode economic value. Nature 2006, 441:223-226.
118. Pezzulo G., Rigoli F., Chersi F. The mixed instrumental controller: using value of information to combine habitual choice and mental simulation. Front. Psychol. 2013, 4:92.
119. Pfeiffer B.E., Foster D.J. Hippocampal place-cell sequences depict future paths to remembered goals. Nature 2013, 497:74-79.
120. Prévost C., McNamee D., Jessup R.K., Bossaerts P., O'Doherty J.P. Evidence for model-based computations in the human amygdala during Pavlovian conditioning. PLoS Comput. Biol. 2013, 9:e1002918.
121. Rangel A., Camerer C., Montague P.R. A framework for studying the neurobiology of value-based decision making. Nat. Rev. Neurosci. 2008, 9:545-556.
122. Reynolds J.N., Hyland B.I., Wickens J.R. A cellular mechanism of reward-related learning. Nature 2001, 413:67-70.
123. Robinson M.J., Berridge K.C. Instant transformation of learned repulsion into motivational "wanting". Curr. Biol. 2013, 23:282-289.
124. Ruediger S., Spirig D., Donato F., Caroni P. Goal-oriented searching mediated by ventral hippocampus early in trial-and-error learning. Nat. Neurosci. 2012, 15:1563-1571.
125. Rummery G.A., Niranjan M. On-line Q-learning Using Connectionist Systems, Technical Report CUED/F-INFENG/TR 166 1994, University of Cambridge, Department of Engineering, Cambridge.
126. Schacter D.L., Addis D.R., Buckner R.L. Remembering the past to imagine the future: the prospective brain. Nat. Rev. Neurosci. 2007, 8:657-661.
127. Schacter D.L., Addis D.R., Hassabis D., Martin V.C., Spreng R.N., Szpunar K.K. The future of memory: remembering, imagining, and the brain. Neuron 2012, 76:677-694.
128. Schoenbaum G., Roesch M. Orbitofrontal cortex, associative learning, and expectancies. Neuron 2005, 47:633-636.
129. Schultz W., Dayan P., Montague P.R. A neural substrate of prediction and reward. Science 1997, 275:1593-1599.
130. Silver, D., Sutton, R.S., and Müller, M. (2008). Sample-based learning and search with permanent and transient memories. In Proceedings of the 25th International Vonference on Machine Learning, 968-975.
131. Simon D.A., Daw N.D. Neural correlates of forward planning in a spatial decision task in humans. J.Neurosci. 2011, 31:5526-5539.
132. Smith K.S., Graybiel A.M. Using optogenetics to study habits. Brain Res. 2013, 1511:102-114.
133. Smittenaar P., Fitzgerald T.H.B., Romei V., Wright N.D., Dolan R.J. Disruption of dorsolateral prefrontal cortex decreases model-based in favor of model-free control in humans. Neuron 2013, 80. Published online October 24, 2013. 10.1016/j.neuron.2013.08.009.
134. Solway A., Botvinick M.M. Goal-directed decision making as probabilistic inference: a computational framework and potential neural correlates. Psychol. Rev. 2012, 119:120-154.
135. Stanovich K., West R. Individual differences in reasoning: implications for the rationality debate?. Heuristics and Biases: The Psychology of Intuitive Judgment 2002, 421-440. Cambridge University Press, Cambridge. T. Gilovich, D. Griffin, D. Kahneman (Eds.).
136. Sutton R.S. Learning to predict by the methods of temporal differences. Mach. Learn. 1988, 3:9-44.
137. Sutton, R.S. (1990). Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Proceedings of the Seventh International Conference on Machine Learning, 216-224.
138. Sutton R.S. Dyna, an Integrated Architecture for Learning, Planning, and Reacting. SIGART Bulletin 1991, 2:160-163.
139. Sutton R.S., Barto A.G. Reinforcement Learning: An Introduction 1998, MIT Press, Cambridge.
140. Tanaka S.C., Balleine B.W., O'Doherty J.P. Calculating consequences: brain systems that encode the causal effects of actions. J.Neurosci. 2008, 28:6750-6755.
141. Thistlethwaite D. A critical review of latent learning and related experiments. Psychol. Bull. 1951, 48:97-129.
142. Thorn C.A., Atallah H., Howe M., Graybiel A.M. Differential dynamics of activity changes in dorsolateral and dorsomedial striatal loops during learning. Neuron 2010, 66:781-795.
143. Thorndike E. Animal Intelligence: Experimental Studies 1911, MacMillan, New York.
144. Thorpe S.J., Rolls E.T., Maddison S. The orbitofrontal cortex: neuronal activity in the behaving monkey. Exp. Brain Res. 1983, 49:93-115.
145. Tolman E.C. The determiners of behavior at a choice point. Psychol. Rev. 1938, 45:1-41.
146. Tolman E.C. Cognitive maps in rats and men. Psychol. Rev. 1948, 55:189-208.
147. Tolman E.C., Honzik C.H. Introduction and removal of reward, and maze performance in rats. University of California Publications in Psychology 1930, 4:257-275.
148. Tricomi E.M., Balleine B.W., O'Doherty J.P. A specific role for posterior dorsolateral striatum in human habit learning. Eur. J. Neurosci. 2009, 29:2225-2232.
149. Valentin V.V., Dickinson A., O'Doherty J.P. Determining the neural substrates of goal-directed learning in the human brain. J.Neurosci. 2007, 27:4019-4026.
150. van der Meer M.A., Redish A.D. Covert expectation-of-reward in rat ventral striatum at decision points. Front Integr Neurosci 2009, 3:1.
151. Vertes R.P. Differential projections of the infralimbic and prelimbic cortex in the rat. Synapse 2004, 51:32-58.
152. Wassum K.M., Cely I.C., Maidment N.T., Balleine B.W. Disruption of endogenous opioid activity during instrumental learning enhances habit acquisition. Neuroscience 2009, 163:770-780.
153. Watkins C.J.C.H. Learning from Delayed Rewards 1989, University of Cambridge, Cambridge.
154. White N.M. Mnemonic functions of the basal ganglia. Curr. Opin. Neurobiol. 1997, 7:164-169.
155. Williams G.V., Goldman-Rakic P.S. Modulation of memory fields by dopamine D1 receptors in prefrontal cortex. Nature 1995, 376:572-575.
156. Williams D.R., Williams H. Auto-maintenance in the pigeon: sustained pecking despite contingent non-reinforcement. J.Exp. Anal. Behav. 1969, 12:511-520.
157. Wirsig C.R., Grill H.J. Contribution of the rat's neocortex to ingestive control: I. Latent learning for the taste of sodium chloride. J.Comp. Physiol. Psychol. 1982, 96:615-627.
158. Wunderlich K., Dayan P., Dolan R.J. Mapping value based planning and extensively trained choice in the human brain. Nat. Neurosci. 2012, 15:786-791.
159. Wunderlich K., Smittenaar P., Dolan R.J. Dopamine enhances model-based over model-free choice behavior. Neuron 2012, 75:418-424.
160. Yaryura-Tobias J.A., Wolpert A., Dana L., Merlis S. Action of L-Dopa in drug induced extrapyramidalism. Dis. Nerv. Syst. 1970, 31:60-63.
161. Yin H.H., Knowlton B.J. The role of the basal ganglia in habit formation. Nat. Rev. Neurosci. 2006, 7:464-476.
162. Yin H.H., Knowlton B.J., Balleine B.W. Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. Eur. J. Neurosci. 2004, 19:181-189.
163. Yin H.H., Ostlund S.B., Knowlton B.J., Balleine B.W. The role of the dorsomedial striatum in instrumental conditioning. Eur. J. Neurosci. 2005, 22:513-523.
164. Yin H.H., Mulcare S.P., Hilário M.R., Clouse E., Holloway T., Davis M.I., Hansson A.C., Lovinger D.M., Costa R.M. Dynamic reorganization of striatal circuits during the acquisition and consolidation of a skill. Nat. Neurosci. 2009, 12:333-341.