The why, what, where, when and how of goal-directed choice: Neuronal and computational principles

Philosophical Transactions of the Royal Society B: Biological Sciences

Volumn 369, Issue 1655, 2014, Pages

  Verschure, Paul F M J ^a,b   Pennartz, Cyriel M A ^c   Pezzulo, Giovanni ^d  

^a UNIVERSITAT POMPEU FABRA   (Spain)

^b INSTITUCIÓ CATALANA DE RECERCA I ESTUDIS AVANÇATS ICREA   (Spain)

^c UNIVERSITY OF AMSTERDAM   (Netherlands)

^d INSTITUTE OF COGNITIVE SCIENCES AND TECHNOLOGIES   (Italy)

Author keywords

Computational modelling; Decision making; Distributed adaptive control; Embodied cognition; Goal directed behaviour; Reward

Indexed keywords

BOOTSTRAPPING; BRAIN; COGNITION; COMPUTER SIMULATION; MEMORY; NEUROLOGY; RODENT;

ANIMAL; ANIMAL BEHAVIOR; ARTIFICIAL INTELLIGENCE; BRAIN; COGNITION; DECISION MAKING; MOTIVATION; PHYSIOLOGY; RODENT;

ANIMALS; ARTIFICIAL INTELLIGENCE; BEHAVIOR, ANIMAL; BRAIN; CHOICE BEHAVIOR; COGNITION; GOALS; RODENTIA;

EID: 84907494746     PISSN: 09628436     EISSN: 14712970     Source Type: Journal    
DOI: 10.1098/rstb.2013.0483     Document Type: Article

References (142)

1. Tolman EC. 1932 Purposive behavior in animals and man. New York, NY: Century Co.
2. Newell A. 1990 Unified theories of cognition. Cambridge, MA: Harvard University Press.
3. Verschure PFMJ, Althaus P. 2003 A real-world rational agent: unifying old and new AI. Cogn. Sci. 27, 561–590. (doi:10.1207/s15516709cog2704_1)
4. Levine J. 1983 Materialism and qualia: the explanatory gap. Pac. Phil. Q. 64, 354–361.
5. Chalmers D. 1995 Facing up to the problem of consciousness. J. Conscious. Stud. 2, 200–219.
6. Cisek P, Pastor-Bernier A. 2014 On the challenges and mechanisms of embodied decisions. Phil. Trans. R. Soc. B 369, 20130479. (doi:10.1098/rstb. 2013.0479)
7. Balleine BW, Dickinson A. 1998 Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology 37, 407–419. (doi:10.1016/ S0028-3908(98)00033-1)
8. Gläscher J, Daw N, Dayan P, O’Doherty JP. 2010 States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66, 585–595. (doi:10.1016/j.neuron.2010.04.016)
9. Pennartz C, Ito R, Verschure P, Battaglia F, Robbins T. 2011 The hippocampal–striatal axis in learning, prediction and goal-directed behavior. Trends Neurosci. 34, 548–559. (doi:10.1016/j.tins.2011.08.001)
10. Verschure PFMJ. 2012 Distributed adaptive control: a theory of the mind, brain, body nexus. Biol. Inspired Cogn. Archit. 1, 55–72. (doi:10.1016/j.bica. 2012.04.005)
11. Verschure PFMJ. 2013 Formal minds and biological brains II: from the mirage of intelligence to a science and engineering of consciousness. IEEE Expert. 28, 33–36.
12. Pavlov IP. 1927 Conditioned reflexes: an investigation of the physiological activity of the cerebral cortex. London, UK: Oxford University Press.
13. Verschure PFMJ, Voegtlin T, Douglas RJ. 2003 Environmentally mediated synergy between perception and behaviour in mobile robots. Nature 425, 620–624. (doi:10.1038/nature02024)
14. Verschure PFMJ, Krose B, Pfeifer R. 1993 Distributed adaptive control: the self-organization of structured behavior. Robot. Auton. Syst. 9, 181–196. (doi:10. 1016/0921-8890(92)90054-3)
15. Gray JA. 1987 The psychology of fear and stress. London, UK: Cambridge University Press.
16. Panksepp J, Biven L. 2012 The archaeology of mind: neuroevolutionary origins of human emotions (Norton series on interpersonal neurobiology). New York, NY: WW Norton and Company.
17. Sterling P, Eyer J. 1988 Allostasis: a new paradigm to explain arousal pathology. Psychiatry Neurosci. 30, 315–318.
18. Sanchez-Fibla M, Bernardet U, Wasserman E, Pelc T, Mintz M, Jackson JC, Lansink C, Pennartz C, Verschure PFJ. 2010 Allostatic control for robot behavior regulation: a comparative rodent-robot study. Adv. Complex Syst. 13, 377–403. (doi:10. 1142/S0219525910002621)
19. Sanchez-Fibla M, Duff A, Verschure PFMJ (eds). 2011 The acquisition of intentionally indexed and object centered affordance gradients: a biomimetic controller and mobile robotics benchmark. In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), 25 September 2011, San Francisco, USA. IEEE.
20. Lewin K. 1936 Principles of topological psychology. New York, NY: McGraw-Hill.
21. Khatib O. 1986 Real-time obstacle avoidance for manipulators and mobile robots. Int. J. Robot. Res. 5, 90–98. (doi:10.1177/027836498600500106)
22. Sanchez-Fibla MS, Bernardet U, Verschure PFMJ. (eds). 2010 Allostatic control for robot behaviour regulation: an extension to path planning. Intelligent Robots and Systems (IROS). In IEEE/RSJ Int. Conf. 18–22 October 2010, Taipei, Taiwan. IEEE.
23. Vouloutsi V, Lallée S, Verschure PMJ. 2013 Modulating behaviors using allostatic control. In Biomimetic and biohybrid systems. Lecture Notes in Computer Science 8064 (eds N Lepora, A Mura, H Krapp, PMJ Verschure, T Prescott), pp. 287–298. Berlin, Germany: Springer.
24. Verschure PFMJ, Pfeifer R (eds). 1992 Categorization, representations, and the dynamics of system-environment interaction: a case study in autonomous systems. In From animals to animats: Proc. Second Int. Conf. on Simulation of Adaptive behavior. Honolulu, Hawaii, 7–11 December 1992. Cambridge, MA: MIT Press.
25. Duff A, Verschure PFMJ. 2010 Unifying perceptual and behavioral learning with a correlative subspace learning rule. Neurocomputing 73, 1818–1830. (doi:10.1016/j.neucom.2009.11.048)
26. Miller S, Konorski J. 1928 Sur une forme particuliere des reflexes conditionnels. C.R. Seances Soc. Polonaise Biol. 49, 1155–1157.
27. Inderbitzin M, Herreros-Alonso I, Verschure PFMJ (eds). 2010 An integrated computational model of the two phase theory of classical conditioning. In Neural Networks (IJCNN), The 2010 Int. Joint Conf. 18–23 July 2010, Barcelona, Spain. IEEE.
28. Searle JR. 1980 Minds brains and programs. Behav. Brain Sci. 3, 417–457. (doi:10.1017/S0140525X 00005756)
29. Friston K. 2010 The free-energy principle: a unified brain theory? Nat. Rev. Neurosci. 11, 127–138. (doi:10.1038/nrn2787)
30. Friston K, Schwartenbeck P, FitzGerald T, Moutoussis M, Behrens T, Dolan RJ. 2014 The anatomy of choice: dopamine and decision-making. Phil. Trans. R. Soc. B 369, 20130481. (doi:10.1098/rstb. 2013.0481)
31. Daw ND, Dayan P. 2014 The algorithmic anatomy of model-based evaluation. Phil. Trans. R. Soc. B 369, 20130478. (doi:10.1098/rstb.2013.0478)
32. Herreros I, Verschure PF. 2013 Nucleo-olivary inhibition balances the interaction between the reactive and adaptive layers in motor control. Neural Netw. 47, 64–71. (doi:10.1016/j.neunet.2013.01.026)
33. Ringwald M, Verschure PFMJ (eds). 2007 The fusion of multiple sources of information in the organization of goal-oriented behavior: spatial attention versus integration. In ECMR 2007–3rd European Conf. on Mobile Robots, Freiburg, Germany, 19–21st September 2007. Berlin, Germany: Springer-Verlag.
34. Marcos E, Ringwald M, Duff A, Sánchez-Fibla M, Verschure PF. 2013 The hierarchical accumulation of knowledge in the distributed adaptive control architecture. In Computational and robotic models of the hierarchical organization of behavior (eds G Baldassarre, M Mirolli), pp. 213–234. New York, NY: Springer.
35. Mathews Z, Verschure PFMJ, Bermudezi Badia S (eds). 2010 An insect-based method for learning landmark reliability using expectation reinforcement in dynamic environments. In 2010 IEEE Int. Conf. on Robotics and Automation (ICRA), 3–7 May 2010, Anchorage, Alaska. IEEE.
36. Marcos E, Duff A, Sanchez-Fibla M, Verschure PFMJ (eds). 2010 The neuronal substrate underlying order and interval representations in sequential tasks: a biologically based robot study. In The 2010 Int. Joint Conf. on Neural Networks (IJCNN), 18–23 July 2010, Barcelona, Spain. IEEE.
37. Mathews Z et al. (eds). 2009 Insect-like mapless navigation based on head direction cells and contextual learning using chemo-visual sensors. In The 2009 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems IROS, 10 October 2009, St Louis, MI, USA. IEEE.
38. Marcos E, Sánchez-Fibla M, Verschure PMJ. 2010 The complementary roles of allostatic and contextual control systems in foraging tasks. In From animals to animats 11. Lecture Notes in Computer Science 6226 (eds S Doncieux, B Girard, A Guillot, J Hallam, J-A Meyer, J-B Mouret), pp. 370–379. Berlin, Germany: Springer.
39. Redish AD. 1999 Beyond the cognitive map: from place cells to episodic memory. Cambridge, MA: MIT Press.
40. Groenewegen HJ, Uylings H. 2000 The prefrontal cortex and the integration of sensory, limbic and autonomic information. Prog. Brain Res. 126, 3–28. (doi:10.1016/S0079-6123(00)26003-2)
41. Berthoud H-R. 2004 Mind versus metabolism in the control of food intake and energy balance. Physiol. Behav. 81, 781–793. (doi:10.1016/j.physbeh.2004. 04.034)
42. Simerly R. 1995 Anatomical substrates of hypothalamic integration. In The rat nervous system (ed. G Paxinos), pp. 353–376, 2nd edn. San Diego, CA: Academic Press.
43. Morrison SF, Nakamura K, Madden CJ. 2008 Central control of thermogenesis in mammals. Exp. Physiol. 93, 773–797. (doi:10.1113/expphysiol. 2007.041848)
44. Bourque CW. 2008 Central mechanisms of osmosensation and systemic osmoregulation. Nat. Rev. Neurosci. 9, 519–531. (doi:10.1038/nrn2400)
45. Sutcliffe JG, de Lecea L. 2002 The hypocretins: setting the arousal threshold. Nat. Rev. Neurosci. 3, 339–349. (doi:10.1038/nrn808)
46. Blouet C, Schwartz GJ. 2010 Hypothalamic nutrient sensing in the control of energy homeostasis. Behav. Brain Res. 209, 1–12. (doi:10.1016/j.bbr. 2009.12.024)
47. Woody EZ, Szechtman H. 2011 Adaptation to potential threat: the evolution, neurobiology, and psychopathology of the security motivation system. Neurosci. Biobehav. Rev. 35, 1019–1033. (doi:10. 1016/j.neubiorev.2010.08.003)
48. Flanagan-Cato LM. 2011 Sex differences in the neural circuit that mediates female sexual receptivity. Front. Neuroendocrinol. 32, 124–136. (doi:10.1016/j.yfrne.2011.02.008)
49. Stolzenberg DS, Numan M. 2011 Hypothalamic interaction with the mesolimbic DA system in the control of the maternal and sexual behaviors in rats. Neurosci. Biobehav. Rev. 35, 826–847. (doi:10. 1016/j.neubiorev.2010.10.003)
50. Lin D, Boyle MP, Dollar P, Lee H, Lein E, Perona P, Anderson DJ. 2011 Functional identification of an aggression locus in the mouse hypothalamus. Nature 470, 221–226. (doi:10.1038/nature09736)
51. Borbely AA, Achermann PJ. 1992 Concepts and models of sleep regulation: an overview. J. Sleep Res. 1, 63–79. (doi:10.1111/j.1365-2869.1992. tb00013.x)
52. Panksepp J. 1998 Affective neuroscience: the foundations of human and animal emotions. Oxford, UK: Oxford University Press.
53. Gray JA, McNaughton N. 2000 The neuropsychology of anxiety: an enquiry into the´ functions of thesepto-hippocampal system. Oxford, UK: Oxford University Press.
54. Davis KL, Panksepp J. 2011 The brain’s emotional foundations of human personality and the affective neuroscience personality scales. Neurosci. Biobehav. Rev. 35, 1946–1958. (doi:10.1016/j.neubiorev. 2011.04.004)
55. Winstanley C et al. 2012 Search, goals and the brain. In Strüngmann Forum volume on cognitive search: evolution, algorithms and the brain (eds PMT ToddTT Hills, TW Robbins), pp. 125–156. Cambridge, MT: MIT Press.
56. Weinberger NM. 1993 Learning-induced changes of auditory receptive fields. Curr. Opin. Neurobiol. 3, 570–577. (doi:10.1016/0959-4388(93)90058-7)
57. Sugrue LP, Corrado GS, Newsome WT. 2004 Matching behavior and the representation of value in the parietal cortex. Science 304, 1782–1787. (doi:10.1126/science.1094765)
58. Platt ML, Glimcher PW. 1999 Neural correlates of decision variables in parietal cortex. Nature 400, 233–238.(doi:10.1038/22268)
59. Shuler MG, Bear MF. 2006 Reward timing in the primary visual cortex. Science 311, 1606–1609. (doi:10.1126/science.1123513)
60. Vickery TJ, Chun MM, Lee D. 2011 Ubiquity and specificity of reinforcement signals throughout the human brain. Neuron 72, 166–177. (doi:10.1016/j. neuron.2011.08.011)
61. 2þ imaging reveals selective reward effects on stimulus-specific assemblies in mouse visual cortex. J. Neurosci. 33, 11540–11555. (doi:10.1523/JNEUROSCI.1341-12.2013)
62. Kilgard MP, Merzenich MM. 1998 Plasticity of temporal information processing in the primary auditory cortex. Nat. Neurosci. 1, 727–731. (doi:10. 1038/3729)
63. Tolman EC. 1948 Cognitive maps in rats and men. Psychol. Rev. 55, 189–208. (doi:10.1037/h0061626)
64. O’Keefe J, Dostrovsky J. 1971 The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat. Brain Res. 34, 171–175. (doi:10.1016/0006-8993(71)90358-1)
65. Tulving E. 2002 Episodic memory: from mind to brain. Annu. Rev. Psychol. 53, 1–25. (doi:10.1146/annurev.psych.53.100901.135114)
66. Wood ER, Dudchenko PA, Eichenbaum H. 1999 The global record of memory in hippocampal neuronal activity. Nature 397, 613–616. (doi:10.1038/17605)
67. Leutgeb S, Leutgeb JK, Barnes CA, Moser EI, McNaughton BL, Moser M-B. 2005 Independent codes for spatial and episodic memory in hippocampal neuronal ensembles. Science 309, 619–623. (doi:10.1126/science.1114037)
68. Lansink CS, Jackson JC, Lankelma JV, Ito R, Robbins T., Everitt BJ, Pennartz CMA. 2012 Reward cues in space: commonalities and differences in neural coding by hippocampal and ventral striatal ensembles. J. Neurosci. 32, 12444–12459. (doi:10. 1523/JNEUROSCI.0593-12.2012)
69. Kraus BJ, Robinson II RJ, White JA, Eichenbaum H, Hasselmo ME. 2013 Hippocampal ‘time cells’: time versus path integration. Neuron 78, 1090–1101. (doi:10.1016/j.neuron.2013.04.015)
70. Pastalkova E, Itskov V, Amarasingham A, Buzsáki G. 2008 Internally generated cell assembly sequences in the rat hippocampus. Science 321, 1322–1327. (doi:10.1126/science.1159775)
71. Muller RU, Kubie JL. 1987 The effects of changes in the environment on the spatial firing of hippocampal complex-spike cells. J. Neurosci. 7, 1951–1968.
72. Rennó-Costa C, Lisman JE, Verschure PFMJ. 2010 The mechanism of rate remapping in the dentate gyrus. Neuron 68, 1051–1058. (doi:10.1016/j. neuron.2010.11.024)
73. Rennó-Costa C, Lisman JE, Verschure PFMJ. 2014 A signature of attractor dynamics in the CA3 region of the hippocampus. PLoS Comput. Biol. 10, e1003641. (doi:10.1371/journal.pcbi.1003641)
74. Lisman JE. 2007 Role of the dual entorhinal inputs to hippocampus: a hypothesis based on cue/action (non-self/self) couplets. Prog. Brain Res. 163, 615–625. (doi:10.1016/S0079-6123(07)63033-7)
75. Leutgeb JK, Leutgeb S, Moser M-B, Moser EI. 2007 Pattern separation in the dentate gyrus and CA3 of the hippocampus. Science 315, 961–966. (doi:10. 1126/science.1135801)
76. Lu L, Leutgeb JK, Tsao A, Henriksen EJ, Leutgeb S, Barnes CA, Witter MP, Moser M-B, Moser EI. 2013 Impaired hippocampal rate coding after lesions of the lateral entorhinal cortex. Nat. Neurosci. 16, 1085–1093. (doi:10.1038/nn.3462)
77. Rondi-Reig L, Petit GH, Tobin C, Tonegawa S, Mariani J, Berthoz A. 2006 Impaired sequential egocentric and allocentric memories in forebrain-specific–NMDA receptor knock-out mice during a new task dissociating strategies of navigation. J. Neurosci. 26, 4071–4081. (doi:10.1523/ JNEUROSCI.3408-05.2006)
78. Cabral HO, Vinck M, Fouquet C, Pennartz C, Rondi-Reig L, Battaglia FP. 2014 Oscillatory dynamics and place field maps reflect hippocampal ensemble processing of sequence and place memory under NMDA receptor control. Neuron 81, 402–415. (doi:10.1016/j.neuron.2013.11.010)
79. Eichenbaum H, Dudchenko P, Wood E, Shapiro M, Tanila H. 1999 The hippocampus, memory, and place cells: is it spatial memory or a memory space? Neuron 23, 209–216. (doi:10.1016/S0896-6273 (00)80773-4)
80. Packard MG, McGaugh JL. 1996 Inactivation of hippocampus or caudate nucleus with lidocaine differentially affects expression of place and response learning. Neurobiol. Learn. Mem. 65, 65–72. (doi:10.1006/nlme.1996.0007)
81. Yin BJK. 2006 The role of the basal ganglia in habit formation. Nat. Rev. Neurosci. 7, 464 –476. (doi:10. 1038/nrn1919)
82. Wilson MA, McNaughton BL. 1994 Reactivation of hippocampal ensemble memories during sleep. Science 265, 676–679. (doi:10.1126/science.8036517)
83. Carr MF, Jadhav SP, Frank LM. 2011 Hippocampal replay in the awake state: a potential substrate for memory consolidation and retrieval. Nat. Neurosci. 14, 147–153. (doi:10.1038/nn.2732)
84. Dragoi G, Tonegawa S. 2010 Preplay of future place cell sequences by hippocampal cellular assemblies. Nature 469, 397-401. (doi:10.1038/nature09633)
85. Pennartz CM., Uylings HBM, Barnes CA, McNaughton BL. 2002 Memory reactivation and consolidation during sleep: from cellular mechanisms to human performance. In Progress in brain research, vol. 138: plasticity in the adult brain: from genes to neurotherapy (eds MA Hofman, GJ Boer, AJGD Holtmaat, EJW Van Someren, J Verhaagen, DF Swaab), pp. 143–166. Amsterdam, The Netherlands: Elsevier.
86. Kudrimoti HS, Barnes CA, McNaughton BL. 1999 Reactivation of hippocampal cell assemblies: effects of behavioral state, experience, and EEG dynamics. J. Neurosci. 19, 4090–4101.
87. Lansink CS, Goltstein PM, Lankelma JV, McNaughton BL, Pennartz CM. 2009 Hippocampus leads ventral striatum in replay of place-reward information. PLoS Biol. 7, e1000173. (doi:10.1371/journal.pbio.1000173)
88. Jadhav SP, Kemere C, German PW, Frank LM. 2012 Awake hippocampal sharp-wave ripples support spatial memory. Science 336, 1454–1458. (doi:10. 1126/science.1217230)
89. Johnson A, Redish AD. 2007 Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. J. Neurosci. 27, 12176–12189. (doi:10.1523/JNEUROSCI.3761-07.2007)
90. Pfeiffer BE, Foster DJ. 2013 Hippocampal place-cell sequences depict future paths to remembered goals. Nature 497, 74–79. (doi:10.1038/nature12112)
91. Pezzulo G, van der Meer MAA, Lansink CS, Pennartz CM.. In press. Internally generated sequences in learning and executing goal-directed behavior. Trends Cogn. Sci.
92. Hollup SA, Hollup SA, Molden S, Molden S, Donnett JG, Donnett JG, Moser M-B, Moser EI. 2001 Place fields of rat hippocampal pyramidal cells and spatial learning in the watermaze. Eur. J. Neurosci. 13, 1197–1208. (doi:10.1046/j.0953-816x.2001. 01487.x)
93. Passingham RE, Wise SP. 2012 The neurobiology of the prefrontal cortex: anatomy, evolution, and the origin of insight. Oxford, UK: Oxford University Press.
94. Wallis JD, Anderson KC, Miller EK. 2001 Single neurons in prefrontal cortex encode abstract rules. Nature 411, 953–956. (doi:10.1038/35082081)
95. Mulder AB, Nordquist RE, Pennartz CMA. 2003 Learning-related changes in response patterns of prefrontal neurons during instrumental conditioning. Behav. Brain Res. 146, 77–88. (doi:10.1016/j.bbr.2003.09.016)
96. Averbeck B, Sohn J, Lee D. 2006 Activity in prefrontal cortex during dynamic selection of action sequences. Nat. Neurosci. 9, 276–282. (doi:10. 1038/nn1634)
97. Histed MH, Miller EK. 2006 Microstimulation of frontal cortex can reorder a remembered spatial sequence. PLoS Biol. 4, e134. (doi:10.1371/journal. pbio.0040134)
98. Hok V, Save E, Lenck-Santini P, Poucet B. 2005 Coding for spatial goals in the prelimbic/infralimbic area of the rat frontal cortex. Proc. Natl Acad. Sci. USA 102, 4602–4607. (doi:10.1073/pnas.0407332102)
99. Tsujimoto S, Genovesio A, Wise SP. 2008 Transient neuronal correlations underlying goal selection and maintenance in prefrontal cortex. Cereb. Cortex 18, 2748–2761. (doi:10.1093/cercor/bhn033)
100. Watanabe M. 1992 Frontal units of the monkey coding the associative significance of visual and auditory stimuli. Exp. Brain Res. 89, 233–247. (doi:10.1007/BF00228241)
101. Tremblay L, Schultz W. 1999 Relative reward preference in primate orbitofrontal cortex. Nature 398, 704–708. (doi:10.1038/19525)
102. Schoenbaum G, Setlow B, Saddoris MP, Gallagher M. 2003 Encoding predicted outcome and acquired value in orbitofrontal cortex during cue sampling depends upon input from basolateral amygdala. Neuron 39, 855–867. (doi:10.1016/S0896-6273 (03)00474-4)
103. van Wingerden M, Vinck M, Tijms V, Ferreira IR, Jonker AJ, Pennartz C. 2012 NMDA receptors control cue-outcome selectivity and plasticity of orbitofrontal firing patterns during associative stimulus-reward learning. Neuron 76, 813–825. (doi:10.1016/j.neuron.2012.09.039)
104. Padoa-Schioppa C, Assad JA. 2006 Neurons in the orbitofrontal cortex encode economic value. Nature 441, 223–226. (doi:10.1038/nature04676)
105. Hosokawa T, Kennerley SW, Sloan J, Wallis JD. 2013 Single-neuron mechanisms underlying cost-benefit analysis in frontal cortex. J. Neurosci. 33, 17385–17397. (doi:10.1523/JNEUROSCI.2221-13.2013)
106. Kennerley SW, Dahmubed AF, Lara AH, Wallis JD. 2009 Neurons in the frontal lobe encode the value of multiple decision variables. J. Cogn. Neurosci. 21, 1162–1178. (doi:10.1162/jocn.2009.21100)
107. Dias R, Robbins T, Roberts A. 1996 Dissociation in prefrontal cortex of affective and attentional shifts. Nature 380, 69–72. (doi:10.1038/380069a0)
108. Birrell JM, Brown VJ. 2000 Medial frontal cortex mediates perceptual attentional set shifting in the rat. J. Neurosci. 20, 4320–4324.
109. Balleine BW KA, Dickinson A. 2003 The effect of lesions of the basolateral amygdala on instrumental conditioning. J. Neurosci. 23, 666–675.
110. Kouneiher F, Charron S, Koechlin E. 2009 Motivation and cognitive control in the human prefrontal cortex. Nat. Neurosci. 12, 939–945. (doi:10.1038/nn.2321)
111. Corbit LH, Leung BK, Balleine BW. 2013 The role of the amygdala-striatal pathway in the acquisition and performance of goal-directed instrumental actions. J. Neurosci. 33, 17682–17690. (doi:10. 1523/JNEUROSCI.3271-13.2013)
112. Rhodes SE, Murray EA. 2013 Differential effects of amygdala, orbital prefrontal cortex, and prelimbic cortex lesions on goal-directed behavior. J. Neurosci. 33, 3380–3389. (doi:10.1523/JNEUROSCI.4374-12.2013)
113. Shadlen MN, Kiani R. 2013 Decision making as a window on cognition. Neuron 80, 791–806. (doi:10.1016/j.neuron.2013.10.047)
114. Marcos E, Pani P, Brunamonti E, Deco G, Verschure P. 2013 Neural variability in premotor cortex is modulated by trial history and predicts behavioral performance. Neuron 78, 1–7. (doi:10.1016/j.neuron. 2013.02.006)
115. Voorn P, Vanderschuren LJ, Groenewegen HJ, Robbins T., Pennartz C. 2004 Putting a spin on the dorsal– ventral divide of the striatum. Trends Neurosci. 27, 468–474. (doi:10.1016/j.tins.2004.06.006)
116. Alexander GE, Crutcher MD, DeLong MR. 1991 Basal ganglia-thalamocortical circuits: parallel substrates for motor, oculomotor,’prefrontal’ and ‘limbic’ functions. Progress Brain Res. 85, 119–146. (doi:10. 1016/S0079-6123(08)62678-3)
117. Tunstall MJ, Oorschot DE, Kean A, Wickens JR. 2002 Inhibitory interactions between spiny projection neurons in the rat striatum. J. Neurophysiol. 88, 1263–1269.
118. Taverna S, Van Dongen YC, Groenewegen HJ, Pennartz CM. 2004 Direct physiological evidence for synaptic connectivity between medium-sized spiny neurons in rat nucleus accumbens in situ. J. Neurophysiol. 91, 1111–1121. (doi:10.1152/jn. 00892.2003)
119. Redgrave P, Prescott T, Gurney KN. 1999 The basal ganglia: a vertebrate solution to the selection problem? Neuroscience 89, 1009–1023. (doi:10. 1016/S0306-4522(98)00319-4)
120. Barto AG. 1995 Adaptive critics and the basal ganglia. In Models of information processing in the basal ganglia (eds CH James, JL Davis, DG Beiser), pp. 215–232. Cambridge, MA: MIT Press.
121. Schultz W. 1998 Predictive reward signal of dopamine neurons. J. Neurophysiol. 80, 1–27.
122. Yin HH, Ostlund SB, Knowlton BJ, Balleine BW. 2005 The role of the dorsomedial striatum in instrumental conditioning. Eur. J. Neurosci. 22, 513–523. (doi:10.1111/j.1460-9568.2005. 04218.x)
123. Di Ciano P, Everitt BJ. 2004 Direct interactions between the basolateral amygdala and nucleus accumbens core underlie cocaine-seeking behavior by rats. J. Neurosci. 24, 7167–7173. (doi:10.1523/ JNEUROSCI.1581-04.2004)
124. Ito R, Robbins T., Pennartz C., Everitt BJ. 2008 Functional interaction between the hippocampus and nucleus accumbens shell is necessary for the acquisition of appetitive spatial context conditioning. J. Neurosci. 28, 6950–6959. (doi:10. 1523/JNEUROSCI.1615-08.2008)
125. Schultz W. 2007 Multiple dopamine functions at different time courses. Annu. Rev. Neurosci. 30, 259–288. (doi:10.1146/annurev.neuro.28.061604.135722)
126. Hikosaka O. 2010 The habenula: from stress evasion to value-based decision-making. Nat. Rev. Neurosci. 11, 503–513. (doi:10.1038/nrn2866)
127. McDannald MA, Takahashi YK, Lopatina N, Pietras BW, Jones JL, Schoenbaum G. 2012 Model-based learning and the contribution of the orbitofrontal cortex to the model-free world. Eur. J. Neurosci. 35, 991–996.(doi:10.1111/j.1460-9568.2011.07982.x)
128. Jones JL, Esber GR, McDannald MA, Gruber AJ, Hernandez A, Mirenzi A, Schoenbaum G. 2012 Orbitofrontal cortex supports behavior and learning using inferred but not cached values. Science 338, 953–956. (doi:10.1126/science.1227489)
129. Van der Meer MA, Redish AD. 2009 Covert expectation-of-reward in rat ventral striatum at decision points. Front. Integr. Neurosci. 3, 1. (doi:10. 3389/neuro.07.001.2009)
130. Steiner AP, Redish AD. 2012 The road not taken: neural correlates of decision making in orbitofrontal cortex. Front. Neurosci. 6, 131. (doi:10.3389/fnins.2012.00131)
131. Balleine BW, Leung BK, Ostlund SB. 2011 The orbitofrontal cortex, predicted value, and choice. Ann. NY Acad. Sci. 1239, 43–50. (doi:10.1111/j. 1749-6632.2011.06270.x)
132. Jones MW, Wilson MA. 2005 Theta rhythms coordinate hippocampal–prefrontal interactions in a spatial memory task. PLoS Biol. 3, e402. (doi:10. 1371/journal.pbio.0030402)
133. Euston DR, Tatsuno M, McNaughton BL. 2007 Fast-forward playback of recent memory sequences in prefrontal cortex during sleep. Science 318, 1147–1150. (doi:10.1126/science.1148979)
134. Daw ND, Niv Y, Dayan P. 2005 Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704–1711. (doi:10.1038/nn1560)
135. Kalenscher T, Pennartz C. 2008 Is a bird in the hand worth two in the future? The neuroeconomics of intertemporal decision-making. Prog. Neurobiol. 84, 284–315. (doi:10.1016/j.pneurobio.2007.11.004)
136. Edelman GM. 2007 Learning in and from brain-based devices. Science 318, 1103–1105. (doi:10. 1126/science.1148677)
137. Pezzulo G, Castelfranchi C. 2009 Thinking as the control of imagination: a conceptual framework for goal-directed systems. Psychol. Res. 73, 559–577. (doi:10.1007/s00426-009-0237-z)
138. Pezzulo G, Castelfranchi C. 2007 The symbol detachment problem. Cogn. Process. 8, 115–131. (doi:10.1007/s10339-007-0164-0)
139. Duff A, Sanchez-Fibla M, Verschure PFMJ. 2011 A biologically based model for the integration of sensory-motor contingencies in rules and plans: a prefrontal cortex based extension of the distributed adaptive control architecture. Brain Res. Bull. 85, 289–304. doi:10.1016/j.brainresbull. 2010.11.008)
140. Verschure PFMJ. 2012 The distributed adaptive control architecture of the mind, brain, body nexus. Biol. Inspired Cogn. Archit. 1, 55–72.
141. Pezzulo G, Rigoli F, Chersi F. 2013 The mixed instrumental controller: using value of Information to combine habitual choice and mental simulation. Front. Psychol. 4, 92. (doi:10.3389/fpsyg.2013.00092)
142. Pezzulo G, Barsalou LW, Cangelosi A, Fischer MH, McRae K, Spivey MJ. 2011 The mechanics of embodiment: a dialog on embodiment and computational modeling. Front. Psychol. 2, 5. (doi:10.3389/fpsyg.2011.00005)