Decision-Theoretic Psychiatry

Clinical Psychological Science

Volumn 3, Issue 3, 2015, Pages 400-421

  Huys, Quentin J M ^a,b   Guitart Masip, Marc ^c   Dolan, Raymond J ^d,e   Dayan, Peter ^f  

^a ETH ZURICH   (Switzerland)

^b UNIVERSITY OF ZURICH   (Switzerland)

^c KAROLINSKA INSTITUTE   (Sweden)

^d UNIVERSITY COLLEGE LONDON   (United Kingdom)

^e UNIVERSITY COLLEGE LONDON   (United Kingdom)

^f UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

Bayesian decision theory; cognition(s); computational psychiatry; environmental prior distributions; reinforcement learning

Indexed keywords

EID: 84924316005     PISSN: 21677026     EISSN: 21677034     Source Type: Journal    
DOI: 10.1177/2167702614562040     Document Type: Review

References (227)

1. R.A.AdamsK.E.StephanH.R.BrownC.D.FrithK.J.Friston (2013). The computational anatomy of psychosis. Frontiers in Psychiatry, 4, 47.
2. G.Ainslie (2001). Breakdown of will. Cambridge, England: Cambridge University Press.
3. G.E.AlexanderM.D.Crutcher (1990). Functional architecture of basal ganglia circuits: Neural substrates of parallel processing. Trends in Neurosciences, 13, 266–271.
4. J.AmatM.V.BarattaE.PaulS.T.BlandL.R.WatkinsS.F.Maier (2005). Medial prefrontal cortex determines how stressor controllability affects behavior and dorsal raphe nucleus. Nature Neuroscience, 8, 365–371.
5. American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders (4th ed.). Washington, DC: Author.
6. J.E.AnsonL.BenderK.B.Melvin (1969). Sources of reinforcement in the establishment of self-punitive behavior. Journal of Comparative and Physiological Psychology, 67, 376–380.
7. A.F.Arnsten (2009). Stress signalling pathways that impair prefrontal cortex structure and function. Nature Reviews Neuroscience, 10, 410–422.
8. B.W.Balleine (2005). Neural bases of food-seeking: Affect, arousal and reward in corticostriatolimbic circuits. Physiology and Behavior, 86, 717–730.
9. R.BandlerM.T.Shipley (1994). Columnar organization in the midbrain periaqueductal gray: Modules for emotional expression?Trends in Neurosciences, 17, 379–389.
10. C.S.BarrT.K.NewmanM.SchwandtC.ShannonR.L.DvoskinS.G.LindellJ.D.. Higley (2004). Sexual dichotomy of an interaction between early adversity and the serotonin transporter gene promoter variant in rhesus macaques. Proceedings of the National Academy of Sciences of the United States of America, 101, 12358–12363.
11. F.C.Bartlett (1932). Remembering: A study in experimental and social psychology. Cambridge, England: Cambridge University Press.
12. A.G.BartoR.S.SuttonC.W.Anderson (1983). Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man and Cybernetics, 13, 834–846.
13. A.T.BeckA.J.RushB.F.ShawG.Emery (1979). Cognitive therapy of depression (1st ed.). New York, NY: Guilford.
14. U.BeierholmM.Guitart-MasipM.EconomidesR.ChowdhuryE.DüzelR.DolanP.Dayan (2013). Dopamine modulates reward-related vigor. Neuropsychopharmacology, 38, 1495–1503.
15. D.BelinS.JonkmanA.DickinsonT.W.RobbinsB.J.Everitt (2009). Parallel and interactive learning processes within the basal ganglia: Relevance for the understanding of addiction. Behavioural Brain Research, 199, 89–102.
16. R.E.Bellman (1957). Dynamic programming. Princeton, NJ: Princeton University Press.
17. R.P.Bentall (2004). Abandoning the concept of schizophrenia. In (Eds.), Models of madness (pp. 195–208). New York, NY: Routledge.
18. J.O.Berger (1985). Statistical decision theory and Bayesian analysis. Berlin, Germany: Springer Verlag.
19. R.C.BollesR.HoltzT.DunnW.Hill (1980). Comparisons of stimulus learning and response learning in a punishment situation. Learning and Motivation, 11, 78–96.
20. M.E.Bouton (2002). Context, ambiguity, and unlearning: Sources of relapse after behavioral extinction. Biological Psychiatry, 52, 976–986.
21. K.BrelandM.Breland (1961). The misbehavior of organisms. American Psychologist, 16, 681–684.
22. C.A.BrennerG.P.KrishnanJ.L.VohsW.-Y.AhnW.P.HetrickS.L.MorzoratiB.F.O’Donnell (2009). Steady state responses: Electrophysiological assessment of sensory function in schizophrenia. Schizophrenia Bulletin, 35, 1065–1077.
23. J.Brock (2012). Alternative Bayesian accounts of autistic perception: Comment on Pellicano and Burr. Trends in Cognitive Sciences, 16, 573–574; author reply 574–575.
24. J.W.BuckholtzM.T.TreadwayR.L.CowanN.D.WoodwardR.LiM.S.AnsariD.H.. Zald (2010). Dopaminergic network differences in human impulsivity. Science, 329, 532.
25. L.M.BylsmaB.H.MorrisJ.Rottenberg (2008). A meta-analysis of emotional reactivity in major depressive disorder. Clinical Psychology Review, 28, 676–691.
26. M.CampbellA.J.HoaneF.-H.Hsu (2002). Deep blue. Artificial Intelligence, 134, 57–83.
27. G.A.CarpenterS.Grossberg (1988). The ART of adaptive pattern recognition by a self-organizing neural network. Computer, 21, 77–88.
28. B.L.CarterS.T.Tiffany (1999). Meta-analysis of cue-reactivity in addiction research. Addiction, 94, 327–340.
29. C.S.CarverR.J.Ganellen (1983). Depression and components of self-punitiveness: High standards, self-criticism, and overgeneralization. Journal of Abnormal Psychology, 92, 330–337.
30. C.S.CarverT.L.White (1994). Behavioral inhibition, behavioral activation, and affective responses to impending reward and punishment: The BIS/BAS scales. Journal of Personality and Social Psychology, 67, 319–333.
31. J.CavanaghI.EisenbergM.Guitart-MasipQ.J.M.HuysM.J.Frank (2013). Frontal theta overrides Pavlovian learning biases. Journal of Neuroscience, 33, 8541–8548.
32. S.R.ChamberlainT.W.Robbins (2013). Noradrenergic modulation of cognition: Therapeutic implications. Journal of Psychopharmacology, 27, 694–718.
33. W.G.ChaseH.A.Simon (1973). Perception in chess. Cognitive Psychology, 4, 55–81.
34. B.T.ChenH.-J.YauC.HatchI.Kusumoto-YoshidaS.L.ChoF.W.HopfA.Bonci (2013). Rescuing cocaine-induced prefrontal cortex hypoactivity prevents compulsive cocaine seeking. Nature, 496, 359–362.
35. N.Chomsky (1965). Aspects of the theory of syntax. Cambridge, MA: MIT Press.
36. P.S.Churchland (1986). Neurophilosophy: Toward a unified science of the mind-brain. Cambridge, MA: MIT Press.
37. D.M.ClarkA.EhlersA.HackmannF.McManusM.FennellN.GreyJ.. Wild (2006). Cognitive therapy versus exposure and applied relaxation in social phobia: A randomized controlled trial. Journal of Consulting and Clinical Psychology, 74, 568–578.
38. A.G.E.CollinsM.J.Frank (2012). How much of reinforcement learning is working memory, not reinforcement learning? A behavioral, computational, and neurogenetic analysis. European Journal of Neuroscience, 35, 1024–1035.
39. M.Coltheart (2007). Cognitive neuropsychiatry and delusional belief. Quarterly Journal of Experimental Psychology, 60, 1041–1062.
40. A.C.CourvilleN.DawG.J.GordonD.S.Touretzky (2004). Model uncertainty in classical conditioning. In (Eds.), Advances in neural information processing systems16 (pp. 977–984). Cambridge, MA: MIT Press.
41. A.C.CourvilleN.D.DawD.S.Touretzky (2005). Similarity and discrimination in classical conditioning: A latent variable account. In (Eds.), Advances in neural information processing systems 17 (pp. 313–320). Cambridge, MA: MIT Press.
42. T.B.CrapseM.A.Sommer (2008). Corollary discharge across the animal kingdom. Nature Reviews: Neuroscience, 9, 587–600.
43. M.J.CrockettB.R.BraamsL.ClarkP.N.ToblerT.W.RobbinsT.Kalenscher (2013). Restricting temptations: Neural mechanisms of precommitment. Neuron, 79, 391–401.
44. M.J.CrockettL.ClarkT.W.Robbins (2009). Reconciling the role of serotonin in behavioral inhibition and aversion: Acute tryptophan depletion abolishes punishment-induced inhibition in humans. Journal of Neuroscience, 29, 11993–11999.
45. H.S.CrombagJ.M.BossertE.KoyaY.Shaham (2008). Context-induced relapse to drug seeking: A review. Philosophical Transactions of the Royal Society B: Biological Sciences, 363, 3233–3243.
46. K.D’ArdenneS.M.McClureL.E.NystromJ.D.Cohen (2008). BOLD responses reflecting dopaminergic signals in the human ventral tegmental area. Science, 319, 1264–1267.
47. N.D.DawA.C.CourvilleP.Dayan (2008). Semi-rational models of conditioning: The case of trial order. In (Eds.), The probabilistic mind (pp. 431–452). Oxford, England: Oxford University Press.
48. N.D.DawS.J.GershmanB.SeymourP.DayanR.J.Dolan (2011). Model-based influences on humans’ choices and striatal prediction errors. Neuron, 69, 1204–1215.
49. N.D.DawY.NivP.Dayan (2005). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience, 8, 1704–1711.
50. P.Dayan (2009). Dopamine, reinforcement learning, and addiction. Pharmacopsychiatry, 42(Suppl. 1), S56–S65.
51. P.Dayan (2012). How to set the switches on this thing. Current Opinion in Neurobiology, 22, 1068–1074.
52. P.DayanK.Berridge (2014). Model-based and model-free Pavlovian reward learning: Revaluation, revision and revelation. Cognitive, Affective and Behavioral Neuroscience, 14, 473–492.
53. P.DayanN.D.Daw (2008). Decision theory, reinforcement learning, and the brain. Cognitive, Affective and Behavioral Neuroscience, 8, 429–453.
54. P.DayanQ.J.M.Huys (2008). Serotonin, inhibition, and negative mood. PLoS Computational Biology, 4, e4.
55. P.DayanY.NivB.SeymourN.D.Daw (2006). The misbehavior of value and the discipline of the will. Neural Networks, 19, 1153–1160.
56. B.Dell’OssoI.D.GlickD.S.BaldwinA.C.Altamura (2013). Can long-term outcomes be improved by shortening the duration of untreated illness in psychiatric disorders? A conceptual framework. Psychopathology, 46, 14–21.
57. A.Dickinson (1980). Contemporary animal learning theory. Cambridge, England: Cambridge University Press.
58. A.Dickinson (1985). Actions and habits: The development of behavioural autonomy. Philosophical Transactions of the Royal Society of London. B, Biological Sciences, 308, 67–78.
59. A.DickinsonB.W.Balleine (2002). The role of learning in motivation. In (Ed.), Learning, motivation & emotion: Steven’s handbook of experimental psychology (pp. 497–533). New York, NY: John Wiley.
60. R.J.DolanP.Dayan (2013). Goals and habits in the human brain. Neuron, 80, 312–325.
61. B.B.DollD.A.SimonN.D.Daw (2012). The ubiquity of model-based reinforcement learning. Current Opinion in Neurobiology, 22, 1075–1081.
62. M.Domjan (2005). Pavlovian conditioning: A functional perspective. Annual Review of Psychology, 56, 179–206.
63. K.Doya (1999). What are the computations of the cerebellum, the basal ganglia and the cerebral cortex?Neural Networks, 12, 961–974.
64. S.Droit-VoletW.H.Meck (2007). How emotions colour our perception of time. Trends in Cognitive Sciences, 11, 504–513.
65. H.D.Ellis (1998). Cognitive neuropsychiatry and delusional misidentification syndromes: An exemplary vindication of the new discipline. Cognitive Neuropsychiatry, 3, 81–89.
66. J.L.Evenden (1999). Varieties of impulsivity. Psychopharmacology, 146, 348–361.
67. B.J.EverittT.W.Robbins (2005). Neural systems of reinforcement for drug addiction: From actions to habits to compulsion. Nature Neuroscience, 8, 1481–1489.
68. C.FassbenderS.HoudeS.Silver-BalbusK.BallardB.K.KimK.J.RutledgeS.M.. McClure (2014). The decimal effect: Behavioral and neural bases for a novel influence on intertemporal choice in healthy individuals and in ADHD. Journal of Cognitive Neuroscience, 26, 2455–2468.
69. A.FaureS.M.ReynoldsJ.M.RichardK.C.Berridge (2008). Mesolimbic dopamine in desire and dread: Enabling motivation to be generated by localized glutamate disruptions in nucleus accumbens. Journal of Neuroscience, 28, 7184–7192.
70. A.FaureJ.M.RichardK.C.Berridge (2010). Desire and dread from the nucleus accumbens: Cortical glutamate and subcortical GABA differentially generate motivation and hedonic impact in the rat. PLoS ONE, 5, e11223.
71. J.FiserP.BerkesG.OrbanM.Lengyel (2010). Statistically optimal perception and learning: From behavior to neural representations. Trends in Cognitive Sciences, 14, 119–130.
72. S.B.FlagelJ.J.ClarkT.E.RobinsonL.MayoA.CzujI.WilluhnH.. Akil (2011). A selective role for dopamine in stimulus-reward learning. Nature, 469, 53–57.
73. P.C.FletcherC.D.Frith (2009). Perceiving is believing: A Bayesian approach to explaining the positive symptoms of schizophrenia. Nature Reviews Neuroscience, 10, 48–58.
74. M.J.Frank (2005). Dynamic dopamine modulation in the basal ganglia: A neurocomputational account of cognitive deficits in medicated and nonmedicated Parkinsonism. Journal of Cognitive Neuroscience, 17, 51–72.
75. P.A.GaretyD.FreemanS.JolleyG.DunnP.E.BebbingtonD.G.FowlerR.. Dudley (2005). Reasoning, emotions, and delusional conviction in psychosis. Journal of Abnormal Psychology, 114, 373–384.
76. M.GelderP.HarrisonP.Cowen (2006). Shorter Oxford textbook of psychiatry. Oxford, England: Oxford University Press.
77. S.J.GershmanD.M.BleiY.Niv (2010). Context, learning, and extinction. Psychological Review, 117, 197–209.
78. S.J.GershmanA.B.MarkmanA.R.Otto (2014). Retrospective revaluation in sequential decision making: A tale of two systems. Journal of Experimental Psychology: General, 143, 182–194.
79. S.J.GershmanY.Niv (2012). Exploring a latent cause theory of classical conditioning. Learning & Behavior, 40, 255–268.
80. J.Gibbon (1977). Scalar expectancy theory and Weber’s law in animal timing. Psychological Review, 84, 279–325.
81. J.A.Gray (1982). The neuropsychology of anxiety: An enquiry into the functions of the septo-hippocampal system. Oxford, England: Clarendon.
82. C.GrillonK.SmithA.HaynosL.K.Nieman (2004). Deficits in hippocampus-mediated Pavlovian conditioning in endogenous hypercortisolism. Biological Psychiatry, 56, 837–843.
83. S.M.GruesserJ.WraseS.KleinD.HermannM.N.SmolkaM.RufA.. Heinz (2004). Cue-induced activation of the striatum and medial prefrontal cortex is associated with subsequent relapse in abstinent alcoholics. Psychopharmacology, 175, 296–302.
84. M.Guitart-MasipR.ChowdhuryT.SharotP.DayanE.DuzelR.J.Dolan (2012). Action controls dopaminergic enhancement of reward representations. Proceedings of the National Academy of Sciences of the United States of America, 109, 7511–7516.
85. M.Guitart-MasipQ.J.M.HuysL.FuentemillaP.DayanE.DuzelR.J.Dolan (2012). Go and no-go learning in reward and punishment: Interactions between affect and effect. NeuroImage, 62, 154–166.
86. S.N.HaberJ.L.FudgeN.R.McFarland (2000). Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. Journal of Neuroscience, 20, 2369–2382.
87. P.W.HalliganA.S.David (2001). Cognitive neuropsychiatry: Towards a scientific psychopathology. Nature Reviews: Neuroscience, 2, 209–215.
88. A.N.HamptonP.BossaertsJ.P.O’Doherty (2006). The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. Journal of Neuroscience, 26, 8360–8367.
89. F.HappeU.Frith (2006). The weak coherence account: Detail-focused cognitive style in autism spectrum disorders. Journal of Autism and Developmental Disorders, 36, 5–25.
90. C.J.HarmerP.J.Cowen (2013). “It’s the way that you look at it”—A cognitive neuropsychological account of SSRI action in depression. Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences, 368, 20120407.
91. M.HarunoM.Kawato (2006). Heterarchical reinforcement-learning model for integration of multiple cortico-striatal loops: fMRI examination in stimulus-action-reward association learning. Neural Networks, 19, 1242–1254.
92. G.HaslerW.C.DrevetsH.K.ManjiD.S.Charney (2004). Discovering endophenotypes for major depression. Neuropsychopharmacology, 29, 1765–1781.
93. D.HassabisD.KumaranS.D.VannE.A.Maguire (2007). Patients with hippocampal amnesia cannot imagine new experiences. Proceedings of the National Academy of Sciences of the United States of America, 104, 1726–1731.
94. D.HassabisE.A.Maguire (2009). The construction system of the brain. Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences, 364, 1263–1271.
95. T.T.HaugS.BlomhoffK.HellstromI.HolmeM.HumbleH.P.MadsbuJ.E.Wold (2003). Exposure therapy and sertraline in social phobia: 1-year follow-up of a randomised controlled trial. British Journal of Psychiatry, 182, 312–318.
96. N.HayS.J.Russell (2011). Metareasoning for Monte Carlo tree search (Tech. Rep. UCB/EECS-2011-119). Berkeley: University of California, Berkeley, EECS Department.
97. C.HeimD.J.NewportS.HeitY.P.GrahamM.WilcoxR.BonsallC.B.. Nemeroff (2000). Pituitary-adrenal and autonomic responses to stress in women after sexual and physical abuse in childhood. Journal of the American Medical Association, 284, 592–597.
98. D.Hemsley (1987). An experimental psychological model for schizophrenia. In (Eds.), Search for the causes of schizophrenia (pp. 179–188). New York, NY: Springer.
99. D.R.Hemsley (1993). A simple (or simplistic?) cognitive model for schizophrenia. Behaviour Research and Therapy, 31, 633–645.
100. D.R.Hemsley (2005). The development of a cognitive model of schizophrenia: Placing it in context. Neuroscience and Biobehavioral Reviews, 29, 977–988.
101. W.A.Hershberger (1986). An approach through the looking-glass. Animal Learning and Behavior, 14, 443–451.
102. G.E.HintonT.J.Sejnowski (1999). Unsupervised learning: Foundations of neural computation. Cambridge, MA: MIT Press.
103. S.G.HofmannJ.Q.WuH.Boettcher (2013). D-Cycloserine as an augmentation strategy for cognitive behavioral therapy of anxiety disorders. Biology of Mood and Anxiety Disorders, 3, 11.
104. Q.J.M.Huys (2007). Reinforcers and control. Towards a computational aetiology of depression. London, England: UCL, University of London, Gatsby Computational Neuroscience Unit. Retrieved from http://www.gatsby.ucl.ac.uk/~qhuys/pub.html
105. Q.J.M.HuysA.BeckP.DayanA.Heinz (in press). Neurobiological structure and computational understanding of addictive behaviour. Phenomenological Neuropsychiatry: Bridging the Clinic and Clinical Neuroscience.
106. Q.J.M.HuysP.Dayan (2009). A Bayesian formulation of behavioral control. Cognition, 113, 314–328.
107. Q.J.M.HuysN.EshelE.O’NionsL.SheridanP.DayanJ.P.Roiser (2012). Bonsai trees in your head: How the Pavlovian system sculpts goal-directed choices by pruning decision trees. PLoS Computational Biology, 8, e1002410.
108. Q.J.M.HuysN.LallyP.Paul FaulknerN.EshelE.SeifritzS.J.GershmanJ.P.… Roiser (in press). The interplay of approximate planning strategies. Proceedings of the National Academy of Sciences, USA.
109. Q.J.M.HuysM.MoutoussisJ.Williams (2011). Are computational models of any use to psychiatry?Neural Networks, 24, 544–551.
110. Q.J.HuysD.A.PizzagalliR.BogdanP.Dayan (2013). Mapping anhedonia onto reinforcement learning: A behavioural meta-analysis. Biology of Mood and Anxiety Disorders, 3, 12.
111. R.K.JessupA.J.BisharaJ.R.Busemeyer (2008). Feedback produces divergence from prospect theory in descriptive choice. Psychological Science, 19, 1015–1022.
112. D.JoelI.Weiner (2000). The connections of the dopaminergic system with the striatum in rats and primates: An analysis with respect to the functional and compartmental organization of the striatum. Neuroscience, 96, 451–474.
113. A.JohnsonA.D.Redish (2007). Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. Journal of Neuroscience, 27, 12176–12189.
114. L.P.KaelblingM.L.LittmanA.R.Cassandra (1998). Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101, 99–134.
115. R.S.KahnW.W.FleischhackerH.BoterM.DavidsonY.VergouweI.P.M.KeetD.E.. Grobbee (2008). Effectiveness of antipsychotic drugs in first-episode schizophrenia and schizophreniform disorder: An open randomised clinical trial. Lancet, 371, 1085–1097.
116. D.KahnemanA.Tversky (1979). Prospect theory: An analysis of decision under risk. Econometrica, 47, 263–292.
117. S.M.Kakade (2003). On the sample complexity of reinforcement learning. London, England: University of London.
118. K.S.KendlerL.M.KarkowskiC.A.Prescott (1999). Causal relationship between stressful life events and the onset of major depression. American Journal of Psychiatry, 156, 837–841.
119. M.KeramatiB.S.Gutkin (2011). A reinforcement learning theory for homeostatic regulation. In (Eds.), Advances in neural information processing systems24 (pp. 82–90). Cambridge, MA: MIT Press.
120. M.KeramatiB.Gutkin (2013). Imbalanced decision hierarchy in addicts emerging from drug-hijacked dopamine spiraling circuit. PLoS ONE, 8, e61489.
121. C.KiddH.PalmeriR.N.Aslin (2013). Rational snacking: Young children’s decision-making on the marshmallow task is moderated by beliefs about environmental reliability. Cognition, 126, 109–114.
122. S.KillcrossE.Coutureau (2003). Coordination of actions and habits in the medial prefrontal cortex of rats. Cerebral Cortex, 13, 400–408.
123. B.King-CasasC.SharpL.Lomax-BreamT.LohrenzP.FonagyP.R.Montague (2008). The rupture and repair of cooperation in borderline personality disorder. Science, 321, 806–810.
124. K.N.KirbyN.M.PetryW.K.Bickel (1999). Heroin addicts have higher discount rates for delayed rewards than non-drug-using controls. Journal of Experimental Psychology: General, 128, 78–87.
125. K.KoerdingD.M.Wolpert (2004a). Bayesian integration in sensorimotor learning. Nature, 427, 244–247.
126. K.P.KoerdingD.M.Wolpert (2004b). The loss function of sensorimotor learning. Proceedings of the National Academy of Sciences of the United States of America, 101, 9839–9842.
127. W.KoolJ.T.McGuireZ.B.RosenM.M.Botvinick (2010). Decision making and the avoidance of cognitive demand. Journal of Experimental Psychology: General, 139, 665–682.
128. S.M.KorteJ.M.KoolhaasJ.C.WingfieldB.S.McEwen (2005). The Darwinian concept of stress: Benefits of allostasis and costs of allostatic load and the trade-offs in health and disease. Neuroscience and Biobehavioral Reviews, 29, 3–38.
129. A.V.KravitzL.D.TyeA.C.Kreitzer (2012). Distinct roles for direct and indirect pathway striatal neurons in reinforcement. Nature Neuroscience, 15, 816–818.
130. J.K.Kruschke (2006). Locally Bayesian learning with applications to retrospective revaluation and highlighting. Psychological Review, 113, 677–699.
131. Z.Kurth-NelsonW.BickelA.D.Redish (2012). A theoretical account of cognitive effects in delay discounting. European Journal of Neuroscience, 35, 1052–1064.
132. S.W.LeeS.ShimojoJ.P.O’Doherty (2014). Neural computations underlying arbitration between model-based and model-free learning. Neuron, 81, 687–699.
133. D.A.Lewis (2013). Inhibitory neurons in human cortical circuits: Substrate for cognitive dysfunction in schizophrenia. Current Opinion in Neurobiology, 26C, 22–26.
134. J.A.LiebermanT.S.StroupJ.P.McEvoyM.S.SwartzR.A.RosenheckD.O.PerkinsJ.K.. Hsiao (2005). Effectiveness of antipsychotic drugs in patients with chronic schizophrenia. New England Journal of Medicine, 353, 1209–1223.
135. F.LiederN.GoodmanQ.J.M.Huys (2013a, February). Controllability and resource-rational planning. Paper presented at COSYNE, Salt Lake City, UT.
136. F.LiederN.GoodmanQ.J.M.Huys (2013b, July). Learned helplessness and generalization. Paper presented at the Cognitive Science Conference, Berlin, Germany.
137. J.E.LismanJ.T.CoyleR.W.GreenD.C.JavittF.M.BenesS.HeckersA.A.Grace (2008). Circuit-based framework for understanding neurotransmitter and risk gene interactions in schizophrenia. Trends in Neurosciences, 31, 234–242.
138. K.LloydD.S.Leslie (2013). Context-dependent decision-making: A simple Bayesian model. Journal of the Royal Society Interface, 10. Retrieved from http://rsif.royalsocietypublishing.org/content/10/82/20130069
139. G.LoewensteinD.Prelec (1992). Anomalies in intertemporal choice: Evidence and an interpretation. Quarterly Journal of Economics, 107, 573–597.
140. V.LovicB.T.SaundersL.M.YagerT.E.Robinson (2011). Rats prone to attribute incentive salience to reward cues are also prone to impulsive action. Behavioural Brain Research, 223, 255–261.
141. D.J.MacKay (2003). Information theory, inference and learning algorithms. Cambridge, England: Cambridge University Press.
142. T.V.MaiaM.J.Frank (2011). From reinforcement learning models to psychiatric and neurological disorders. Nature Neuroscience, 14, 154–162.
143. S.F.MaierL.R.Watkins (2005). Stressor controllability and learned helplessness: The roles of the dorsal raphe nucleus, serotonin, and corticotropin-releasing factor. Neuroscience and Biobehavioral Reviews, 29, 829–841.
144. D.Marr (1982). Vision. New York, NY: Freeman.
145. P.MazzoniA.HristovaJ.W.Krakauer (2007). Why don’t we move faster? Parkinson’s disease, movement vigor, and implicit motivation. Journal of Neuroscience, 27, 7105–7116.
146. S.M.McClureN.D.DawP.R.Montague (2003). A computational substrate for incentive salience. Trends in Neurosciences, 26, 423–428.
147. B.S.McEwen (1998). Stress, adaptation, and disease: Allostasis and allostatic load. Annals of the New York Academy of Sciences, 840, 33–44.
148. J.T.McGuireM.M.Botvinick (2010). Prefrontal cortex, cognitive control, and the registration of decision costs. Proceedings of the National Academy of Sciences of the United States of America, 107, 7922–7926.
149. P.J.MeyerV.LovicB.T.SaundersL.M.YagerS.B.FlagelJ.D.MorrowT.E.Robinson (2012). Quantifying individual variation in the propensity to attribute incentive salience to reward cues. PLoS ONE, 7, e38987.
150. R.MitraS.JadhavB.S.McEwenA.VyasS.Chattarji (2005). Stress duration modulates the spatiotemporal patterns of spine formation in the basolateral amygdala. Proceedings of the National Academy of Sciences of the United States of America, 102, 9371–9376.
151. K.MiyazakiK.W.MiyazakiK.Doya (2012). The role of serotonin in the regulation of patience and impulsivity. Molecular Neurobiology, 45, 213–224.
152. J.MonahanH.J.SteadmanE.SilverP.S.AppelbaumP.C.RobbinsE.P.MulveyS.. Banks (2001). Rethinking risk assessment: The MacArthur study of mental disorder and violence. Oxford, England: Oxford University Press.
153. P.R.MontagueP.DayanT.J.Sejnowski (1996). A framework for mesencephalic dopamine systems based on predictive Hebbian learning. Journal of Neuroscience, 16, 1936–1947.
154. P.R.MontagueR.J.DolanK.J.FristonP.Dayan (2012). Computational psychiatry. Trends in Cognitive Sciences, 16, 72–80.
155. G.MorrisA.NevetD.ArkadirE.VaadiaH.Bergman (2006). Midbrain dopamine neurons encode decisions for future action. Nature Neuroscience, 9, 1057–1063.
156. W.MorseR.N.MeadR.Kelleher (1967). Modulation of elicited behavior by a fixed-interval schedule of electric shock presentation. Science, 157, 215–217.
157. L.MottronM.DawsonI.SoulieresB.HubertJ.Burack (2006). Enhanced perceptual functioning in autism: An update, and eight principles of autistic perception. Journal of Autism and Developmental Disorders, 36, 27–43.
158. M.MoutoussisR.P.BentallW.El-DeredyP.Dayan (2011). Bayesian modelling of jumping-to-conclusions bias in delusional patients. Cognitive Neuropsychiatry, 16, 422–447.
159. M.MoutoussisR.P.BentallJ.WilliamsP.Dayan (2008). A temporal difference account of avoidance learning. Network, 19, 137–160.
160. Y.NivN.D.DawD.JoelP.Dayan (2007). Tonic dopamine: Opportunity costs and the control of response vigor. Psychopharmacology, 191, 507–520.
161. T.J.O’DonnellN.D.GoodmanJ.B.Tenenbaum (2009). Fragment grammars: Exploring computation and reuse in language (Tech. rep.). Cambridge, MA: MIT, Computer Science and Artificial Intelligence Laboratory.
162. R.C.O’ReillyM.J.Frank (2006). Making working memory work: A computational model of learning in the prefrontal cortex and basal ganglia. Neural Computation, 18, 283–328.
163. A.R.OttoS.J.GershmanA.B.MarkmanN.D.Daw (2013). The curse of planning: Dissecting multiple reinforcement-learning systems by taxing the central executive. Psychological Science, 24, 751–761.
164. S.PecinaK.C.Berridge (2005). Hedonic hot spot in nucleus accumbens shell: Where do mu-opioids cause increased hedonic impact of sweetness?Journal of Neuroscience, 25, 11777–11786.
165. E.PellicanoD.Burr (2012). When the world becomes “too real”: A Bayesian explanation of autistic perception. Trends in Cognitive Sciences, 16, 504–510.
166. N.M.PetryW.K.BickelM.Arnett (1998). Shortened time horizons and insensitivity to future consequences in heroin addicts. Addiction, 93, 729–738.
167. B.E.PfeifferD.J.Foster (2013). Hippocampal place-cell sequences depict future paths to remembered goals. Nature, 497, 74–79.
168. D.A.PizzagalliA.L.JahnJ.P.O’Shea (2005). Toward an objective characterization of an anhedonic phenotype: A signal-detection approach. Biological Psychiatry, 57, 319–327.
169. M.L.Puterman (2005). Markov decision processes: Discrete stochastic dynamic programming. Wiley Series in Probability and Statistics. New York, NY: John Wiley.
170. C.R.RabyD.M.AlexisA.DickinsonN.S.Clayton (2007). Planning for the future by western scrub-jays. Nature, 445, 919–921.
171. A.D.Redish (2004). Addiction as a computational process gone awry. Science, 306, 1944–1947.
172. A.D.RedishS.JensenA.Johnson (2008). A unified framework for addiction: Vulnerabilities in the decision process. Behavioral and Brain Sciences, 31, 415–437; discussion 437–487.
173. A.D.RedishS.JensenA.JohnsonZ.Kurth-Nelson (2007). Reconciling reinforcement learning models with behavioral extinction and renewal: Implications for addiction, relapse, and problem gambling. Psychological Review, 114, 784–805.
174. S.M.ReynoldsK.C.Berridge (2002). Positive and negative motivation in nucleus accumbens shell: Bivalent rostrocaudal gradients for GABA-elicited eating, taste “liking”/“disliking” reactions, place preference/avoidance, and fear. Journal of Neuroscience, 22, 7308–7320.
175. M.J.F.RobinsonK.C.Berridge (2013). Instant transformation of learned repulsion into motivational “wanting.”Current Biology, 23, 282–289.
176. O.J.RobinsonD.R.CharneyC.OverstreetK.VytalC.Grillon (2012). The adaptive threat bias in anxiety: Amygdala-dorsomedial prefrontal cortex coupling and aversive amplification. NeuroImage, 60, 523–529.
177. T.E.RobinsonK.C.Berridge (1993). The neural basis of drug craving: An incentive-sensitization theory of addiction. Brain Research Reviews, 18, 247–291.
178. M.R.RoeschD.J.CaluG.Schoenbaum (2007). Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nature Neuroscience, 10, 1615–1624.
179. J.P.RoiserR.ElliottB.J.Sahakian (2012). Cognitive mechanisms of treatment in depression. Neuropsychopharmacology, 37, 117–136.
180. R.R.RozeskeA.Der-AvakianL.R.WatkinsS.F.Maier (2012). Activation of the medial prefrontal cortex by escapable stress is necessary for protection against subsequent inescapable stress-induced potentiation of morphine conditioned place preference. European Journal of Neuroscience, 35, 160–165.
181. J.D.SalamoneM.Correa (2002). Motivational views of reinforcement: Implications for understanding the behavioral functions of nucleus accumbens dopamine. Behavioural Brain Research, 137, 3–25.
182. T.Savage (2003). The grounding of motivation in artificial animals: Indices of motivational behavior. Cognitive Systems Research, 4, 23–55.
183. D.L.Schacter (2012). Adaptive constructive processes and the future of memory. American Psychologist, 67, 603–613.
184. D.L.SchacterD.R.AddisD.HassabisV.C.MartinR.N.SprengK.K.Szpunar (2012). The future of memory: Remembering, imagining, and the brain. Neuron, 76, 677–694.
185. G.SchoenbaumM.R.RoeschT.A.StalnakerY.K.Takahashi (2009). A new perspective on the role of the orbitofrontal cortex in adaptive behaviour. Nature Reviews: Neuroscience, 10, 885–892.
186. W.SchultzP.DayanP.R.Montague (1997). A neural substrate of prediction and reward. Science, 275, 1593–1599.
187. L.SchwabeM.TegenthoffO.HöffkenO.T.Wolf (2012). Simultaneous glucocorticoid and noradrenergic activity disrupts the neural basis of goal-directed action in the human brain. Journal of Neuroscience, 32, 10146–10155.
188. L.SchwabeO.T.Wolf (2009). Stress prompts habit behavior in humans. Journal of Neuroscience, 29, 7191–7198.
189. N.SchweighoferM.BertinK.ShishidaY.OkamotoS.C.TanakaS.YamawakiK.Doya (2008). Low-serotonin levels increase delayed reward discounting in humans. Journal of Neuroscience, 28, 4528–4532.
190. D.A.SimonN.D.Daw (2011). Neural correlates of forward planning in a spatial decision task in humans. Journal of Neuroscience, 31, 5526–5539.
191. A.C.P.Sims (2003). Symptoms of the mind: And introduction to descriptive psychopathology (3rd ed.). Philadelphia, PA: Saunders.
192. J.M.Smith (1993). Evolution and the theory of games. New York, NY: Springer.
193. K.S.SmithA.VirkudK.DeisserothA.M.Graybiel (2012). Reversible online control of habitual behavior by optogenetic perturbation of medial prefrontal cortex. Proceedings of the National Academy of Sciences of the United States of America, 109, 18932–18937.
194. A.StankeviciusQ.J.M.HuysA.KalraP.Series (2014). Optimism as a prior belief about the probability of future reward. PLoS Computational Biology.
195. E.E.SteinbergR.KeiflinJ.R.BoivinI.B.WittenK.DeisserothP.H.Janak (2013). A causal link between prediction errors, dopamine neurons and learning. Nature Neuroscience, 16, 966–973.
196. K.E.StephanT.BaldewegK.J.Friston (2006). Synaptic plasticity and dysconnection in schizophrenia. Biological Psychiatry, 59, 929–939.
197. K.E.StephanK.J.FristonC.D.Frith (2009). Dysconnection in schizophrenia: From abnormal synaptic plasticity to failures of self-monitoring. Schizophrenia Bulletin, 35, 509–527.
198. D.R.StrunkH.LopezR.J.DeRubeis (2006). Depressive symptoms are associated with unrealistic negative predictions of future life events. Behaviour Research and Therapy, 44, 861–882.
199. S.J.Suomi (2006). Risk, resilience, and gene–environment interactions in rhesus monkeys. Annals of the New York Academy of Sciences, 1094, 52–62.
200. R.Sutton (1991). Dyna, an integrated architecture for learning, planning and reacting. Sigart Bulletin, 2, 160–163.
201. R.S.SuttonA.G.Barto (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
202. L.-H.TaiA.M.LeeN.BenavidezA.BonciL.Wilbrecht (2012). Transient stimulation of distinct subpopulations of striatal neurons mimics changes in action value. Nature Neuroscience, 15, 1281–1289.
203. Y.K.TakahashiM.R.RoeschR.C.WilsonK.ToresonP.O’DonnellY.NivG.Schoenbaum (2011). Expectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex. Nature Neuroscience, 14, 1590–1597.
204. S.C.TanakaK.DoyaG.OkadaK.UedaY.OkamotoS.Yamawaki (2004). Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nature Neuroscience, 7, 887–893.
205. S.C.TanakaN.SchweighoferS.AsahiK.ShishidaY.OkamotoS.YamawakiK.Doya (2007). Serotonin differentially regulates short- and long-term prediction of rewards in the ventral and dorsal striatum. PLoS ONE, 2, e1333.
206. W.TimberlakeD.L.Grant (1975). Auto-shaping in rats to the presentation of another rat predicting food. Science, 190, 690–692.
207. W.TimberlakeG.WahlD.King (1982). Stimulus and response contingencies in the misbehavior of rats. Journal of Experimental Psychology: Animal Behavior Processes, 8, 62–85.
208. S.M.TomC.R.FoxC.TrepelR.A.Poldrack (2007). The neural basis of loss aversion in decision-making under risk. Science, 315, 515–518.
209. A.TomieA.S.AguadoL.A.PohoreckyD.Benjamin (1998). Ethanol induces impulsive-like responding in a delay-of-reward operant choice procedure: Impulsivity predicts autoshaping. Psychopharmacology, 139, 376–382.
210. M.T.TreadwayD.H.Zald (2011). Reconsidering anhedonia in depression: Lessons from translational neuroscience. Neuroscience and Biobehavioral Reviews, 35, 537–555.
211. K.M.TyeJ.J.MirzabekovM.R.WardenE.A.FerencziH.-C.TsaiJ.FinkelsteinK.. Deisseroth (2013). Dopamine neurons modulate neural encoding and expression of depression-related behaviour. Nature, 493, 537–541.
212. L.J.M.J.VanderschurenB.J.Everitt (2004). Drug seeking becomes compulsive after prolonged cocaine self-administration. Science, 305, 1017–1019.
213. N.D.VolkowJ.S.FowlerG.J.WangR.BalerF.Telang (2009). Imaging dopamine’s role in drug abuse and addiction. Neuropharmacology, 56(Suppl. 1), 3–8.
214. J.A.WaltzM.J.FrankB.M.RobinsonJ.M.Gold (2007). Selective reinforcement learning deficits in schizophrenia support predictions from computational models of striatal-cortical dysfunction. Biological Psychiatry, 62, 756–764.
215. M.R.WardenA.SelimbeyogluJ.J.MirzabekovM.LoK.R.ThompsonS.-Y.KimK.. Deisseroth (2012). A prefrontal cortex-brainstem neuronal projection that controls response to behavioural challenge. Nature, 492, 428–432.
216. C.J.C.H.Watkins (1989). Learning from delayed rewards. Cambridge, England: King’s College, Cambridge University.
217. Y.WeissE.H.Adelson (1998). Slow and smooth: A Bayesian theory for the combination of local motion signals in human vision. Cambridge, MA: MIT, Artificial Intelligence Laboratory.
218. D.R.WilliamsH.Williams (1969). Auto-maintenance in the pigeon: Sustained pecking despite contingent non-reinforcement. Journal of the Experimental Analysis of Behavior, 12, 511–520.
219. J.WilliamsP.Dayan (2005). Dopamine, learning, and impulsivity: A biological account of attention-deficit/hyperactivity disorder. Journal of Child and Adolescent Psychopharmacology, 15, 160–179; discussion 157–169.
220. J.WilliamsE.Taylor (2006). The evolution of hyperactivity, impulsivity and cognitive diversity. Journal of the Royal Society Interface, 3, 399–413.
221. J.M.G.Williams (1992). The psychological treatment of depression. New York, NY: Routledge.
222. J.M.G.WilliamsT.BarnhoferC.CraneD.HermanF.RaesE.WatkinsT.Dalgleish (2007). Autobiographical memory specificity and emotional disorder. Psychological Bulletin, 133, 122–148.
223. P.Willner (2005). Chronic mild stress (CMS) revisited: Consistency and behavioural-neurobiological concordance in the effects of CMS. Neuropsychobiology, 52, 90–110.
224. D.WingateC.DiukT.O’DonnellJ.TenenbaumS.Gershman (2013). Compositional policy priors. Cambridge, MA: MIT Press.
225. D.M.WolpertM.Kawato (1998). Multiple paired forward and inverse models for motor control. Neural Networks, 11, 1317–1329.
226. World Health Organization. (1990). International classification of diseases. Geneva, Switzerland: Author.
227. J.WraseF.SchlagenhaufT.KienastT.WüstenbergF.BermpohlT.KahntA.. Heinz (2007). Dysfunction of reward processing correlates with alcohol craving in detoxified alcoholics. NeuroImage, 35, 787–794.