A Bayesian formulation of behavioral control

Cognition

Volumn 113, Issue 3, 2009, Pages 314-328

  Huys, Quentin J M ^a,b   Dayan, Peter ^a  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

^b Och Spine at New York Presbyterian Hospitals   (United States)

Author keywords

Animal behavior; Animal models of depression; Bayesian; Computational; Computational psychiatry; Controllability; Depression; Helplessness; Learned helplessness; Reinforcement learning

Indexed keywords

ANIMAL BEHAVIOR; ARTICLE; ATTENTION; BAYES THEOREM; BAYESIAN LEARNING; BEHAVIOR CONTROL; DEPRESSION; ENTROPY; ENVIRONMENT; LEARNING; NONHUMAN; PRIORITY JOURNAL; REINFORCEMENT; REWARD;

ANIMALS; BEHAVIOR, ANIMAL; DEPRESSION; ENVIRONMENT; HELPLESSNESS, LEARNED; HUMANS; MODELS, PSYCHOLOGICAL; REINFORCEMENT (PSYCHOLOGY); SOCIAL ENVIRONMENT;

EID: 70350570499     PISSN: 00100277     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.cognition.2009.01.008     Document Type: Article

References (101)

1. Abramson L.Y., Alloy L.B., Hogan M.E., Whitehouse W.G., Cornette M., Akhavan S., et al. Suicidality and cognitive vulnerability to depression among college students: A prospective study. Journal of Adolescence 21 4 (1998) 473-487
2. Abramson L.Y., Metalsky G.I., and Alloy L.B. Judgment of contingency in depressed and nondepressed students: Sadder but wiser?. Journal of Experimental Psychology: General 108 4 (1979) 441-485
3. Abramson L.Y., Metalsky G.I., and Alloy L.B. Hopelessness depression: A theory-based subtype of depression. Psychological Review 96 2 (1989) 358-372
4. Abramson L.Y., Seligman M.E., and Teasdale J.D. Learned helplessness in humans: Critique and reformulation. Journal of Abnormal Psychology 87 1 (1978) 49-74
5. Alloy L.B., and Abramson L.Y. Learned helplessness, depression, and the illusion of control. Journal of Personality and Social Psychology 42 6 (1982) 1114-1126
6. Alloy L.B., Abramson L.Y., Whitehouse W.G., Hogan M.E., Tashman N.A., Steinberg D.L., et al. Depressogenic cognitive styles: Predictive validity, information processing and personality characteristics, and developmental origins. Behaviour Research and Therapy 37 6 (1999) 503-531
7. Alloy L.B., and Tabachnik N. Assessment of covariation by humans and animals: The joint influence of prior expectations and current situational information. Psychological Review 91 1 (1984) 112-149
8. Amat J., Baratta M.V., Paul E., Bland S.T., Watkins L.R., and Maier S.F. Medial prefrontal cortex determines how stressor controllability affects behavior and dorsal raphe nucleus. Nature Neuroscience 8 3 (2005) 365-371
9. Anisman H., and Matheson K. Stress, depression, and anhedonia: Caveats concerning animal models. Neuroscience and Biobehavioral Reviews 29 4-5 (2005) 525-546
10. Balleine B.W., and Dickinson A. Goal-directed instrumental action: Contingency and incentive learning and their cortical substrates. Neuropharmacology 37 4-5 (1998) 407-419
11. Beck A.T. Depression: Clinical, experimental and theoretical aspects (1967), Harper & Row, New York
12. Beck A.T. Cognitive models of depression. Journal of Cognitive Psychotherapy, An International Quarterly 1 (1987) 5-37
13. Beck A.T., Rush A.J., Shaw B.F., and Emery G. Cognitive therapy of depression. 1st ed. The Guilford clinical psychology and psychotherapy series (1979), Guilford Press
14. Bertsekas D.P., and Tsitsiklis J.N. Neuro-dynamic programming (1996), Athena Scientific
15. Blaney P.H. Contemporary theories of depression: Critique and comparison. Journal of Abnormal Psychology 86 3 (1977) 203-223
16. Bolles R.C. Species-specific defense reactions and avoidance learning. Psychological Review 77 (1970) 32-48
17. Bouton M.E. Learning and behavior: A contemporary synthesis (2006), Sinauer
18. Brickman P., Coates D., and Janoff-Bulman R. Lottery winners and accident victims: Is happiness relative?. Journal of Personality and Social Psychology 36 8 (1978) 917-927
19. Buchwald A.M., Coyne J.C., and Cole C.S. A critical evaluation of the learned helplessness model of depression. Journal of Abnormal Psychology 87 1 (1978) 180-193
20. Cabib S., and Puglisi-Allegra S. Stress, depression and the mesolimbic dopamine system. Psychopharmacology (Berl) 128 4 (1996) 331-342
21. Caspi A., Sugden K., Moffitt T.E., Taylor A., Craig I.W., Harrington W., et al. Influence of life stress on depression: Moderation by a polymorphism in the 5-HTt gene. Science 301 (2003) 386-389
22. Cohen J.D., Braver T.S., and O'Reilly R.C. A computational approach to prefrontal cortex, cognitive control and schizophrenia: Recent developments and current challenges. Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences 351 1346 (1996) 1515-1527
23. Cordero M.I., Venero C., Kruyt N.D., and Sandi C. Prior exposure to a single stress session facilitates subsequent contextual fear conditioning in rats. Evidence for a role of corticosterone. Hormones and Behavior 44 (2003) 338-345
24. Costello C.G. A critical review of seligman's laboratory experiments on learned helplessness and depression in humans. Journal of Abnormal Psychology 87 1 (1978) 21-31
25. Daw N.D., Kakade S., and Dayan P. Opponent interactions between serotonin and dopamine. Neural Networks 15 (2002) 603-616
26. Daw N.D., Niv Y., and Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience 8 12 (2005) 1704-1711
27. Dayan, P., & Huys, Q. J. M. (in press). Serotonin in affective control. Annual Reviews in Neuroscience.
28. Dayan P., and Huys Q.J.M. Serotonin, inhibition, and negative mood. PLoS Computational Biology 4 2 (2008) e4
29. Dayan P., Niv Y., Seymour B., and Daw N.D. The misbehavior of value and the discipline of the will. Neural Networks 19 8 (2006) 1153-1160
30. Dayan P., and Sejnowski T. Exploration bonuses and dual control. Machine Learning 25 (1996) 5-22
31. Dearden, R., Friedman, N., & Russell, S. (1998). Bayesian Q-learning. In Proceedings of the fifteenth national conference on artificial intelligence (pp. 761-768).
32. Dearden, R., Friedman, N., & Andre, D. (1999). Model-based Bayesian exploration. In Proceedings of the fifteenth conference on uncertainty in artificial intelligence (pp. 150-159). Stockholm.
33. Dickinson A., and Balleine B. The role of learning in the operation of motivational systems. In: Gallistel R. (Ed). Stevens' handbook of experimental psychology Vol. 3 (2002), Wiley, New York 497-534
34. Dickinson A., and Pearce J. Inhibitory interactions between appetitive and aversive stimuli. Psychological Bulletin 84 (1977) 690-711
35. Doya K. Metalearning and neuromodulation. Neural Networks 15 4-6 (2002) 495-506
36. Dulawa S.C., and Hen R. Recent advances in animal models of chronic antidepressant effects: The novelty-induced hypophagia test. Neuroscience and Biobehavioral Reviews 29 4-5 (2005) 771-783
37. Frazer A., and Morilak D.A. What should animal models of depression model?. Neuroscience and Biobehavioral Reviews 29 (2005) 5150523
38. Friedman N., and Singer Y. Efficient Bayesian parameter estimation in large discrete domains. In: Solla S.A., Leen T.K., and Müller K.-R. (Eds). Advances in neural information processing systems Vol. 11 (1999), MIT Press
39. Gambarana C., Masi F., Tagliamonte A., Scheggi S., Ghiglieri O., and De Montis M.G. A chronic stress that impairs reactivity in rats also decreases dopaminergic transmission in the nucleus accumbens. Journal of Neurochemistry 72 5 (1999) 2039-2046
40. Gardner, J., & Oswald, A. (2001). Does money buy happiness? A longitudinal study using data on windfalls. Technical report. University of Warwick; .
41. Gittins J.C. Multi-armed bandit allocation indices. Wiley interscience series in systems and optimization (1989), John Wiley & Sons Inc
42. Goodkin F. Rats learn the relationship between responding and environmental events: An expansion of the learned helplessness hypothesis. Learning and Motivation 7 (1976) 382-393
43. Horvitz J.C. Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events. Neuroscience 96 4 (2000) 651-656
44. Huys, Q. J. M. (2007). Reinforcers and control. Towards a computational ætiology of depression. Ph.D. Thesis. Gatsby Computational Neuroscience Unit, UCL, University of London.
45. Huys Q.J.M., Vogelstein J., and Dayan P. Psychiatry: Insights into depression through normative decision-making models. In: Koller D., Schuurmans D., Bengio Y., and Bottou L. (Eds). Advances in neural information processing systems Vol. 21 (2009), MIT Press
46. Jackson R.L., Maier S.F., and Rapaport P.M. Exposure to inescapable shock produces both activity and associative deficits in the rat. Learning and Motivation 9 (1978) 69-98
47. Job R.F.S. The effects of uncontrollable, unpredictable aversive and appetitive events: Similar effects warrant similar, but not identical, explanations?. Integrative Physiological and Behavioral Science: The Official Journal of the Pavlovian Society 37 1 (2002) 59-81
48. Kendler K.S., Gardner C.O., and Prescott C.A. Toward a comprehensive developmental model for major depression in women. The American Journal of Psychiatry 159 7 (2002) 1133-1145
49. Kendler K.S., Hettema J.M., Butera F., Gardner C.O., and Prescott C.A. Life event dimensions of loss, humiliation, entrapment, and danger in the prediction of onsets of major depression and generalized anxiety. Archives of General Psychiatry 60 8 (2003) 789-796
50. Kendler K.S., Kessler R.C., Walters E.E., MacLean C., Neale M.C., Heath A.C., et al. Stressful life events, genetic liability, and onset of an episode of major depression in women. The American Journal of Psychiatry 152 6 (1995) 833-842
51. Klopf A., Weaver S., and Morgan J. A hierarchical network of control systems that learn: Modeling nervous system function during classical and instrumental conditioning. Adaptive Behavior 1 3 (1993) 263-319
52. Lefcourt H. Locus of control: Current trends in theory and research (1982), Lawrence Erlbaum Associates
53. Lewinsohn P., Youngren M., and Grosscup S. Reinforcement and depression. In: Depue R.A. (Ed). The psychobiology of depressive disorders: Implications for the effects of stress (1979), Academic Press, New York 291-316
54. Maier S., and Seligman M. Learned helplessness: Theory and evidence. Journal of Experimental Psychology: General 105 1 (1976) 3-46
55. Maier S.F., and Watkins L.R. Stressor controllability and learned helplessness: The roles of the dorsal raphe nucleus, serotonin, and corticotropin-releasing factor. Neuroscience and Biobehavioral Reviews 29 4-5 (2005) 829-841
56. Mayberg H., Lozano A., Voon V., McNeely H., Seminowicz D., Hamani C., et al. Deep brain stimulation for treatment-resistant depression. Neuron 45 5 (2005) 651-660
57. Miller W.R., and Seligman M.E. Depression and learned helplessness in man. Journal of Abnormal Psychology 84 3 (1975) 228-238
58. Mineka S., and Hendersen R.W. Controllability and predictability in acquired motivation. Annual Review of Psychology 36 (1985) 495-529
59. Mitra R., Jadhav S., McEwen B.S., Vyas A., and Chattarji S. Stress duration modulates the spatiotemporal patterns of spine formation in the basolateral amygdala. Proceedings of the National Academy of Sciences of the United States of America 102 26 (2005) 9371-9376
60. Montague P.R., Dayan P., and Sejnowski T.J. A framework for mesencephalic dopamine systems based on predictive hebbian learning. Journal of Neuroscience 16 5 (1996) 1936-1947
61. Moore B. Principal component analysis in linear systems: Controllability, observability, and model reduction. Automatic Control, IEEE Transactions on 26 1 (1981) 17-32
62. Moutoussis M., Bentall R.P., Williams J., and Dayan P. A temporal difference account of avoidance learning. Network 19 2 (2008) 137-160
63. Mowrer O. On the dual nature of learning: A reinterpretation of conditionin and problem-solving. Harvard Educational Review 17 2 (1947) 102-150
64. Msetfi R.M., Murphy R.A., Simpson J., and Kornbrot D.E. Depressive realism and outcome density bias in contingency judgments: The effect of the context and intertrial interval. Journal of Experimental Psychology: General 134 1 (2005) 10-22
65. Muscat R., and Willner P. Suppression of sucrose drinking by chronic mild unpredictable stress: A methodological analysis. Neuroscience and Biobehavioral Reviews 16 4 (1992) 507-517
66. Nesse R.M. Is depression and adaptation?. Archives of General Psychiatry 57 (2000) 14-20
67. Niv Y., Daw N., and Dayan P. How fast to work: Response vigor, motivation and tonic dopamine. Advances in neural information processing (2005), MIT Press 1019-1026
68. Niv Y., Daw N.D., Joel D., and Dayan P. Tonic dopamine: Opportunity costs and the control of response vigor. Psychopharmacology (Berl) 191 3 (2007) 507-520
69. Overmier J.B., Patterson J., and Wielkiewicz R.M. Environmental contingencies as sources of stress in animals. In: Levine S., and Ursin H. (Eds). Coping and health (1980), Plenum Press
70. Overmier J.B., and Seligman M.E. Effects of inescapable shock upon subsequent escape and avoidance responding. Journal of Comparative and Physiological Psychology 63 1 (1967) 28-33
71. Parker G., and Hadzi-Pavlovic D. Melancholia: A disorder of movement and mood (1996), Cambridge University Press
72. Peterson C., Maier S.F., and Seligman M.E.P. Learned helplessness: A theory for the age of personal control (1993), OUP, Oxford, UK
73. Rottenberg J., Gross J.J., and Gotlib I.H. Emotion context insensitivity in major depressive disorder. Journal of Abnormal Psychology 114 4 (2005) 627-639
74. Rottenberg J., Kasch K.L., Gross J.J., and Gotlib I.H. Sadness and amusement reactivity differentially predict concurrent and prospective functioning in major depressive disorder. Emotion 2 2 (2002) 135-146
75. Schmajuk N., and Zanutto B. Escape, avoidance, and imitation: A neural network approach. Adaptive Behavior 6 1 (1997) 63
76. Schultz W. Predictive reward signal of dopamine neurons. Journal of Neurophysiology 80 1 (1998) 1-27
77. Seligman M.E.P. Helplessness on depression, development and death (1975), W.H. Freeman & Co, San Francisco, USA
78. Seligman M.E., and Maier S.F. Failure to escape traumatic shock. Journal of Experimental Psychology: General 74 1 (1967) 1-9
79. Servan-Schreiber D., Printz H., and Cohen J.D. A network model of catecholamine effects: Gain, signal-to-noise ratio, and behavior. Science 249 4971 (1990) 892-895
80. Smith A.J., Becker S., and Kapur S. A computational model of the functional role of the ventral-striatal d2 receptor in the expression of previously acquired behaviors. Neural Computation 17 2 (2005) 361-395
81. Smith A., Li M., Becker S., and Kapur S. A model of antipsychotic action in conditioned avoidance: A computational approach. Neuropsychopharmacology 29 6 (2004) 1040-1049
82. Smith A., Li M., Becker S., and Kapur S. Dopamine, prediction error and associative learning: A model-based account. Network 17 1 (2006) 61-84
83. Stevens A., and Price J. Evolutionary psychiatry. A new beginning. 2nd ed. (2000), Routledge, London, UK
84. Strekalova T., Spanagel R., Bartsch D., Henn F.A., and Gass P. Stress-induced anhedonia in mice is associated with deficits in forced swimming and exploration. Neuropsychopharmacology 29 11 (2004) 2007-2011
85. Strens, M. (2000). A Bayesian framework for reinforcement learning. In Proceeedings of the 17th international conference on machine learning (ICML).
86. Sutton R. Dyna, an integrated architecture for learning, planning and reacting. Sigart Bulletin 2 (1991) 160-163
87. Sutton R.S., and Barto A.G. Reinforcement learning: An introduction (1998), MIT Press, Cambridge, MA
88. Wichers M., Myin-Germeys I., Jacobs N., Peeters F., Kenis G., Derom C., et al. Genetic risk of depression and stress-induced negative affect in daily life. The British Journal of Psychiatry: The Journal of Mental Science 191 (2007) 218-223
89. Williams J.M.G. The psychological treatment of depression (1992), Routledge
90. Williams J., and Dayan P. Dopamine, learning, and impulsivity: A biological account of attention-deficit/hyperactivity disorder. Journal of Child and Adolescent Psychopharmacology 15 2 (2005) 160-179 Discussion 157-159
91. Willner P. Dopamine and depression: A review of recent evidence. I. Empirical studies. Brain Research Reviews 287 3 (1983) 211-224
92. Willner P. Antidepressants and serotonergic neurotransmission: An integrative review. Psychopharmacology (Berl) 85 4 (1985) 387-404
93. Willner P. Depression: A psychobiological synthesis (1985), John Wiley & Sons, New York
94. Willner P. Validation criteria for animal models of human mental disorders: Learned helplessness as a paradigm case. Progress in Neuro-Psychopharmacology & Biological Psychiatry 10 6 (1986) 677-690
95. Willner P. Animal models of depression: Validity and applications. Advances in Biochemical Psychopharmacology 49 (1995) 19-41
96. Willner P. Validity, reliability and utility of the chronic mild stress model of depression: A 10-year review and evaluation. Psychopharmacology (Berl) 134 (1997) 319-329
97. Willner P., and Mitchell P.J. The validity of animal models of predisposition to depression. Behavioural Pharmacology 13 3 (2002) 169-188
98. Willner P., and Mitchell P.J. Animal models of subtypes of depression. In: Kasper S., den Boer J.A., and Sitsen J.M.A. (Eds). Handbook of depression and anxiety. 2nd ed. (2003), Marcel Dekker 505-544
99. Yu A.J., and Dayan P. Uncertainty, neuromodulation, and attention. Neuron 46 4 (2005) 681-692
100. Zacharko R.M., and Anisman H. Stressor-induced anhedonia in the mesocorticolimbic system. Neuroscience and Biobehavioral Reviews 15 3 (1991) 391-405
101. Zacharko R.M., Bowers W.J., Kokkinidis L., and Anisman H. Region-specific reductions of intracranial self-stimulation after uncontrollable stress: Possible effects on reward processes. Behavioural Brain Research 9 2 (1983) 129-141