Disentangling the roles of approach, activation and valence in instrumental and pavlovian responding

PLoS Computational Biology

Volumn 7, Issue 4, 2011, Pages

  Huys, Quentin J M ^a,b,c   Cools, Roshan ^d   Gölzer, Martin ^e   Friedel, Eva ^e   Heinz, Andreas ^e   Dolan, Raymond J ^a   Dayan, Peter ^b  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

^b UNIVERSITY COLLEGE LONDON   (United Kingdom)

^c ROYAL FREE AND UNIVERSITY COLLEGE MEDICAL SCHOOL   (United Kingdom)

^d RADBOUD UNIVERSITY NIJMEGEN   (Netherlands)

^e CHARITÉ UNIVERSITÄTSMEDIZIN BERLIN   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

REINFORCEMENT LEARNING;

COMPUTATIONAL COSTS; CRITICAL STEPS; HARD-WIRED; INCREASED FLEXIBILITY; MAIN GROUP; PERFORMANCE; REINFORCEMENT LEARNING MODELS;

CHEMICAL ACTIVATION;

ADULT; AFFECT; APPETITE; ARTICLE; AUDITORY STIMULATION; AVERSIVE BEHAVIOR; CONDITIONED REFLEX; CONTROLLED STUDY; EVOKED AUDITORY RESPONSE; EVOKED VISUAL RESPONSE; FEMALE; HUMAN; HUMAN EXPERIMENT; INSTRUMENTAL CONDITIONING; MALE; MATHEMATICAL COMPUTING; MATHEMATICAL MODEL; MOTIVATION; NORMAL HUMAN; PUNISHMENT; REINFORCEMENT; RESPONSE TIME; REWARD; SIGNAL NOISE RATIO; TASK PERFORMANCE; VISUAL STIMULATION; WITHDRAWAL REFLEX; ANALYSIS OF VARIANCE; BIOLOGY; DECISION MAKING; LEARNING; METHODOLOGY; PSYCHOLOGICAL MODEL; REFLEX;

ADULT; ANALYSIS OF VARIANCE; CHOICE BEHAVIOR; COMPUTATIONAL BIOLOGY; CONDITIONING, CLASSICAL; FEMALE; HUMANS; LEARNING; MALE; MODELS, PSYCHOLOGICAL; PUNISHMENT; REFLEX; REWARD;

EID: 79955553488     PISSN: 1553734X     EISSN: 15537358     Source Type: Journal    
DOI: 10.1371/journal.pcbi.1002028     Document Type: Article

References (110)

1. Rescorla RA, Solomon RL, (1967) Two-process learning theory: Relationships between pavlovian conditioning and instrumental learning. Psychol Rev 74: 151-182.
2. Sutton RS, Barto AG, (1998) Reinforcement Learning: An Introduction Cambridge, MA MIT Press Available: http://www.cs.ualberta.ca/sutton/book/the-book.html.
3. Bouton ME, (2006) Learning and Behavior: A Contemporary Synthesis Sinauer.
4. Dickinson A, (1980) Contemporary animal learning theory Cambridge, UK Cambridge University Press.
5. Gray JA, (1991) The psychology of fear and stress, volume 5 of Problems in the behavioural sciences Cambridge, UK Cambridge University Press, 2 edition.
6. Estes W, Skinner B, (1941) Some quantitative aspects of anxiety. J Exp Psychol 29: 390-400.
7. Dickinson A, Dearing MF, (1979) Appetitive-aversive interactions and inhibitory processes. In: Dickinson A, Boakes RA, editors. Mechanisms of learning and motivation Hillsdale, NJ Erlbaum pp. 203-231.
8. Lovibond PF, (1983) Facilitation of instrumental behavior by a Pavlovian appetitive conditioned stimulus. J Exp Psychol Anim Behav Process 9: 225-247.
9. Dayan P, Niv Y, Seymour B, Daw ND, (2006) The misbehavior of value and the discipline of the will. Neural Netw 19: 1153-1160.
10. Williams DR, Williams H, (1969) Auto-maintenance in the pigeon: sustained pecking despite contingent non-reinforcement. J Exp Anal Behav 12: 511-520.
11. Hershberger WA, (1986) An approach through the looking-glass. Anim Learn Behav 14: 443-51.
12. Ainslie G, (2001) Breakdown of will Cambridge University Press.
13. Martino BD, Kumaran D, Seymour B, Dolan RJ, (2006) Frames, biases, and rational decision-making in the human brain. Science 313: 684-687.
14. Kahneman D, Knetsch J, Thaler R, (1990) Experimental tests of the endowment effect and the Coase theorem. J Polit Econ 98: 1325.
15. Ariely D, (2008) Predictably irrational: The hidden forces that shape our decisions HarperCollins London.
16. Frank MJ, Seeberger LC, O'Reilly RC, (2004) By carrot or by stick: cognitive reinforcement learning in Parkinsonism. Science 306: 1940-3.
17. Frank MJ, Hutchison K, (2009) Genetic contributions to avoidance-based decisions: striatal D2 receptor polymorphisms. Neuroscience 164: 131-140.
18. Frank MJ, (2005) Dynamic dopamine modulation in the basal ganglia: a neurocomputational account of cognitive deficits in medicated and nonmedicated Parkinsonism. J Cogn Neurosci 17: 51-72.
19. Mazzoni P, Hristova A, Krakauer JW, (2007) Why don't we move faster? Parkinson's disease, movement vigor, and implicit motivation. J Neurosci 27: 7105-7116.
20. Drevets WC, Gautier C, Price JC, Kupfer DJ, Kinahan PE, et al. (2001) Amphetamine-induced dopamine release in human ventral striatum correlates with euphoria. Biol Psychiatry 49: 81-96.
21. Heinz A, (2002) Dopaminergic dysfunction in alcoholism and schizophrenia-psychopathological and behavioral correlates. Eur Psychiatry 17: 9-16.
22. Boileau I, Assaad JM, Pihl RO, Benkelfat C, Leyton M, et al. (2003) Alcohol promotes dopamine release in the human nucleus accumbens. Synapse 49: 226-231.
23. Robinson TE, Berridge KC, (2003) Addiction. Annu Rev Psychol 54: 25-53.
24. Huys QJM, (2007) Reinforcers and control. Towards a computational ætiology of depression. Ph.D. thesis, Gatsby Computational Neuroscience Unit, UCL, University of London. Available:http://www.gatsby.ucl.ac.uk/qhuys/pub.html.
25. Dayan P, Huys QJM, (2008) Serotonin, inhibition, and negative mood. PLoS Comput Biol 4: e4.
26. Cox SML, Benkelfat C, Dagher A, Delaney JS, Durand F, et al. (2009) Striatal dopamine responses to intranasal cocaine self-administration in humans. Biol Psychiatry 65: 846-850.
27. Bolles RC, (1970) Species-specific defense reactions and avoidance learning. Psychol Rev 77: 32-48.
28. Seligman ME, (1970) On the generality of the laws of learning. Psychol Rev 77: 406-18.
29. Wise R, (1982) Neuroleptics and operant behavior: The anhedonia hypothesis. Behav Brain Sci 5: 39-87.
30. O'Doherty JP, Dayan P, Friston K, Critchley H, Dolan RJ, (2003) Temporal difference models and reward-related learning in the human brain. Neuron 38: 329-337.
31. Knutson B, Greer SM, (2008) Anticipatory affect: neural correlates and consequences for choice. Philos Trans R Soc Lond B Biol Sci 363: 3771-3786.
32. Carter CJ, Pycock CJ, (1978) Differential effects of central serotonin manipulation on hyperactive and stereotyped behaviour. Life Sci 23: 953-960.
33. Murschall A, Hauber W, (2006) Inactivation of the ventral tegmental area abolished the general excitatory influence of pavlovian cues on instrumental performance. Learn Mem 13: 123-126.
34. Niv Y, Daw ND, Joel D, Dayan P, (2007) Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology (Berl) 191: 507-520.
35. Ikemoto S, Panksepp J, (1999) The role of nucleus accumbens dopamine in motivated behavior: a unifying interpretation with special reference to reward-seeking. Brain Res Brain Res Rev 31: 6-41.
36. Montague PR, Dayan P, Sejnowski TJ, (1996) A framework for mesencephalic dopamine systems based on predictive hebbian learning. J Neurosci 16: 1936-47.
37. Suri RE, Schultz W, (1999) A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task. Neuroscience 91: 871-890.
38. Waelti P, Dickinson A, Schultz W, (2001) Dopamine responses comply with basic assumptions of formal learning theory. Nature 412: 43-48.
39. O'Doherty J, Dayan P, Schultz J, Deichmann R, Friston K, et al. (2004) Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304: 452-4.
40. Bayer HM, Glimcher PW, (2005) Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47: 129-141.
41. Morris G, Nevet A, Arkadir D, Vaadia E, Bergman H, (2006) Midbrain dopamine neurons encode decisions for future action. Nat Neurosci 9: 1057-1063.
42. Roesch MR, Calu DJ, Schoenbaum G, (2007) Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat Neurosci 10: 1615-1624.
43. Daw ND, Kakade S, Dayan P, (2002) Opponent interactions between serotonin and dopamine. Neural Netw 15: 603-16.
44. Takase LF, Nogueira MI, Baratta M, Bland ST, Watkins LR, et al. (2004) Inescapable shock activates serotonergic neurons in all raphe nuclei of rat. Behav Brain Res 153: 233-239.
45. Takase LF, Nogueira MI, Bland ST, Baratta M, Watkins LR, et al. (2005) Effect of number of tailshocks on learned helplessness and activation of serotonergic and noradrenergic neurons in the rat. Behav Brain Res 162: 299-306.
46. Bolles R, Riley A, (1973) Freezing as an avoidance response: Another look at the operant-respondent distinction* 1. Learn Motiv 4: 268-275.
47. Soubrié P, (1986) Reconciling the role of central serotonin neurons in human and animal behaviour. Behav Brain Sci 9: 319-364.
48. Deakin JFW, Graeff FG, (1991) 5-HT and mechanisms of defence. J Psychopharmacol 5: 305-16.
49. Cools R, Robinson OJ, Sahakian B, (2008) Acute tryptophan depletion in healthy volunteers enhances punishment prediction but does not affect reward prediction. Neuropsychopharmacology 33: 2291-2299.
50. Dayan P, Huys QJM, (2009) Serotonin in affective control. Annu Rev Neurosci 32: 95-126.
51. Crockett MJ, Clark L, Robbins TW, (2009) Reconciling the role of serotonin in behavioral inhibition and aversion: acute tryptophan depletion abolishes punishment-induced inhibition in humans. J Neurosci 29: 11993-11999.
52. Boureau YL, Dayan P, (2011) Opponency revisited: competition and cooperation between dopamine and serotonin. Neuropsychopharmacology 36: 74-97.
53. Hikida T, Kimura K, Wada N, Funabiki K, Nakanishi S, (2010) Distinct roles of synaptic transmission in direct and indirect striatal pathways to reward and aversive behavior. Neuron 66: 896-907.
54. Lobo MK, Covington HE, Chaudhury D, Friedman AK, Sun H, et al. (2010) Cell type-specific loss of bdnf signaling mimics optogenetic control of cocaine reward. Science 330: 385-390.
55. Alexander G, Crutcher M, (1990) Functional architecture of basal ganglia circuits: neural substrates of parallel processing. Trends Neurosci 13: 266-271.
56. Haber SN, Knutson B, (2010) The reward circuit: linking primate anatomy and human imaging. Neuropsychopharmacology 35: 4-26.
57. Faure A, Reynolds SM, Richard JM, Berridge KC, (2008) Mesolimbic dopamine in desire and dread: enabling motivation to be generated by localized glutamate disruptions in nucleus accumbens. J Neurosci 28: 7184-7192.
58. Faure A, Richard JM, Berridge KC, (2010) Desire and dread from the nucleus accumbens: cortical glutamate and subcortical gaba differentially generate motivation and hedonic impact in the rat. PLoS One 5: e11223.
59. Bandler R, Shipley MT, (1994) Columnar organization in the midbrain periaqueductal gray: modules for emotional expression? Trends Neurosci 17: 379-389.
60. Gray JA, McNaughton N, (2003) The neuropsychology of anxiety Oxford University Press, 2nd edition.
61. Dickinson A, Pearce J, (1977) Inhibitory interactions between appetitive and aversive stimuli. Psychol Bull 84: 690-711.
62. Talmi D, Seymour B, Dayan P, Dolan RJ, (2008) Human Pavlovian-instrumental transfer. J Neurosci 28: 360-368.
63. Bray S, Rangel A, Shimojo S, Balleine B, O'Doherty JP, (2008) The neural mechanisms underlying the influence of pavlovian cues on human decision making. J Neurosci 28: 5861-5866.
64. Overmier J, Bull J, Pack K, (1971) On instrumental response interaction as explaining the influences of Pavlovian Cs+ s upon avoidance behavior. Learn Motiv 2: 103-112.
65. Hirsch S, Bolles R, (1980) On the ability of prey to recognize predators. Z Tierpsychol 54: 71-84.
66. Stephan KE, Penny WD, Daunizeau J, Moran RJ, Friston KJ, (2009) Bayesian model selection for group studies. Neuroimage 46: 1004-1017.
67. Corrado GS, Sugrue LP, Seung HS, Newsome WT, (2005) Linear-Nonlinear-Poisson models of primate choice dynamics. J Exp Anal Behav 84: 581-617.
68. Lau B, Glimcher PW, (2005) Dynamic response-by-response models of matching behavior in rhesus monkeys. J Exp Anal Behav 84: 555-579.
69. Timberlake W, Grant D, (1975) Auto-shaping in rats to the presentation of another rat predicting food. Science 190: 690-692.
70. Holland PC, (1977) Conditioned stimulus as a determinant of the form of the Pavlovian conditioned response. J Exp Psychol Anim Behav Process 3: 77-104.
71. Bushong B, King L, Camerer C, Rangel A, (2010) Pavlovian processes in consumer choice: The physical presence of a good increases willingness-to-pay. Am Econ Rev 100: 1556.
72. Dickinson A, Smith J, Mirenowicz J, (2000) Dissociation of Pavlovian and instrumental incentive learning under dopamine antagonists. Behav Neurosci 114: 468-483.
73. Cools R, Roberts AC, Robbins TW, (2008) Serotoninergic regulation of emotional and behavioural control processes. Trends Cogn Sci 12: 31-40.
74. Roelofs K, Minelli A, Mars RB, van Peer J, Toni I, (2009) On the neural control of social emotional behavior. Soc Cogn Affect Neurosci 4: 50-58.
75. Corbit LH, Balleine BW, (2005) Double dissociation of basolateral and central amygdala lesions on the general and outcome-specific forms of Pavlovian-instrumental transfer. J Neurosci 25: 962-970.
76. Tye NC, Everitt BJ, Iversen SD, (1977) 5-hydroxytryptamine and punishment. Nature 268: 741-743.
77. Lister S, Pearce JM, Butcher SP, Collard KJ, Foster GA, (1996) Acquisition of conditioned inhibition in rats is impaired by ablation of serotoninergic pathways. Eur J Neurosci 8: 415-423.
78. McNaughton N, Corr PJ, (2004) A two-dimensional neuropsychology of defense: fear/anxiety and defensive distance. Neurosci Biobehav Rev 28: 285-305.
79. Wickens JR, Budd CS, Hyland BI, Arbuthnott GW, (2007) Striatal contributions to reward and decision making: making sense of regional variations in a reiterated processing matrix. Ann N Y Acad Sci 1104: 192-212.
80. Hollis K, (1984) The biological function of Pavlovian conditioning: the best defense is a good offense. J Exp Psychol Anim Behav Process 10: 413-425.
81. Heinz A, Mann K, Weinberger DR, Goldman D, (2001) Serotonergic dysfunction, negative mood states, and response to alcohol. Alcohol Clin Exp Res 25: 487-495.
82. Horvitz JC, (2000) Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events. Neuroscience 96: 651-6.
83. Fadok JP, Dickerson TMK, Palmiter RD, (2009) Dopamine is necessary for cue-dependent fear conditioning. J Neurosci 29: 11089-11097.
84. Beninger R, Mason S, Phillips A, Fibiger H, (1980) The use of extinction to investigate the nature of neuroleptic-induced avoidance deficits. Psychopharmacology 69: 11-18.
85. Brischoux F, Chakraborty S, Brierley DI, Ungless MA, (2009) Phasic excitation of dopamine neurons in ventral vta by noxious stimuli. Proc Natl Acad Sci U S A 106: 4894-4899.
86. Canli T, Lesch KP, (2007) Long story short: the serotonin transporter in emotion regulation and social cognition. Nat Neurosci 10: 1103-1109.
87. Wyvell CL, Berridge KC, (2000) Intra-accumbens amphetamine increases the conditioned incentive salience of sucrose reward: enhancement of reward "wanting" without enhanced "liking" or response reinforcement. J Neurosci 20: 8122-8130.
88. Lex B, Hauber W, (2010) The role of nucleus accumbens dopamine in outcome encoding in instrumental and pavlovian conditioning. Neurobiol Learn Mem 93: 283-290.
89. Heinz A, Goldman D, Jones DW, Palmour R, Hommer D, et al. (2000) Genotype influences in vivo dopamine transporter availability in human striatum. Neuropsychopharmacology 22: 133-139.
90. Frank MJ, Moustafa AA, Haughey HM, Curran T, Hutchison KE, (2007) Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning. Proc Natl Acad Sci U S A 104: 16311-16316.
91. Cools R, Nakamura K, Daw ND, (2011) Serotonin and dopamine: unifying affective, activational, and decision functions. Neuropsychopharmacology 36: 98-113.
92. Frank MJ, Fossella JA, (2011) Neurogenetics and pharmacology of learning, motivation, and cognition. Neuropsychopharmacology 36: 133-152.
93. Chase HW, Frank MJ, Michael A, Bullmore ET, Sahakian BJ, et al. (2010) Approach and avoidance learning in patients with major depression and healthy controls: relation to anhedonia. Psychol Med 40: 433-440.
94. Kahnt T, Park SQ, Cohen MX, Beck A, Heinz A, et al. (2009) Dorsal striatal-midbrain connectivity in humans predicts how reinforcements are used to guide decisions. J Cogn Neurosci 21: 1332-1345.
95. Bush R, Mosteller F, (1955) Stochastic Models for Learning New York, NY John Wiley & Sons.
96. Narendra KS, Thathachar MAL, (1974) Learning automata - A survey. IEEE Trans Syst Man Cybern 4: 323-334.
97. Tsypkin Y, Poznyak A, (1972) Finite learning automata. Eng Cybern 10: 478-490.
98. Killcross S, Coutureau E, (2003) Coordination of actions and habits in the medial prefrontal cortex of rats. Cereb Cortex 13: 400-408.
99. Daw ND, Niv Y, Dayan P, (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 8: 1704-1711.
100. Balleine BW, (2005) Neural bases of food-seeking: affect, arousal and reward in corticostriatolimbic circuits. Physiol Behav 86: 717-730.
101. Hall J, Parkinson JA, Connor TM, Dickinson A, Everitt BJ, (2001) Involvement of the central nucleus of the amygdala and nucleus accumbens core in mediating Pavlovian influences on instrumental behaviour. Eur J Neurosci 13: 1984-1992.
102. Garcia J, Koelling R, (1996) Relation of cue to consequence in avoidance learning. In: Houck LD, Drickamer LC, editors. Foundations of animal behavior: classic papers with commentaries Chicago University of Chicago Press.
103. Beck A, Steer R, Brown G, (1996) Manual for the Beck Depression Inventory-II San Antonio, TX Psychological Corporation.
104. Beck AT, Epstein N, Brown G, Steer RA, (1988) An inventory for measuring clinical anxiety: Psychometric properties. J Consult Clin Psychol 56: 893-897.
105. Spielberger C, Gorsuch R, (1970) STAI manual for the State-trait anxiety inventory (" self-evaluation questionnaire") Consulting Psychologists Press.
106. Montgomery S, Asberg M, (1979) A new depression scale designed to be sensitive to change. Br J Psychiatry 134: 382.
107. Hamilton M, (1960) A rating scale for depression. J Neurol Neurosurg Psychiatry 23: 56-62.
108. Shear MK, Bilt JV, Rucci P, Endicott J, Lydiard B, et al. (2001) Reliability and validity of a structured interview guide for the hamilton anxiety rating scale (sigh-a). Depress Anxiety 13: 166-178.
109. MacKay DJ, (2003) Information theory, inference and learning algorithms Cambridge, UK Cambridge University Press.
110. Kass R, Raftery A, (1995) Bayes factors. J Am Stat Assoc 90.