A temporal difference account of avoidance learning

Network: Computation in Neural Systems

Volumn 19, Issue 2, 2008, Pages 137-160

  Moutoussis, Michael ^a   Bentall, Richard P ^b   Williams, Jonathan ^c   Dayan, Peter ^d  

^a Surrey   (United Kingdom)

^b BANGOR UNIVERSITY   (United Kingdom)

^c KING'S COLLEGE LONDON   (United Kingdom)

^d UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

Blockade; Conditioned avoidance response; Dopamine; Psychosis

Indexed keywords

ARTICLE; AVOIDANCE BEHAVIOR; BIOLOGICAL MODEL; BRAIN; HUMAN; PHYSIOLOGY;

AVOIDANCE LEARNING; BRAIN; HUMANS; MODELS, NEUROLOGICAL;

EID: 45749092960     PISSN: 0954898X     EISSN: 13616536     Source Type: Journal    
DOI: 10.1080/09548980802192784     Document Type: Article

References (81)

1. Aguilar MA, Minarro J, Simon VM. 2004. Morphine potentiates the impairing effects of neuroleptics on two-way active conditioned avoidance response in male mice. Progress in Neuropsychopharmacology & Biological Psychiatry 28:225-237.
2. Anisman H. 1978. Psychopharmacology of aversively motivated behavior. New York: Plenum Press. pp 1-62.
3. Bach P, Hayes SC. 2002. The use of acceptance and commitment therapy to prevent the rehospitalization of psychotic patients: a randomized controlled trial. Journal of Consulting and Clinical Psychology 70:1129-1139.
4. Bardin L, Auclair A, Kleven MS, Prinssen EP, Koek W, Newman-Tancredi A, Depoortere R. 2007. Pharmacological profiles in rats of novel antipsychotics with combined dopamine D2/serotonin 5-HT1A activity: Comparison with typical and atypical conventional antipsychotics. Behavioural Pharmacology 18:103-118.
5. Bayer HM, Glimcher PW. 2005. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47:129-141.
6. Belin D, Everitt BJ. 2008. Cocaine seeking habits depend upon dopamine-dependent serial connectivity linking the ventral with the dorsal striatum. Neuron 57:432-441.
7. Beninger R, Mason S, Phillips A, Fibiger H. 1980a. The use of conditioned suppression to evaluate the nature of neuroleptic-induced avoidance deficits. Journal of Pharmacology and Experimental Therapeutics 213:623-627.
8. Beninger R, Mason S, Phillips A, Fibiger H. 1980b. The use of extinction to investigate the nature of neuroleptic-induced avoidance deficits. Psychopharmacology (Berlin) 69:11-18.
9. Bentall RP, Kinderman P, Howard R, Blackwood N, Cummins S, Rowse G, et al. In press. Paranoid delusions in schizophrenia and depression: The transdiagnostic role of negative self-esteem. Journal of Nervous Mental Disease (in press).
10. Blundell P, Hall G, Killcross S. 2003. Preserved sensitivity to outcome value after lesions of the basolateral amygdala. Journal of Neuroscience 23:7702-7709.
11. Cagniard B, Beeler JA, Britt JP, McGehee DS, Marinelli M, Zhuang X. 2006. Dopamine scales performance in the absence of new learning. Neuron 51:541-547.
12. Christy D, Reid L. 1975. Methods of deconditioning persisting avoidance: Amphetamine and amobarbital as adjuncts to response prevention. Bulletin of the Psychonomic Society 5:175-177.
13. Cooper S, Coon K, Mejta C, Reid L. 1974. Methods of deconditioning persisting avoidance: Amphetamine chlorpromazine and chlordiazepoxide as adjuncts to response prevention. Physiological Psychology 2:519-522.
14. Corcoran R, Cummins S, Rowse G, Moore R, Blackwood N, Howard R, Kinderman P, Bentall R. 2006. Reasoning under uncertainty: Heuristic judgements in patients with persecutory delusions or depression. Psychological Medicine 36:1109-1118.
15. Coutureau E, Killcross S. 2003. Inactivation of the infralimbic prefrontal cortex reinstates goal-directed responding in overtrained rats. Behavioural Brain Research 146:167-174.
16. Daw N, Kakade S, Dayan P. 2002. Opponent interactions between serotonin and dopamine. Neural Networks 15:603-616.
17. Daw N, Niv Y, Dayan P. 2005. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience 8:1704-1711.
18. Dayan P, Balleine BW. 2002. Reward motivation and reinforcement learning. Neuron 36:285-298.
19. de Oliveira AR, Reimer AE, Brandao ML. 2006. Dopamine D2 receptor mechanisms in the expression of conditioned fear. Pharmacology, Biochemistry & Behavior 84:102-111.
20. Deakin J, Graeff F. 1991. 5-HT and mechanisms of defence. Journal of Psychopharmacology 5:305-315.
21. Forsyth J, Eifert G, Barrios V. 2006. In: Craske MG, Hermans D, Vansteenwegen D, editors. Fear and Learning: From basic processes to clinical implications. Washington, DC: American Psychological Association. pp 133-153.
22. Freeman D, Garety PA, Kuipers E. 2001. Persecutory delusions: Developing the understanding of belief maintenance and emotional distress. Psychological Medicine 31:1293-1306.
23. Goto Y, Otani S, Grace AA. 2007. The Yin and Yang of dopamine release: A new perspective. Neuropharmacology 53:583-587.
24. Grace AA. 2000. The tonic/phasic model of dopamine system regulation and its implications for understanding alcohol and psychostimulant craving. Addiction 95 (Supplement 2) S119-S128.
25. Gray J. 1987. The psychology of fear and stress. New York, NY: McGraw-Hill.
26. Gray JA, McNaughton N. 1996. The neuropsychology of anxiety: Reprise. Nebraska Symposia on Motivation 43:61-134.
27. Grossberg S. 1972. A neural theory of punishment and avoidance. Mathematical Biosciences 15:39-68.
28. Horvitz JC. 2000. Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events. Neuroscience 96:651-656.
29. Iordanova MD, Westbrook RF, Killcross AS. 2006. Dopamine activity in the nucleus accumbens modulates blocking in fear conditioning. European Journal of Neuroscience 24:3265-3270.
30. Janssen I, Hanssen M, Bak M, Bijl RV, De Graaf R, Vollenberg W, et al. 2003. Discrimination and delusional ideation. British Journal of Psychiatry 182:71-76.
31. Jensen A, Sauerberg P, Jeppesen L, Sheardown M, Swedberg M. 1999. Muscarinic receptor agonists like dopamine receptor antagonists antipsychot inhibit conditioned avoidance response in rats. Journal of Pharmacology and Experimental Therapeutics 290(2):901-907.
32. Jensen J, McIntosh AR, Crawley AP, Mikulis DJ, Remington G, Kapur S. 2003. Direct activation of the ventral striatum in anticipation of aversive stimuli. Neuron 40:1251-1257.
33. Johnson J, Li W, Li J, Klopf A. 2002. A computational model of learned avoidance behavior in a one-way avoidance experiment. Adaptive Behavior 9:91-104.
34. Johnson S, Sandrow D, Meyer B, Winters R, Miller I, Solomon D, Keitner G. 2000. Increases in manic symptoms after life events involving goal attainment. Journal of Abnormal Psychology 109:721-727.
35. Kaney S, Bowen-Jones K, Dewey ME, Bentall RP. 1997. Frequency and consensus judgements of paranoid paranoid-depressed and depressed psychiatric patients: Subjective estimates for positive negative and neutral events. British Journal of Clinical Psychology 36:349-364.
36. Killcross S, Coutureau E. 2003. Coordination of actions and habits in the medial prefrontal cortex of rats. Cerebral Cortex 13:400-408.
37. Killcross S, Robbins TW, Everitt BJ. 1997. Different types of fear-conditioned behaviour mediated by separate nuclei within amygdala. Nature 388:377-380.
38. Kim H, Shimojo S, O'Doherty JP. 2006. Is avoiding an aversive outcome rewarding? Neural substrates of avoidance learning in the human brain. PLoS Biology 4:e233.
39. Lovibond P. 2006. Fear and Avoidance: An integrated expectancy model. In: Craske MG, Hermans D, Vansteenwegen D, editors. Fear and Learning: From basic processes to clinical implications. Washington, DC: American Psychological Association. pp 117-132.
40. Ma T, Yu Q. 1993. Effect of 20(S)-ginsenoside-Rg2 and cyproheptadine on two-way active avoidance learning and memory in rats. Arzneimittelforschung 43:1049-1052.
41. McAllister D, McAllister W, Hampton S, Scoles T. 1980. Escape-from-fear performance as affected by handling method and an additional CS-shock treatment. Animal Learning Behavior 8:417-423.
42. McAllister WR, McAllister DE, Scoles MT, Hampton SR. 1986. Persistence of fear-reducing behavior: Relevance for the conditioning theory of neurosis. Journal of Abnormal Psychology 95:365-372.
43. Menon M, Jensen J, Vitcu I, Graff-Guerrero A, Crawley A, Smith MA, Kapur S. 2007. Temporal difference modeling of the blood-oxygen level dependent response during aversive conditioning in humans: Effects of dopaminergic modulation. Biological Psychiatry 62:765-772.
44. Mineka S. 1979. The role of fear in theories of avoidance learning flooding and extinction. Psychological Bulletin 86:985-1010.
45. Miyamoto J, Tsuji M, Takeda H, Ohzeki M, Nawa H, Matsumiya T. 2004. Characterization of the anxiolytic-like effects of fluvoxamine milnacipran and risperidone in mice using the conditioned fear stress paradigm. European Journal of Pharmacology 504:97-103.
46. Montague R, Dayan P, Sejnowski T. 1996. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. The Journal of Neuroscience 76:1936-1947.
47. Moutoussis M, Williams J, Dayan P, Bentall RP. 2007. Persecutory delusions and the conditioned avoidance paradigm: Towards an integration of the psychology and biology of paranoia. Cognitive Neuropsychiatry 12:495-510.
48. Mowrer O. 1947. On the dual nature of learning: A reinterpretation of conditioning and problem solving. Harvard Educational Review 17:102-148.
49. Niv Y. 2007. Cost benefit tonic phasic: What do response rates tell us about dopamine and motivation?. Annals of the New York Academy of Sciences 1104:357-376.
50. Niv Y, Daw ND, Joel D, Dayan P. 2007. Tonic dopamine: Opportunity costs and the control of response vigor. Psychopharmacology (Berlin) 191:507-520.
51. O'Doherty J, Dayan P, Schultz J, Deichmann R, Friston K, Dolan R. 2004. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304:452-454.
52. Poremba A, Gabriel M. 1995. The amygdala is necessary for the initial acquisition but not for maintenance of discriminative avoidance behavior in rabbits. Society of Neuroscience Abstracts 21:1930.
53. Poremba A, Gabriel M. 1999. Amygdala neurons mediate acquisition but not maintenance of instrumental avoidance behavior in rabbits. Journal of Neuroscience 19:9635-9641.
54. Reis FL, Masson S, de Oliveira AR, Brandao ML. 2004. Dopaminergic mechanisms in the conditioned and unconditioned fear as assessed by the two-way avoidance and light switch-off tests. Pharmacology, Biochemistry & Behavior 79:359-365.
55. Rosenkranz JA, Grace AA. 2001. Dopamine attenuates prefrontal cortical suppression of sensory inputs to the basolateral amygdala of rats. Journal of Neuroscience 21:4090-4103.
56. Rosenkranz JA, Grace AA. 2002. Cellular mechanisms of infralimbic and prelimbic prefrontal cortical inhibition and dopaminergic modulation of basolateral amygdala neurons in vivo. Journal of Neuroscience 22:324-337.
57. Samaha AN, Seeman P, Stewart J, Rajabi H, Kapur S. 2007. "Breakthrough" dopamine supersensitivity during ongoing antipsychotic treatment leads to treatment failure over time. Journal of Neuroscience 27:2979-2986.
58. Schmajuk N, Zanutto B. 1997. Escape avoidance and imitation: A neural network approach. Adaptive Behavior 6:63-129.
59. Schultz W, Apicella P, Ljungberg T. 1993. Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. Journal of Neuroscience 13:900-913.
60. Seligman M, Johnston J. 1973. Contemporary approaches to conditioning and learning Washington, DC: Winston. pp 69-110.
61. Seymour B, O'Doherty JP, Dayan P, Koltzenburg M, Jones AK, Dolan RJ, Friston KJ, Frackowiak RS. 2004. Temporal difference models describe higher-order learning in humans. Nature 429:664-667.
62. Seymour B, Singer T, Dolan R. 2007. The neurobiology of punishment. Nature Reviews Neuroscience 8:300-311.
63. Shannon H, Hart J, Bymaster F, Calligaro D, DeLapp N, Mitch C, Ward J, Fink-Jensen A, Sauerberg P, Jeppesen L, et al. 1999. Muscarinic receptor agonists like dopamine receptor antagonist antipsychotics inhibit conditioned avoidance response in rats. Journal of Pharmacology and Experimental Therapeutics 290:901-907.
64. Siuciak JA, Chapin DS, McCarthy SA, Guanowsky V, Brown J, Chiang P, Marala R, Patterson T, Seymour PA, Swick A, et al. 2007. CP-809101 a selective 5-HT2C agonist shows activity in animal models of antipsychotic activity. Neuropharmacology 52:279-290.
65. Smith A, Becker S, Kapur S. 2005. A computational model for the functional role of ventral-striatal d2 receptor in the expression of previously acquired behaviors. Neural Computation 17:361-395.
66. Smith A, Li M, Becker S, Kapur S. 2006. Dopamine prediction error and associative learning: A model-based account. Network: Computation in Neural Systems 17:61-84.
67. Smith A, Li M, Becker S, Kapur S. 2007. Linking animal models of psychosis to computational models of dopamine function. Neuropsychopharmacology 32:54-66.
68. Solomon RL, Corbit JD. 1974. An opponent-process theory of motivation: Temporal dynamics of affect. Psychological Review 81:119-145.
69. Solomon R, Wynne L. 1953. Traumatic avoidance learning: Acquisition in normal dogs. Psychological Monographs 67:19.
70. Sutton R, Barto A. 1998. Reinforcement learning: An introduction Cambridge, Massachusetts: MIT Press.
71. Titov SA, Shamakina IIu Ashmarin IP. 1983. Effect of lysyl vasopressin and vasotocin on a disorder of the conditioned avoidance reaction by a serotonin receptor blockader. Biulleten Eksperimentalnoi Biologii I Meditsiny 95:31-33.
72. Tobler PN, Dickinson A, Schultz W. 2003. Coding of predicted reward omission by dopamine neurons in a conditioned inhibition paradigm. Journal of Neuroscience 23:10402-10410.
73. Unger W, Evans I, Rourke P, Levis D. 2003. The s-s construct of expectancy versus the s-r construct of fear: Which motivates the acquisition of avoidance behaviour ? The Journal of General Psychology 130:131-147.
74. Ungless M. 2004. Dopamine: The salient issue. Trends in the Neurosciences 27:702-706.
75. Wadenberg M, Browning J, Young K, Hicks P. 2001. Antagonism at 5-HT(2A) receptors potentiates the effect of haloperidol in a conditioned avoidance response task in rats. Pharmacology, Biochemistry & Behavior 68:363-370.
76. Wadenberg M, Hicks P. 1999. The conditioned avoidance response test re-evaluated: Is it a sensitive test for the detection of potentially atypical antipsychotics ? Neuroscience and Biobehavioral Reviews 23:851-862.
77. Williams JOH, Dayan P. 2005. Dopamine, learning and impulsivity: A biological account of ADHD. Journal of Child and Adolescent Psychopathology 15:160-179.
78. Wise R, Spindler J, deWit H, Gerberg G. 1978. Neuroleptic-induced "anhedonia" in rats: Pimozide blocks reward quality of food. Science 201:262-264.
79. Wolpert D, Miall R. 1996. Forward models for physiological motor control. Neural Networks 9:1265-1279.
80. Yacubian J, Glascher J, Schroeder K, Sommer T, Braus DF, Buchel C. 2006. Dissociable systems for gain- and loss-related value predictions and errors of prediction in the human brain. Journal of Neuroscience 26:9530-9537.
81. Young AM. 2004. Increased extracellular dopamine in nucleus accumbens in response to unconditioned and conditioned aversive stimuli: Studies using 1 min microdialysis in rats. Journal of Neuroscience Methods 138:57-63.