Adaptive properties of differential learning rates for positive and negative outcomes

Biological Cybernetics

Volumn 107, Issue 6, 2013, Pages 711-719

  Cazé, Romain D ^a   Van Der Meer, Matthijs A A ^b  

^a IMPERIAL COLLEGE LONDON   (United Kingdom)

^b UNIVERSITY OF WATERLOO   (Canada)

Author keywords

Basal ganglia; Decision making; Meta learning; Reinforcement learning; Reward prediction error

Indexed keywords

ASYMMETRIC LEARNING; BASAL GANGLIA; COGNITIVE SCIENCE; METALEARNING; PREDICTION ERRORS; REWARD-PREDICTION ERROR; SUB-OPTIMAL CHOICES; SUPPORT LEARNING;

DECISION MAKING; ERRORS; LEARNING ALGORITHMS; REINFORCEMENT LEARNING;

FORECASTING;

ADAPTATION; ALGORITHM; ARTICLE; BIOLOGICAL MODEL; DECISION MAKING; HUMAN; LEARNING; PHYSIOLOGY; PREDICTIVE VALUE; REINFORCEMENT;

ADAPTATION, PHYSIOLOGICAL; ALGORITHMS; CHOICE BEHAVIOR; HUMANS; LEARNING; MODELS, BIOLOGICAL; PREDICTIVE VALUE OF TESTS; REINFORCEMENT (PSYCHOLOGY);

EID: 84890950743     PISSN: 03401200     EISSN: 14320770     Source Type: Journal    
DOI: 10.1007/s00422-013-0571-5     Document Type: Article

References (41)

1. Behrens TEJ, Woolrich MW, Walton ME, Rushworth MFS (2007) Learning the value of information in an uncertain world. Nat Neurosci 10(9):1214-1221
2. Bromberg-Martin ES, Matsumoto M, Hikosaka O (2010) Dopamine in motivational control: rewarding, aversive, and alerting. Neuron 68(5):815-834
3. Cavanagh JF, Frank MJ (2011) Social stress reactivity alters reward and punishment learning. Soc Cogn Affect Neurosci 6(3):311-320
4. Chase HW, Clark L (2010) Gambling severity predicts midbrain response to near-miss outcomes. J Neurosci 30(18):6180-6187
5. D'Acremont M, Bossaerts P (2008) Neurobiological studies of risk assessment: a comparison of expected utility and mean-variance approaches. Cogn Affect Behav Neurosci 8(4):363-374
6. Redish AD, Jensen S, Johnson A, Kurth-Nelson Z (2007) Reconciling reinforcement learning models with behavioral extinction and renewal: implications for addiction, relapse, and problem gambling. Psychol Rev 114(3):784-805
7. Daw ND, O'Doherty JP, Dayan P, Seymour B, Dolan RJ (2006) Cortical substrates for exploratory decisions in humans. Nature 441(7095):876-879
8. Daw ND, Gershman SJ, Seymour B, Dayan P, Dolan RJ (2011) Model-based influences on humans' choices and striatal prediction errors. Neuron 69(6):1204-1215
9. Dayan P, Niv Y (2008) Reinforcement learning: the good, the bad and the ugly. Curr Opin Neurobiol 18(2):185-196
10. Doll BB, Jacobs WJ, Sanfey AG, Frank MJ (2009) Instructional control of reinforcement learning: a behavioral and neurocomputational investigation. Brain Res 1299:74-94
11. Doya K (2002) Metalearning and neuromodulation. Neural Netw 15(4-6):495-506
12. Doya K, Samejima K, Katagiri K, Kawato M (2002) Multiple model-based reinforcement learning. Neural Comput 14(6):1347-1369
13. Fiorillo CD (2013) Two dimensions of value: dopamine neurons represent reward but not aversiveness. Science 341(6145):546-549
14. Frank MJ, Seeberger LC, O'reilly RC (2004) By carrot or by stick: cognitive reinforcement learning in parkinsonism. Science 306(5703):1940-1943
15. Frank MJ, Moustafa AA, Haughey HM, Curran T, Hutchison KE (2007) Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning. Proc Natl Acad Sci 104(41):16311-16316
16. Frank MJ, Doll BB, Oas-Terpstra J, Moreno F (2009) Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation. Nat Neurosci 12(8):1062-1068
17. Gerfen CR, Engber TM, Mahan LC, Susel Z, Chase TN, Monsma FJ Jr, Sibley DR (1990) $$\text{ D }-1$$ D 1 and $$\text{ D }-2$$ D 2 dopamine receptor-regulated gene expression of striatonigral and striatopallidal neurons. Science 250:1429-1432
18. Gershman SJ, Niv Y (2010) Learning latent structure: carving nature at its joints. Curr Opin Neurobiol 20(2):251-256
19. Grace AA (2012) Dopamine system dysregulation by the hippocampus: implications for the pathophysiology and treatment of schizophrenia. Neuropharmacology 62(3):1342-1348
20. Humphries MD, Khamassi M, Gurney K (2012) Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia. Front Neurosci 6(February):9
21. Kahneman D, Tversky A (1979) Prospect theory: an analysis of decision under risk. Econ J Econ Soc 47(2):263-292
22. Khamassi M, Lallée S, Enel P, Procyk E, Dominey PF (2011) Robot cognitive control with a neurophysiologically inspired reinforcement learning model. Front Neurorobotic 5(July):1
23. Khamassi M, Enel P, Dominey PF, Procyk E (2013) Medial prefrontal cortex and the adaptive regulation of reinforcement learning parameters. Prog Brain Res 202:441-464
24. Kravitz AV, Tye LD, Kreitzer AC (2012) Distinct roles for direct and indirect pathway striatal neurons in reinforcement. Nat Neurosci 15:816-818
25. Kurth-Nelson Z, Redish AD (2009) Temporal-difference reinforcement learning with distributed representations. PLoS One 4(10):e7362
26. Maia TV, Frank MJ (2011) From reinforcement learning models to psychiatric and neurological disorders. Nat Neurosci 14(2):154-162
27. Mihatsch O, Neuneier R (2002) Risk-sensitive reinforcement learning. Mach Learn 49:267-290
28. Niv Y, Duff MO, Dayan P (2005) Dopamine, uncertainty and TD learning. Behav Brain Funct 1:6
29. Niv Y, Daw ND, Joel D, Dayan P (2007) Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology 191(3):507-520
30. O'Doherty JP, Hampton A, Kim H (2007) Model-based fMRI and its application to reward learning and decision making. Ann NY Acad Sci 1104:35-53
31. Redish AD (2004) Addiction as a computational process gone awry. Science 306(5703):1944-1947
32. Schultz W (2006) Behavioral theories and the neurophysiology of reward. Annu Rev Psychol 57:87-115
33. Schweighofer N, Doya K (2003) Meta-learning in reinforcement learning. Neural Netw 16(1):5-9
34. Sharot T (2011) The optimism bias. Curr Biol 21(23):R941-R945
35. Sharot T, Korn CW, Dolan RJ (2011) How unrealistic optimism is maintained in the face of reality. Nat Neurosci 14(11):1475-1479
36. Shenhav A, Botvinick MM, Cohen JD (2013) The expected value of control: an integrative theory of anterior cingulate cortex function. Neuron 79(2):217-240
37. Sutton RS (1984) Temporal credit assignment in reinforcement learning. Doctoral Dissertation, UMass Amherst
38. Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. MIT Press, Cambridge, MA
39. van der Meer M, Kurth-Nelson Z, Redish AD (2012) Information processing in decision-making systems. Neuroscientist 18(4):342-359
40. Watkins C (1989) Learning from delayed rewards. PhD thesis
41. Yu AJ (2007) Adaptive behavior: humans act as bayesian learners. Curr Biol 17(22):R977-R980