An extended Reinforcement Learning model of basal ganglia to understand the contributions of serotonin and dopamine in risk-based decision making, reward prediction, and punishment learning

Frontiers in Computational Neuroscience

Volumn 8, Issue 1 APR, 2014, Pages

  Balasubramani, Pragathi P ^a   Chakravarthy, V Srinivasa ^a   Ravindran, Balaraman ^a   Moustafa, Ahmed A ^b  

^a INDIAN INSTITUTE OF TECHNOLOGY MADRAS   (India)

^b WESTERN SYDNEY UNIVERSITY   (Australia)

Author keywords

Basal ganglia; Decision making; Dopamine; Punishment; Reinforcement Learning; Reward; Risk; Serotonin

Indexed keywords

AMINES; FORECASTING; NEUROPHYSIOLOGY; REINFORCEMENT LEARNING; RISK ASSESSMENT; RISKS;

BASAL GANGLIA; DOPAMINE; PUNISHMENT; REWARD; SEROTONIN;

DECISION MAKING;

DOPAMINE; SEROTONIN;

ARTICLE; BASAL GANGLION; DECISION MAKING; HUMAN; PROTEIN FUNCTION; PUNISHMENT; REINFORCEMENT; REWARD; RISK ASSESSMENT; RISK FACTOR; SENSITIVITY ANALYSIS; SIGNAL TRANSDUCTION; SIMULATION; STIMULUS RESPONSE;

EID: 84898896943     PISSN: None     EISSN: 16625188     Source Type: Journal    
DOI: 10.3389/fncom.2014.00047     Document Type: Article

References (70)

1. Acheson, A., Farrar, A. M., Patak, M., Hausknecht, K. A., Kieres, A. K., Choi, S., et al. (2006). Nucleus accumbens lesions decrease sensitivity to rapid changes in the delay to reinforcement. Behav. Brain Res. 173, 217-228. doi: 10. 1016/j. bbr. 2006. 06. 024.
2. Alex, K. D., and Pehek, E. A. (2007). Pharmacologic mechanisms of serotonergic regulation of dopamine neurotransmission. Pharmacol. Ther. 113, 296-320. doi: 10. 1016/j. pharmthera. 2006. 08. 004.
3. Angiolillo, P. J., and Vanderkooi, J. M. (1996). Hydrogen atoms are produced when tryptophan within a protein is irradiated with ultraviolet light. Photochem. Photobiol. 64, 492-495. doi: 10. 1111/j. 1751-1097. 1996. tb03095. x.
4. Azmitia, E. C. (1999). Serotonin neurons, neuroplasticity, and homeostasis of neural tissue. Neuropsychopharmacology 21, 33S-45S. doi: 10. 1016/S0893-133X(99)00022-6.
5. Azmitia, E. C. (2001). Modern views on an ancient chemical: serotonin effects on cell proliferation, maturation, and apoptosis. Brain Res. Bull. 56, 413-424. doi: 10. 1016/S0361-9230(01)00614-1.
6. Bell, D. E. (1995). Risk, return and utility. Manage. Sci. 41, 23-30. doi: 10. 1287/mnsc. 41. 1. 23.
7. Bertler, A., and Rosengren, E. (1966). Possible role of brain dopamine. Pharmacol. Rev. 18, 769-773.
8. Bertran-Gonzalez, J., Bosch, C., Maroteaux, M., Matamales, M., Herve, D., Valjent, E., et al. (2008). Opposing patterns of signaling activation in dopamine D1 and D2 receptor-expressing striatal neurons in response to cocaine and haloperidol. J. Neurosci. 28, 5671-5685. doi: 10. 1523/JNEUROSCI. 1039-08. 2008.
9. Bodi, N., Keri, S., Nagy, H., Moustafa, A., Myers, C. E., Daw, N., et al. (2009). Reward-learning and the novelty-seeking personality: a between-and within-subjects study of the effects of dopamine agonists on young Parkinson's patients. Brain 132, 2385-2395. doi: 10. 1093/brain/awp094.
10. Boureau, Y. L., and Dayan, P. (2011). Opponency revisited: competition and cooperation between dopamine and serotonin. Neuropsychopharmacology 36, 74-97. doi: 10. 1038/npp. 2010. 151.
11. Chakravarthy, V. S., Joseph, D., and Bapi, R. S. (2010). What do the basal ganglia do? A modeling perspective. Biol. Cybern. 103, 237-253. doi: 10. 1007/s00422-010-0401-y.
12. Chao, M. Y., Komatsu, H., Fukuto, H. S., Dionne, H. M., and Hart, A. C. (2004). Feeding status and serotonin rapidly and reversibly modulate a Caenorhabditis elegans chemosensory circuit. Proc. Natl. Acad. Sci. U. S. A. 101, 15512-15517. doi: 10. 1073/pnas. 0403369101.
13. Cools, R., Nakamura, K., and Daw, N. D. (2011). Serotonin and dopamine: unifying affective, activational, and decision functions. Neuropsychopharmacology 36, 98-113. doi: 10. 1038/npp. 2010. 121.
14. Cools, R., Robinson, O. J., and Sahakian, B. (2008). Acute tryptophan depletion in healthy volunteers enhances punishment prediction but does not affect reward prediction. Neuropsychopharmacology 33, 2291-2299. doi: 10. 1038/sj. npp. 1301598.
15. D'acremont, M., Lu, Z. L., Li, X., Van Der Linden, M., and Bechara, A. (2009). Neural correlates of risk prediction error during reinforcement learning in humans. Neuroimage 47, 1929-1939. doi: 10. 1016/j. neuroimage. 2009. 04. 096.
16. Daw, N. D., Kakade, S., and Dayan, P. (2002). Opponent interactions between serotonin and dopamine. Neural Netw. 15, 603-616. doi: 10. 1016/S0893-6080(02)00052-7.
17. Dayan, P., and Huys, Q. J. (2008). Serotonin, inhibition, and negative mood. PLoS Comput. Biol. 4: e4. doi: 10. 1371/journal. pcbi. 0040004.
18. Di Giovanni, G., Di Matteo, V., Pierucci, M., and Esposito, E. (2008). Serotonin-dopamine interaction: electrophysiological evidence. Prog. Brain Res. 172, 45-71. doi: 10. 1016/S0079-6123(08)00903-5.
19. Di Mascio, M., Di Giovanni, G., Di Matteo, V., Prisco, S., and Esposito, E. (1998). Selective serotonin reuptake inhibitors reduce the spontaneous activity of dopaminergic neurons in the ventral tegmental area. Brain Res. Bull. 46, 547-554. doi: 10. 1016/S0361-9230(98)00054-9.
20. Di Matteo, V., Di Giovanni, G., Pierucci, M., and Esposito, E. (2008a). Serotonin control of central dopaminergic function: focus on in vivo microdialysis studies. Prog. Brain Res. 172, 7-44. doi: 10. 1016/S0079-6123(08)00902-3.
21. Di Matteo, V., Pierucci, M., Esposito, E., Crescimanno, G., Benigno, A., and Di Giovanni, G. (2008b). Serotonin modulation of the basal ganglia circuitry: therapeutic implication for Parkinson's disease and other motor disorders. Prog. Brain Res. 172, 423-463. doi: 10. 1016/S0079-6123(08)00921-7.
22. Doya, K. (2002). Metalearning and neuromodulation. Neural Netw. 15, 495-506. doi: 10. 1016/S0893-6080(02)00044-8.
23. Ferre, S., Cortes, R., and Artigas, F. (1994). Dopaminergic regulation of the serotonergic raphe-striatal pathway: microdialysis studies in freely moving rats. J. Neurosci. 14, 4839-4846.
24. Flores-Hernandez, J., Cepeda, C., Hernandez-Echeagaray, E., Calvert, C. R., Jokel, E. S., Fienberg, A. A., et al. (2002). Dopamine enhancement of NMDA currents in dissociated medium-sized striatal neurons: role of D1 receptors and DARPP-32. J. Neurophysiol. 88, 3010-3020. doi: 10. 1152/jn. 00361. 2002.
25. Floresco, S. B., Ghods-Sharifi, S., Vexelman, C., and Magyar, O. (2006). Dissociable roles for the nucleus accumbens core and shell in regulating set shifting. J. Neurosci. 26, 2449-2457. doi: 10. 1523/JNEUROSCI. 4431-05. 2006.
26. Frank, M. J., Samanta, J., Moustafa, A. A., and Sherman, S. J. (2007). Hold your horses: impulsivity, deep brain stimulation, and medication in parkinsonism. Science 318, 1309-1312. doi: 10. 1126/science. 1146157.
27. Gervais, J., and Rouillard, C. (2000). Dorsal raphe stimulation differentially modulates dopaminergic neurons in the ventral tegmental area and substantia nigra. Synapse 35, 281-291. doi: 10. 1002/(SICI)1098-2396(20000315)35: 4<281:: AID-SYN6>3. 0. CO; 2-A.
28. Gillette, R. (2006). Evolution and function in serotonergic systems. Integr. Comp. Biol. 46, 838-846. doi: 10. 1093/icb/icl024.
29. Goetz, C. G., Chmura, T. A., and Lanska, D. J. (2001). Seminal figures in the history of movement disorders: sydenham, Parkinson, and Charcot: Part 6 of the MDS-sponsored history of movement disorders exhibit, Barcelona, June 2000. Mov. Disord. 16, 537-540. doi: 10. 1002/mds. 1113.
30. Goldberg, D. E. (1989). Genetic Algorithms in Search Optimization and Machine Learning. Boston, MA: Addison-Wesley Longman Publishing Co.
31. Halford, J. C., Harrold, J. A., Lawton, C. L., and Blundell, J. E. (2005). Serotonin (5-HT) drugs: effects on appetite expression and use for the treatment of obesity. Curr. Drug Targets 6, 201-213. doi: 10. 2174/1389450053174550.
32. He, Q., Xue, G., Chen, C., Lu, Z., Dong, Q., Lei, X., et al. (2010). Serotonin transporter gene-linked polymorphic region (5-HTTLPR) influences decision making under ambiguity and risk in a large Chinese sample. Neuropharmacology 59, 518-526. doi: 10. 1016/j. neuropharm. 2010. 07. 008.
33. Hikosaka, O. (2010). The habenula: from stress evasion to value-based decision-making. Nat. Rev. Neurosci. 11, 503-513. doi: 10. 1038/nrn2866.
34. Humphries, M. D., Lepora, N., Wood, R., and Gurney, K. (2009). Capturing dopaminergic modulation and bimodal membrane behaviour of striatal medium spiny neurons in accurate, reduced models. Front. Comput. Neurosci. 3: 26. doi: 10. 3389/neuro. 10. 026. 2009.
35. Joel, D., Niv, Y., and Ruppin, E. (2002). Actor-critic models of the basal ganglia: new anatomical and computational perspectives. Neural Netw. 15, 535-547. doi: 10. 1016/S0893-6080(02)00047-3.
36. Kahneman, D., and Tversky, A. (1979). Prospect theory: an analysis of decision under risk. Econometrica 47, 263-292. doi: 10. 2307/1914185.
37. Kalva, S. K., Rengaswamy, M., Chakravarthy, V. S., and Gupte, N. (2012). On the neural substrates for exploratory dynamics in basal ganglia: a model. Neural Netw. 32, 65-73. doi: 10. 1016/j. neunet. 2012. 02. 031.
38. Kötter, R., and Wickens, J. (1998). Striatal mechanisms in Parkinson's disease: new insights from computer modeling. Artif. Intell. Med. 13, 37-55. doi: 10. 1016/S0933-3657(98)00003-7.
39. Kravitz, E. A. (2000). Serotonin and aggression: insights gained from a lobster model system and speculations on the role of amine neurons in a complex behavior. J. Comp. Physiol. A 186, 221-238. doi: 10. 1007/s003590050423.
40. Krishnan, R., Ratnadurai, S., Subramanian, D., Chakravarthy, V. S., and Rengaswamy, M. (2011). Modeling the role of basal ganglia in saccade generation: is the indirect pathway the explorer? Neural Netw. 24, 801-813. doi: 10. 1016/j. neunet. 2011. 06. 002.
41. Kuhnen, C. M., Samanez-Larkin, G. R., and Knutson, B. (2013). Serotonergic Genotypes, Neuroticism, and Financial Choices. PLoS ONE 8: e54632. doi: 10. 1371/journal. pone. 0054632.
42. Lecourtier, L., and Kelly, P. H. (2007). A conductor hidden in the orchestra? Role of the habenular complex in monoamine transmission and cognition. Neurosci Biobehav. Rev. 31, 658-672. doi: 10. 1016/j. neubiorev. 2007. 01. 004.
43. Long, A. B., Kuhn, C. M., and Platt, M. L. (2009). Serotonin shapes risky decision making in monkeys. Soc. Cogn. Affect. Neurosci. 4, 346-356. doi: 10. 1093/scan/nsp020.
44. Montague, P. R., Dayan, P., Person, C., and Sejnowski, T. J. (1995). Bee foraging in uncertain environments using predictive hebbian learning. Nature 377, 725-728. doi: 10. 1038/377725a0.
45. Morita, K., Morishima, M., Sakai, K., and Kawaguchi, Y. (2012). Reinforcement learning: computing the temporal difference of values via distinct corticostriatal pathways. Trends Neurosci. 35, 457-467. doi: 10. 1016/j. tins. 2012. 04. 009.
46. Moyer, J. T., Wolf, J. A., and Finkel, L. H. (2007). Effects of dopaminergic modulation on the integrative properties of the ventral striatal medium spiny neuron. J. Neurophysiol. 98, 3731-3748. doi: 10. 1152/jn. 00335. 2007.
47. Murphy, S. E., Longhitano, C., Ayres, R. E., Cowen, P. J., Harmer, C. J., and Rogers, R. D. (2009). The role of serotonin in nonnormative risky choice: the effects of tryptophan supplements on the "reflection effect" in healthy adult volunteers. J. Cogn. Neurosci. 21, 1709-1719. doi: 10. 1162/jocn. 2009. 21122.
48. O'doherty, J. P., Buchanan, T. W., Seymour, B., and Dolan, R. J. (2006). Predictive neural coding of reward preference involves dissociable responses in human ventral midbrain and ventral striatum. Neuron 49, 157-166. doi: 10. 1016/j. neuron. 2005. 11. 014.
49. Perreault, M. L., Hasbi, A., Alijaniaram, M., Fan, T., Varghese, G., Fletcher, P. J., et al. (2010). The dopamine D1-D2 receptor heteromer localizes in dynorphin/enkephalin neurons: increased high affinity state following amphetamine and in schizophrenia. J. Biol. Chem. 285, 36625-36634. doi: 10. 1074/jbc. M110. 159954.
50. Preuschoff, K., Bossaerts, P., and Quartz, S. R. (2006). Neural differentiation of expected reward and risk in human subcortical structures. Neuron 51, 381-390. doi: 10. 1016/j. neuron. 2006. 06. 024.
51. Real, L. A. (1981). Uncertainty and plant-pollinator interactions: the foraging behavior of bees and wasps on artificial flowers. Ecology 62, 20-26 doi: 10. 2307/1936663.
52. Robinson, O. J., Cools, R., and Sahakian, B. J. (2012). Tryptophan depletion disinhibits punishment but not reward prediction: implications for resilience. Psychopharmacology (Berl). 219, 599-605. doi: 10. 1007/s00213-011-2410-5.
53. Rogers, R. D. (2011). The roles of dopamine and serotonin in decision making: evidence from pharmacological experiments in humans. Neuropsychopharmacology 36, 114-132. doi: 10. 1038/npp. 2010. 165.
54. Schultz, W. (1998). The phasic reward signal of primate dopamine neurons. Adv. Pharmacol. 42, 686-690. doi: 10. 1016/S1054-3589(08)60841-8.
55. Schultz, W., Dayan, P., and Montague, P. R. (1997). A neural substrate of prediction and reward. Science 275, 1593-1599. doi: 10. 1126/science. 275. 5306. 1593.
56. Schulz, D. J., and Robinson, G. E. (1999). Biogenic amines and division of labor in honey bee colonies: behaviorally related changes in the antennal lobes and age-related changes in the mushroom bodies. J. Comp. Physiol. A 184, 481-488. doi: 10. 1007/s003590050348.
57. Servan-Schreiber, D., Printz, H., and Cohen, J. D. (1990). A network model of catecholamine effects: gain, signal-to-noise ratio, and behavior. Science 249, 892-895. doi: 10. 1126/science. 2392679.
58. So, C. H., Verma, V., Alijaniaram, M., Cheng, R., Rashid, A. J., O'dowd, B. F., et al. (2009). Calcium signaling by dopamine D5 receptor and D5-D2 receptor hetero-oligomers occurs by a mechanism distinct from that for dopamine D1-D2 receptor hetero-oligomers. Mol. Pharmacol. 75, 843-854. doi: 10. 1124/mol. 108. 051805.
59. Stopper, C. M., and Floresco, S. B. (2011). Contributions of the nucleus accumbens and its subregions to different aspects of risk-based decision making. Cogn. Affect. Behav. Neurosci. 11, 97-112. doi: 10. 3758/s13415-010-0015-9.
60. Surmeier, D. J., Song, W.-J., and Yan, Z. (1996). Coordinated expression of dopamine receptors in neostriatal medium spiny neurons. J. Neurosci. 16, 6579-6591.
61. Sutton, R., and Barto, A. (1998). Reinforcement Learning: An Introduction. Adaptive Computations and Machine Learning. Bradford: MIT Press.
62. Suzuki, M., Hurd, Y. L., Sokoloff, P., Schwartz, J. C., and Sedvall, G. (1998). D3 dopamine receptor mRNA is widely expressed in the human brain. Brain Res. 779, 58-74. doi: 10. 1016/S0006-8993(97)01078-0.
63. Tanaka, S. C., Schweighofer, N., Asahi, S., Shishida, K., Okamoto, Y., Yamawaki, S., et al. (2007). Serotonin differentially regulates short-and long-term prediction of rewards in the ventral and dorsal striatum. PLoS ONE2: e1333. doi: 10. 1371/journal. pone. 0001333.
64. Tanaka, S. C., Shishida, K., Schweighofer, N., Okamoto, Y., Yamawaki, S., and Doya, K. (2009). Serotonin affects association of aversive outcomes to past actions. J. Neurosci. 29, 15669-15674. doi: 10. 1523/JNEUROSCI. 2799-09. 2009.
65. Thurley, K., Senn, W., and Lüscher, H.-R. (2008). Dopamine increases the gain of the input-output response of rat prefrontal pyramidal neurons. J. Neurophysiol. 99, 2985-2997. doi: 10. 1152/jn. 01098. 2007.
66. Tops, M., Russo, S., Boksem, M. A., and Tucker, D. M. (2009). Serotonin: modulator of a drive to withdraw. Brain Cogn. 71, 427-436. doi: 10. 1016/j. bandc. 2009. 03. 009.
67. Wagener-Hulme, C., Kuehn, J. C., Schulz, D. J., and Robinson, G. E. (1999). Biogenic amines and division of labor in honey bee colonies. J. Comp. Physiol. A 184, 471-479. doi: 10. 1007/s003590050347.
68. Wallman, M. J., Gagnon, D., and Parent, M. (2011). Serotonin innervation of human basal ganglia. Eur. J. Neurosci. 33, 1519-1532. doi: 10. 1111/j. 1460-9568. 2011. 07621. x.
69. Zhong, S., Israel, S., Xue, H., Ebstein, R. P., and Chew, S. H. (2009a). Monoamine oxidase A gene (MAOA) associated with attitude towards longshot risks. PLoS ONE 4: e8516. doi: 10. 1371/journal. pone. 0008516.
70. Zhong, S., Israel, S., Xue, H., Sham, P. C., Ebstein, R. P., and Chew, S. H. (2009b). A neurochemical approach to valuation sensitivity over gains and losses. Proc. R. Soc. B Biol. Sci. 276, 4181-4188. doi: 10. 1098/rspb. 2009. 1312.