Opponent actor learning (OpAL): Modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive

Psychological Review

Volumn 121, Issue 3, 2014, Pages 337-366

  Collins, Anne G E ^a   Frank, Michael J ^a  

^a BROWN UNIVERSITY   (United States)

Author keywords

Choice incentive; Computational model; Dopamine; Reinforcement learning; Striatum

Indexed keywords

DOPAMINE;

DECISION MAKING; HUMAN; LEARNING; NEOSTRIATUM; PHYSIOLOGY; REINFORCEMENT; THEORETICAL MODEL;

CHOICE BEHAVIOR; DOPAMINE; HUMANS; LEARNING; MODELS, THEORETICAL; NEOSTRIATUM; REINFORCEMENT (PSYCHOLOGY);

EID: 84905503383     PISSN: 0033295X     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0037015     Document Type: Article

References (68)

1. Aberman, J. E., & Salamone, J. D. (1999). Nucleus accumbens dopamine depletions make rats more sensitive to high ratio requirements but do not impair primary food reinforcement. Neuroscience, 92, 545-552. doi: 10.1016/S0306-4522(99)00004-4
2. Amtage, J., & Schmidt, W. J. (2003). Context-dependent catalepsy intensification is due to classical conditioning and sensitization. Behavioural Pharmacology, 14, 563-567. doi:10.1097/00008877-200311000-00009
3. Arias-Carrión, O., Stamelou, M., Murillo-Rodríguez, E., Menéndez-González, M., & Pöppel, E. (2010). Dopaminergic reward system: A short integrative review. International Archives of Medicine, 3, 24.
4. Bayer, H. M., & Glimcher, P. W. (2005). Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron, 47, 129- 141. doi:10.1016/j.neuron.2005.05.020
5. Bayer, H. M., Lau, B., & Glimcher, P. W. (2007). Statistics of midbrain dopamine neuron spike trains in the awake primate. Journal of Neurophysiology, 98, 1428-1439. doi:10.1152/jn.01140.2006
6. Beeler, J., Frank, M., McDaid, J., & Alexander, E. (2012). A role for dopamine-mediated learning in the pathophysiology and treatment of Parkinson's disease. Cell Reports, 2, 1747-1761.
7. Beeler, J. A., Daw, N., Frazier, C. R. M., & Zhuang, X. (2010). Tonic dopamine modulates exploitation of reward learning. Frontiers in Behavioral Neuroscience. doi:10.3389/fnbeh.2010.00170
8. Berridge, K. C. (2007). The debate over dopamine's role in reward: The case for incentive salience. Psychopharmacology, 191, 391-431. doi: 10.1007/s00213-006-0578-x
9. Berridge, K. C. (2012). From prediction error to incentive salience: Mesolimbic computation of reward motivation. European Journal of Neuroscience, 35, 1124-1143. doi:10.1111/j.1460-9568.2012.07990.x
10. Bódi, N., Kéri, S., Nagy, H., Moustafa, A., Myers, C. E., Daw, N., Gluck, M. A. (2009). Reward-learning and the novelty-seeking personality: A between- and within-subjects study of the effects of dopamine agonists on young Parkinson's patients. Brain: A Journal of Neurology, 132, 2385-2395. doi:10.1093/brain/awp094
11. Collins, A. G. E., & Frank, M. J. (2013). Cognitive control over learning: Creating, clustering, and generalizing task-set structure. Psychological Review, 120, 190-229. doi:10.1037/a0030852
12. Cools, R., Frank, M. J., Gibbs, S. E., Miyakawa, A., Jagust, W., & D'Esposito, M. (2009). Striatal dopamine predicts outcome-specific reversal learning and its sensitivity to dopaminergic drug administration. The Journal of Neuroscience, 29, 1538-1543. doi:10.1523/JNEUROSCI.4467-08.2009
13. Cousins, M., Atherton, A., Turner, L., & Salamone, J. (1996). Nucleus accumbens dopamine depletions alter relative response allocation in a T-maze cost/benefit task. Behavioural Brain Research, 74, 189-197.
14. Cousins, M. S., & Salamone, J. D. (1994). Nucleus accumbens dopamine depletions in rats affect relative response allocation in a novel cost/ benefit procedure. Pharmacology, Biochemistry and Behavior, 49, 85- 91. doi:10.1016/0091-3057(94)90460-X
15. Dayan, P. (2012). Instrumental vigour in punishment and reward. European Journal of Neuroscience, 35, 1152-1168. doi:10.1111/j.1460-9568 .2012.08026.x
16. Dayan, P., & Daw, N. D. (2008). Decision theory, reinforcement learning, and the brain. Cognitive, Affective & Behavioral Neuroscience, 8, 429- 453. doi:10.3758/CABN.8.4.429
17. Doll, B. B., Hutchison, K. E., & Frank, M. J. (2011). Dopaminergic genes predict individual differences in susceptibility to confirmation bias. The Journal of Neuroscience, 31, 6188-6198.
18. Doll, B. B., Jacobs, W. J., Sanfey, A. G., & Frank, M. J. (2009). Instructional control of reinforcement learning: A behavioral and neurocomputational investigation. Brain Research, 1299, 74-94. doi:10.1016/j.brainres.2009.07.007
19. Farrar, A. M., Font, L., Pereira, M., Mingote, S., Bunce, J. G., Chrobak, J. J., & Salamone, J. D. (2008). Forebrain circuitry involved in effortrelated choice: Injections of the GABAA agonist muscimol into ventral pallidum alter response allocation in food-seeking behavior. Neuroscience, 152, 321-330. doi:10.1016/j.neuroscience.2007.12.034
20. Farrar, A. M., Segovia, K. N., Randall, P. A., Nunes, E. J., Collins, L. E., Stopper, C. M., Salamone, J. D. (2010). Nucleus accumbens and effort-related functions: Behavioral and neural markers of the interactions between adenosine A2A and dopamine D2 receptors. Neuroscience, 166, 1056-1067. doi:10.1016/j.neuroscience.2009.12.056
21. Floresco, S. B., Tse, M. T. L., & Ghods-Sharifi, S. (2008). Dopaminergic and glutamatergic regulation of effort- and delay-based decision making. Neuropsychopharmacology, 33, 1966-1979.
22. Frank, M. J. (2005). Dynamic dopamine modulation in the basal ganglia: A neurocomputational account of cognitive deficits in medicated and nonmedicated Parkinsonism. Journal of Cognitive Neuroscience, 17, 51-72. doi:10.1162/0898929052880093
23. Frank, M. J., Moustafa, A. A., Haughey, H. M., Curran, T., & Hutchison, K. E. (2007). Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning. Proceedings of the National Academy of Sciences of the United States of America, 104, 16311-16316. doi:10.1073/pnas.0706111104
24. Frank, M. J., & O'Reilly, R. C. (2006). A mechanistic account of striatal dopamine function in human cognition: Psychopharmacological studies with cabergoline and haloperidol. Behavioral Neuroscience, 120, 497- 517. doi:10.1037/0735-7044.120.3.497
25. Frank, M. J., Santamaria, A., Reilly, R. C. O., & Willcutt, E. (2007). Testing computational models of dopamine and noradrenaline dysfunction in attention deficit/hyperactivity disorder. Neuropsychopharmacology, 32, 1583-1599.
26. Frank, M. J., Seeberger, L. C., & O'Reilly, R. C. (2004). By carrot or by stick: Cognitive reinforcement learning in parkinsonism. Science, 306, 1940-1943.
27. Gerfen, C. R. (2000). Molecular effects of dopamine on striatal-projection pathways. Trends in Neurosciences, 23 (Suppl), S64-S70. doi:10.1016/ S1471-1931(00)00019-7
28. Hazy, T. E., Frank, M. J., & O'Reilly, R. C. (2010). Neural mechanisms of acquired phasic dopamine responses in learning. Neuroscience and Biobehavioral Reviews, 34, 701-720. doi:10.1016/j.neubiorev.2009.11.019
29. Hikida, T., Kimura, K., Wada, N., Funabiki, K., & Nakanishi, S. (2010). Distinct roles of synaptic transmission in direct and indirect striatal pathways to reward and aversive behavior. Neuron, 66, 896-907. doi: 10.1016/j.neuron.2010.05.011
30. Huys, Q. J. M., Cools, R., Gölzer, M., Friedel, E., Heinz, A., Dolan, R. J., & Dayan, P. (2011). Disentangling the roles of approach, activation and valence in instrumental and Pavlovian responding. PLoS Computational Biology, 7, e1002028. doi:10.1371/journal.pcbi.1002028
31. Jocham, G., Klein, T. A.,&Ullsperger, M. (2011). Dopamine-mediated reinforcement learning signals in the striatum and ventromedial prefrontal cortex underlie value-based choices. The Journal of Neuroscience, 31, 1606-1613.
32. Klein, A., & Schmidt, W. J. (2003). Catalepsy intensifies contextdependently irrespective of whether it is induced by intermittent or chronic dopamine deficiency. Behavioural Pharmacology, 14, 49-53.
33. Kravitz, A. V., Freeze, B. S., Parker, P. R. L., Kay, K., Thwin, M. T., Deisseroth, K., & Kreitzer, A. C. (2010). Regulation of Parkinsonian motor behaviours by optogenetic control of basal ganglia circuitry. Nature, 466, 622-626. doi:10.1038/nature09159
34. Kravitz, A. V., Tye, L. D., & Kreitzer, A. C. (2012). Distinct roles for direct and indirect pathway striatal neurons in reinforcement. Nature Neuroscience, 15, 816-818. doi:10.1038/nn.3100
35. Maia, T. V., & Frank, M. J. (2011). From reinforcement learning models to psychiatric and neurological disorders. Nature Neuroscience, 14, 154-162.
36. McClure, S. M., Daw, N. D., & Read Montague, P. (2003). A computational substrate for incentive salience. Trends in Neurosciences, 26, 423-428. doi:10.1016/S0166-2236(03)00177-2
37. Mingote, S., Font, L., Farrar, A. M., Vontell, R., Worden, L. T., Stopper, C. M., Salamone, J. D. (2008). Nucleus accumbens adenosine A2A receptors regulate exertion of effort by acting on the ventral striatopallidal pathway. The Journal of Neuroscience, 28, 9037-9046.
38. Montague, P. R., Dayan, P., & Sejnowski, T. J. (1996). A framework for mesencephalic dopamine systems based on predictive Hebbian learning. The Journal of Neuroscience, 16, 1936-1947.
39. Morris, G., Nevet, A., Arkadir, D., Vaadia, E., & Bergman, H. (2006). Midbrain dopamine neurons encode decisions for future action. Nature Neuroscience, 9, 1057-1063. doi:10.1038/nn1743
40. Moustafa, A. A., Sherman, S. J., & Frank, M. J. (2008). A dopaminergic basis for working memory, learning and attentional shifting in Parkinsonism. Neuropsychologia, 46, 3144-3156. doi:10.1016/j.neuropsychologia. 2008.07.011
41. Nakahara, H., Itoh, H., Kawagoe, R., Takikawa, Y., & Hikosaka, O. (2004). Dopamine neurons can represent context-dependent prediction error. Neuron, 41, 269-280. doi:10.1016/S0896-6273(03)00869-9
42. Niv, Y., Daw, N. D., Joel, D., & Dayan, P. (2007). Tonic dopamine: Opportunity costs and the control of response vigor. Psychopharmacology, 191, 507-520. doi:10.1007/s00213-006-0502-4
43. Nomoto, K., Schultz, W., Watanabe, T., & Sakagami, M. (2010). Temporally extended dopamine responses to perceptually demanding rewardpredictive stimuli. The Journal of Neuroscience, 30, 10692-10702.
44. Nunes, E. J., Randall, P. A., Santerre, J. L., Given, A. B., Sager, T. N., Correa, M., & Salamone, J. D. (2010). Differential effects of selective adenosine antagonists on the effort-related impairments induced by dopamine D1 and D2 antagonism. Neuroscience, 170, 268-280. doi: 10.1016/j.neuroscience.2010.05.068
45. O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004). Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science, 304, 452-454.
46. Ott, D. V. M., Ullsperger, M., Jocham, G., Neumann, J., & Klein, T. A. (2011). Continuous theta-burst stimulation (cTBS) over the lateral prefrontal cortex alters reinforcement learning bias. NeuroImage, 57, 617- 623. doi:10.1016/j.neuroimage.2011.04.038
47. Palminteri, S., Boraud, T., Lafargue, G., Dubois, B., & Pessiglione, M. (2009). Brain hemispheres selectively track the expected value of contralateral options. The Journal of Neuroscience, 29, 13465-13472.
48. Pessiglione, M., Seymour, B., Flandin, G., Dolan, R. J., & Frith, C. D. (2006). Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature, 442, 1042-1045. doi:10.1038/nature05051
49. Ratcliff, R., & Frank, M. J. (2011). Reinforcement-based decision making in corticostriatal circuits: Mutual constraints by neurocomputational anddiffusion models. Neural Computation, 24, 1186-1229. doi:10.1162/ NECO_a_00270
50. Reynolds, J. N., Hyland, B. I., & Wickens, J. R. (2001). A cellular mechanism of reward-related learning. Nature, 413, 67-70. doi:10.1038/ 35092560
51. Roesch, M. R., Calu, D. J., & Schoenbaum, G. (2007). Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nature Neuroscience, 10, 1615-1624. doi:10.1038/ nn2013
52. Salamone, J. D., Correa, M., Mingote, S. M., & Weber, S. M. (2005). Beyond the reward hypothesis: Alternative functions of nucleus accumbens dopamine. Current Opinion in Pharmacology, 5, 34-41. doi: 10.1016/j.coph.2004.09.004
53. Samejima, K., Ueda, Y., Doya, K., & Kimura, M. (2005). Representation of action-specific reward values in the striatum. Science, 310, 1337- 1340.
54. Satoh, T., Nakai, S., Sato, T., & Kimura, M. (2003). Correlated coding of motivation and outcome of decision by dopamine neurons. The Journal of Neuroscience, 23, 9913-9923.
55. Schönberg, T., Daw, N. D., Joel, D., & O'Doherty, J. P. (2007). Reinforcement learning signals in the human striatum distinguish learners from nonlearners during reward-based decision making. The Journal of Neuroscience, 27, 12860-12867.
56. Schultz, W. (1997). A neural substrate of prediction and reward. Science, 275, 1593-1599. doi:10.1126/science.275.5306.1593
57. Shen, W., Flajolet, M., Greengard, P., & Surmeier, D. J. (2008). Dichotomous dopaminergic control of striatal synaptic plasticity. Science, 321, 848-851. doi:10.1126/science.1160575
58. Shiner, T., Seymour, B., Wunderlich, K., Hill, C., Bhatia, K. P., Dayan, P., & Dolan, R. J. (2012). Dopamine and performance in a reinforcement learning task: Evidence from Parkinson's disease. Brain: A Journal of Neurology, 135, 1871-1883. doi:10.1093/brain/aws083
59. Smith, K. S., Berridge, K. C., & Aldridge, J. W. (2011). Disentangling pleasure from incentive salience and learning signals in brain reward circuitry. Proceedings of the National Academy of Sciences of the United States of America, 108, E255-E264. doi:10.1073/pnas.1101920108
60. Smittenaar, P., Chase, H. W., Aarts, E., Nusselein, B., Bloem, B. R., & Cools, R. (2012). Decomposing effects of dopaminergic medication in Parkinson's disease on probabilistic action selection: Learning or performance? European Journal of Neuroscience, 35, 1144-1151. doi: 10.1111/j.1460-9568.2012.08043.x
61. Surmeier, D. J., Ding, J., Day, M., Wang, Z., & Shen, W. (2007). D1 and D2 dopamine-receptor modulation of striatal glutamatergic signaling in striatal medium spiny neurons. Trends in Neurosciences, 30, 228-235. doi:10.1016/j.tins.2007.03.008
62. Sutton, R., & Barto, A. (1998). Reinforcement learning. Journal of Cognitive Neuroscience, 11, 126-134. doi:10.1162/089892999563184
63. Tai, L.-H., Lee, A. M., Benavidez, N., Bonci, A., & Wilbrecht, L. (2012). Transient stimulation of distinct subpopulations of striatal neurons mimics changes in action value. Nature Neuroscience, 15, 1281-1289. doi: 10.1038/nn.3188
64. Treadway, M. T., Buckholtz, J. W., Cowan, R. L., Woodward, N. D., Li, R., Ansari, M. S., Zald, D. H. (2012). Dopaminergic mechanisms of individual differences in human effort-based decision-making. The Journal of Neuroscience, 32, 6170-6176.
65. Wassum, K. M., Ostlund, S. B., Balleine, B. W., & Maidment, N. T. (2011). Differential dependence of Pavlovian incentive motivation and instrumental incentive learning processes on dopamine signaling. Learning & Memory, 18, 475-483. doi:10.1101/lm.2229311
66. Wiecki, T. V., & Frank, M. J. (2013). A computational model of inhibitory control in frontal cortex and basal ganglia. Psychological Review, 120, 329-355. doi:10.1037/a0031542
67. Wiecki, T. V., Riedinger, K., von Ameln-Mayerhofer, A., Schmidt, W. J., & Frank, M. J. (2009). A neurocomputational account of catalepsy sensitization induced by D2 receptor blockade in rats: Context dependency, extinction, and renewal. Psychopharmacology, 204, 265-277. doi:10.1007/s00213-008-1457-4
68. Zhang, J., Berridge, K. C., Tindell, A. J., Smith, K. S., & Aldridge, J. W. (2009). A neural computational model of incentive salience. PLoS Computational Biology, 5, e1000437. doi:10.1371/journal.pcbi.100 0437