Extraversion differentiates between model-based and model-free strategies in a reinforcement learning task

Frontiers in Human Neuroscience

Volumn , Issue SEP, 2013, Pages

  Skatova, Anya ^a,b   Chan, Patricia A ^b   Daw, Nathaniel D ^b  

^a UNIVERSITY OF NOTTINGHAM   (United Kingdom)

^b NEW YORK UNIVERSITY   (United States)

Author keywords

Decision making; Dopamine; Extraversion; Personality; Reinforcement learning

Indexed keywords

EID: 84885030824     PISSN: 16625161     EISSN: None     Source Type: Journal    
DOI: 10.3389/fnhum.2013.00525     Document Type: Article

References (60)

1. Ashby, F. G., and Maddox, W. T. (2005). Human category learning. Annu. Rev. Psychol. 56, 149-178. doi: 10.1146/annurev.psych.56.091103.070217
2. Bates, D., Maechler, M., and Bolker, B. (2012). lme4: Linear Mixed-Effects Models Using S4 Classees. R package version 0.999999-0.
3. Bodi, N., Keri, S., Nagy, H., Moustafa, A., Myers, C. E., Daw, N. D., et al. (2009). Reward-learning and the novelty-seeking personality: a between-and within-subjects study of the effects of dopamine agonists on young Parkinsons patients. Brain 132, 2385-2395. doi: 10.1093/brain/awp094
4. Bush, R. R., and Mosteller, F. (1953). A stochastic model with applications to learning. Ann. Math. Stat. 24, 559-585. doi: 10.1214/aoms/1177728914
5. Cohen, M. X. (2007). Individual differences and the neural representations of reward expectation and reward prediction error. Soc. Cogn. Affect. Neurosci. 2, 20-30. doi: 10.1093/scan/nsl021
6. Cohen, M. X., Young, J., Baek, J. M., Kessler, C., and Ranganath, C. (2005). Individual differences in extraversion and dopamine genetics predict neural reward responses. Brain Res. Cogn. Brain Res. 25, 851-861. doi: 10.1016/j.cogbrainres.2005.09.018
7. Conner, B. T., Hellemann, G. S., Ritchie, T. L., and Noble, E. P. (2010). Genetic, personality, and environmental predictors of drug use in adolescents. J. Subst. Abuse Treat. 38, 178-190. doi: 10.1016/j.jsat.2009.07.004
8. Cools, R., Frank, M. J., Gibbs, S. E., Miyakawa, A., Jagust, W., and D'Esposito, M. (2009). Striatal dopamine predicts outcome-specific reversal learning and its sensitivity to dopaminergic drug administration. J. Neurosci. 29, 1538-1543. doi: 10.1523/JNEUROSCI.4467-08.2009
9. Corr, P. J., Pickering, A. D., and Gray, J. A. (1997). Personality, punishment, and procedural learning: a test of J. A. Gray's anxiety theory. J. Pers. Soc. psychol. 73, 337-344. doi: 10.1037/0022-3514.73.2.337
10. Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., and Dolan, R. J. (2011). Model-based influences on humans' choices and striatal prediction errors. Neuron 69, 1204-1215. doi: 10.1016/j.neuron.2011.02.027
11. Daw, N. D., Niv, Y., and Dayan, P. (2005). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704-1711. doi: 10.1038/nn1560
12. Daw, N. D., and O'Doherty, J. P. (2013). "Multiple systems for value learning, " in Neuroeconomics: Decision Making, and the Brain, 2nd Edn., eds P. W. Glimcher and E. Fehr (London; San Diego, CA; Burlington, MA: Elsevier).
13. Daw, N. D., and Shohamy, D. (2008). The cognitive neuroscience of motivation and learning. Soc. Cognit. 26, 593-620. doi: 10.1521/soco.2008.26.5.593
14. Dawe, S., Gullo, M. J., and Loxton, N. J. (2004). Reward drive and rash impulsiveness as dimensions of impulsivity: implications for substance misuse. Addict. Behav. 29, 1389-1405. doi: 10.1016/j.addbeh.2004.06.004
15. Dawe, S., and Loxton, N. J. (2004). The role of impulsivity in the development of substance use and eating disorders. Neurosci. Biobehav. Rev. 28, 343-351. doi: 10.1016/j.neubiorev.2004.03.007
16. Depue, R. A., and Collins, P. F. (1999). Neurobiology of the structure of personality: dopamine, facilitation of incentive motivation, and extraversion. Behav. Brain Sci. 22, 491-517. doi: 10.1017/S0140525X99002046
17. Dickinson, A., and Balleine, B. (2003). "The role of learning in the operation of motivational systems, " in Stevens' Handbook Experimental Psychology: Learning, Motivation, and Emotion, Vol. 3, eds H. Pashler and S. Yantis (New York, NY: John Wiley & Sons, Inc.), 497-532. doi: 10.1002/0471214426.pas0312
18. Everitt, B. J., and Robbins, T. W. (2005). Neural systems of reinforcement for drug addiction: from actions to habits to compulsion. Nat. Neurosci. 8, 1481-1489. doi: 10.1038/nn1579
19. Eysenck, S. B., Eysenck, H., and Barrett, P. (1985). A revised version of the psychoticism scale. Pers. Individ. Dif. 6, 21-29. doi: 10.1016/0191-8869(85)90026-1
20. Francis, L. J. (1997). The impact of personality and religion on attitude towards substance use among 13-15 year olds. Drug Alcohol Depend. 44, 95-103. doi: 10.1016/S0376-8716(96)01325-7
21. Frank, M. J., Seeberger, L. C., and O'Reilly, R. C. (2004). By carrot or by stick: cognitive reinforcement learning in parkinsonism. Science 306, 1940-1943. doi: 10.1126/science.1102941
22. Gelman, A., Carlin, J. B., Stern, H. S., and Rubin, D. B. (2003). Bayesian Data Analysis. Boca Raton, FL: CRC press.
23. Gelman, A., and Hill, J. (2007). Data Analysis Using Regression and Multilevel/Hierarchical Models. New York, NY: Cambridge University Press.
24. Gershman, S. J., Markman, A. B., and Otto, A. R. (2012). Retrospective revaluation in sequential decision making: a tale of two systems. J. Exp. Psychol. Gen. doi: 10.1037/a0030844
25. Gläscher, J., Daw, N. D., Dayan, P., and O'Doherty, J. P. (2010). States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66, 585-595. doi: 10.1016/j.neuron.2010.04.016
26. Herringer, L. G. (1998). Facets of extraversion related to life satisfaction. Pers. Individ. Dif. 24, 731-733. doi: 10.1016/S0191-8869(97)00194-3
27. Houk, J. C., Adams, J. L., and Barto, A. G. (1995). "A model of how the basal ganglia generate and use neural signals that predict reinforcement, " in Models of Information Processing in the Basal Ganglia, eds J. C. Houk, J. L. Davis and D. G. Beiser (Cambridge: The MIT Press) 249-270.
28. Jylhä, P., Melartin, T., and Isometsä, E. (2009). Relationships of neuroticism and extraversion with axis I and II comorbidity among patients with DSM-IV major depressive disorder. J. Affect. Disord. 114, 110-121. doi: 10.1016/j.jad.2008.06.011
29. Kass, R. E., and Raftery, A. E. (1995). Bayes factors. J. Am. Stat. Assoc. 90, 773-795. doi: 10.1080/01621459.1995.10476572
30. Matthews, G., and Gilliland, K. (1999). The personality theories of H.J. Eysenck and J.A. Gray: a comparative review. Pers. Individ. Dif. 26, 583-626. doi: 10.1016/S0191-8869(98)00158-5
31. McClure, S. M., Berns, G. S., and Montague, P. R. (2003). Temporal prediction errors in a passive learning task activate human striatum. Neuron 38, 339-346. doi: 10.1016/S0896-6273(03)00154-5
32. Nissen, M. J., and Bullemer, P. (1987). Attentional requirements of learning: evidence from performance measures. Cogn. Psychol. 19, 1-32. doi: 10.1016/0010-0285(87)90002-8
33. O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H., and Dolan, R. J. (2003). Temporal difference models and reward-related learning in the human brain. Neuron 38, 329-337. doi: 10.1016/S0896-6273(03)00169-7
34. Otto, A. R., Gershman, S. J., Markman, A. B., and Daw, N. D. (2013). The curse of planning: dissecting multiple reinforcement learning systems by taxing the central executive. Psychol. Sci. 24, 751-761. doi: 10.1177/0956797612463080
35. Pessiglione, M., Schmidt, L., Draganski, B., Kalisch, R., Lau, H., Dolan, R. J., et al. (2007). How the brain translates money into force: a neuroimaging study of subliminal motivation. Science 316, 904-906. doi: 10.1126/science.1140459
36. Pessiglione, M., Seymour, B., Flandin, G., Dolan, R. J., and Frith, C. D. (2006). Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature 442, 1042-1045. doi: 10.1038/nature05051
37. Pickering, A. D. (2004). "The neuropsychology of impulsive antisocial sensation seeking personality traits: from dopamine to hippocampal function, " in On the Psychobiology of Personality: Essays in Honor of Marvin Zuckerman, ed R. M. Stelmack (San Diego, CA: Elsevier), 455-478.
38. Pickering, A. D., Díaz, A., and Gray, J. A. (1995). Personality and reinforcement: an exploration using a maze-learning task. Pers. Individ. Dif. 18, 541-558. doi: 10.1016/0191-8869(94)00182-R
39. Pickering, A. D., and Smillie, L. D. (2008). "The behavioral activation system: challenges and opportunities, " in The Reinforcement Sensitivity Theory of Personality, ed P. J. Corr (Cambridge: Cambridge University Press), 120-154. doi: 10.1017/CBO9780511819384.005
40. R Development Core Team. (2011). "R: A language and enviroment for statistical computing, " in R Foundation for Statistical Computing (Vienna). ISBN 3-900051-07-0, URL http://www.R-project.org/
41. Rangel, A., Camerer, C., and Montague, P. R. (2008). A framework for studying the neurobiology of value-based decision making. Nat. Rev. Neurosci. 9, 545-556. doi: 10.1038/nrn2357
42. Redish, A. D., Jensen, S., and Johnson, A. (2008). A unified framework for addiction: vulnerabilities in the decision process. Behav. Brain Sci. 31, 415-436.
43. Renner, W., and Anderle, F.-G. (2000). Venturesomeness and extraversion as correlates of juvenile drivers' traffic violations. Accid. Anal. Prev. 32, 673-678. doi: 10.1016/S0001-4575(99)00103-7
44. Rescorla, R. A., and Wagner, A. R. (1972). "A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement, " in Classical Conditioning II: Current Research and Theory, eds A. H. Black and W. F. Prokasy (New York, NY: Appleton Century Crofts), 64-99.
45. Ryan, R. M., and Deci, E. L. (2001). On happiness and human potentials: a review of research on hedonic and eudaimonic well-being. Annu. Rev. Psychol. 52, 141-166. doi: 10.1146/annurev.psych.52.1.141
46. Schonberg, T., Daw, N. D., Joel, D., and O'Doherty, J. P. (2007). Reinforcement learning signals in the human striatum distinguish learners from nonlearners during reward-based decision making. J. Neurosci. 27, 12860-12867. doi: 10.1523/JNEUROSCI.2496-07.2007
47. Schonberg, T., O'Doherty, J. P., Joel, D., Inzelberg, R., Segev, Y., and Daw, N. D. (2010). Selective impairment of prediction error signaling in human dorsolateral but not ventral striatum in Parkinson's disease patients: evidence from a model-based fMRI study. Neuroimage 49, 772-781. doi: 10.1016/j.neuroimage.2009.08.011
48. Schultz, W., Dayan, P., and Montague, P. R. (1997). A neural substrate of prediction and reward. Science 275, 1593-1599. doi: 10.1126/science.275.5306.1593
49. Sloman, S. A. (1996). The empirical case for two systems of reasoning. Psychol. Bull. 119, 3-22. doi: 10.1037/0033-2909.119.1.3
50. Smillie, L. D. (2013). Extraversion and reward-processing. Curr. Dir. Psychol. Sci. 22, 167-172. doi: 10.1177/0963721412470133
51. Smillie, L. D., Cooper, A. J., and Pickering, A. D. (2011). Individual differences in reward-prediction-error: extraversion and feedback-related negativity. Soc. Cogn. Affect. Neurosci. 6, 646-652. doi: 10.1093/scan/nsq078
52. Smillie, L. D., Cooper, A. J., Proitsi, P., Powell, J. F., and Pickering, A. D. (2010). Variation in DRD2 dopamine gene predicts extraverted personality. Neurosci. Lett. 468, 234-237. doi: 10.1016/j.neulet.2009.10.095
53. Smillie, L. D., Dalgleish, L. I., and Jackson, C. J. (2007). Distinguishing between learning and motivation in behavioral tests of the reinforcement sensitivity theory of personality. Pers. Soc. Psychol. Bull. 33, 476-489. doi: 10.1177/0146167206296951
54. Sutton, R. S., and Barto, A. G. (1998). Reinforcement Learning: an Introduction, Vol. 1. Cambridge, MA: Cambridge University Press.
55. Thorndike, E. L. (1911). Edward Lee Thorndike. Anim. Intell. 1874, 1949.
56. Tolman, E. C., Ritchie, B. F., and Kalish, D. (1946). Studies in spatial learning: orientation and the short-cut. J. Exp. Psychol. 36, 13. doi: 10.1037/h0053944
57. Voon, V., Pessiglione, M., Brezing, C., Gallea, C., Fernandez, H. H., Dolan, R. J., et al. (2010). Mechanisms underlying dopamine-mediated reward bias in compulsive behaviors. Neuron 65, 135-142. doi: 10.1016/j.neuron.2009.12.027
58. Wacker, J., Chavanon, M.-L., and Stemmler, G. (2006). Investigating the dopaminergic basis of extraversion in humans: a multilevel approach. J. Pers. Soc. Psychol. 91, 171. doi: 10.1037/0022-3514.91.1.171
59. Williams, P. G., O'Brien, C. D., and Colder, C. R. (2004). The effects of neuroticism and extraversion on self-assessed health and health-relevant cognition. Pers. Individ. Dif. 37, 83-94. doi: 10.1016/j.paid.2003.08.001
60. Wunderlich, K., Smittenaar, P., and Dolan, R. J. (2012). Dopamine enhances model-based over model-free choice behavior. Neuron 75, 418-424. doi: 10.1016/j.neuron.2012.03.042