Conflict acts as an implicit cost in reinforcement learning

Nature Communications

Volumn 5, Issue , 2014, Pages

  Cavanagh, James F ^a   Masters, Sean E ^b   Bath, Kevin ^c   Frank, Michael J ^b  

^a UNIVERSITY OF NEW MEXICO   (United States)

^b BROWN UNIVERSITY   (United States)

^c BROWN UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CABERGOLINE; PLACEBO; DOPAMINE; DOPAMINE RECEPTOR STIMULATING AGENT; ERGOLINE DERIVATIVE; PHOSPHOPROTEIN DARPP 32; PPP1R1B PROTEIN, HUMAN;

AVOIDANCE REACTION; COST-BENEFIT ANALYSIS; POLYMORPHISM; SENSITIVITY ANALYSIS;

ADULT; ARTICLE; CONFLICT; CORRELATIONAL STUDY; COST; DIZZINESS; ELECTROENCEPHALOGRAPHY; EYELID REFLEX; FEMALE; GENETIC ASSOCIATION; GENETIC POLYMORPHISM; HUMAN; HUMAN EXPERIMENT; NAUSEA; NORMAL HUMAN; PERSONAL EXPERIENCE; PUNISHMENT; REINFORCEMENT; REWARD; VALIDATION STUDY; ADOLESCENT; ASSOCIATION; EYE MOVEMENT; GENETICS; MALE; METABOLISM; THETA RHYTHM; YOUNG ADULT;

ADOLESCENT; ADULT; CONFLICT (PSYCHOLOGY); CUES; DOPAMINE; DOPAMINE AGONISTS; DOPAMINE AND CAMP-REGULATED PHOSPHOPROTEIN 32; ERGOLINES; EYE MOVEMENTS; FEMALE; HUMANS; MALE; REINFORCEMENT (PSYCHOLOGY); THETA RHYTHM; YOUNG ADULT;

EID: 84922975531     PISSN: None     EISSN: 20411723     Source Type: Journal    
DOI: 10.1038/ncomms6394     Document Type: Article

References (52)

1. Botvinick, M. M. Conflict monitoring and decision making: reconciling two perspectives on anterior cingulate function. Cogn. Affect. Behav. Neurosci. 7, 356-366 (2007).
2. Kool, W., McGuire, J. T., Rosen, Z. B. & Botvinick, M. M. Decision making and the avoidance of cognitive demand. J. Exp. Psychol. Gen. 139, 665-682 (2010).
3. Botvinick, M. M., Huffstetler, S. & McGuire, J. T. Effort discounting in human nucleus accumbens. Cogn. Affect. Behav. Neurosci. 9, 16-27 (2009).
4. McGuire, J. T. & Botvinick, M. M. Prefrontal cortex, cognitive control, and the registration of decision costs. Proc. Natl Acad. Sci. USA 107, 7922-7926 (2010).
5. Schmidt, L., Lebreton, M., Cléry-Melin, M.-L., Daunizeau, J. & Pessiglione, M. Neural mechanisms underlying motivation of mental versus physical effort. PLoS Biol. 10, e1001266 (2012).
6. Dreisbach, G. & Fischer, R. Conflicts as aversive signals. Brain Cogn. 78, 94-98 (2012).
7. Fritz, J. & Dreisbach, G. Conflicts as aversive signals: conflict priming increases negative judgments for neutral stimuli. Cogn. Affect. Behav. Neurosci. 13, 311-317 (2013).
8. Kurniawan, I. T., Guitart-Masip, M., Dayan, P. & Dolan, R. J. Effort and valuation in the brain: the effects of anticipation and execution. J. Neurosci. 33, 6160-6169 (2013).
9. Croxson, P. L., Walton, M. E., O'Reilly, J. X., Behrens, T. E. J. & Rushworth, M. F. S. Effort-based cost-benefit valuation and the human brain. J. Neurosci. 29, 4531-4541 (2009).
10. Walton, M. E., Bannerman, D. M., Alterescu, K. & Rushworth, M. F. S. Functional specialization within medial frontal cortex of the anterior cingulate for evaluating effort-related decisions. J. Neurosci. 23, 6475-6479 (2003).
11. Treadway, M. T. et al. Dopaminergic mechanisms of individual differences in human effort-based decision-making. J. Neurosci. 32, 6170-6176 (2012).
12. Salamone, J. D. Dopamine, effort, and decision making: theoretical comment on Bardgett et al. (2009). Behav. Neurosci. 123, 463-467 (2009).
13. Wardle, M. C., Treadway, M. T., Mayo, L. M., Zald, D. H. & de Wit, H. Amping up effort: effects of d-amphetamine on human effort-based decision-making. J. Neurosci. 31, 16597-16602 (2011).
14. Salamone, J. D., Correa, M., Farrar, A. M., Nunes, E. J. & Pardo, M. Dopamine, behavioral economics, and effort. Front. Behav. Neurosci. 3, 13 (2009).
15. Randall, P. A. et al. Dopaminergic modulation of effort-related choice behavior as assessed by a progressive ratio chow feeding choice task: pharmacological studies and the role of individual differences. PLoS ONE 7, e47934 (2012).
16. Denk, F. et al. Differential involvement of serotonin and dopamine systems in cost-benefit decisions about delay or effort. Psychopharmacology (Berl.) 179, 587-596 (2005).
17. Drew, M. R. et al. Transient overexpression of striatal D2 receptors impairs operant motivation and interval timing. J. Neurosci. 27, 7731-7739 (2007).
18. Simpson, E. H. et al. Pharmacologic rescue of motivational deficit in an animal model of the negative symptoms of schizophrenia. Biol. Psychiatry 69, 928-935 (2011).
19. Cavanagh, J. F., Zambrano-Vazquez, L. & Allen, J. J. B. Theta lingua franca: a common mid-frontal substrate for action monitoring processes. Psychophysiology 49, 220-238 (2012).
20. Cavanagh, J. F. & Frank, M. J. Frontal theta as a mechanism for cognitive control. Trends Cogn. Sci. 18, 1-8 (2014).
21. Cavanagh, J. F. & Shackman, A. J. Frontal midline theta reflects anxiety and cognitive control: meta-analytic evidence. J. Physiol. Paris doi:10.1016/j.jphysparis.2014.04.003 (2014).
22. Kravitz, A. V. et al. Regulation of parkinsonian motor behaviours by optogenetic control of basal ganglia circuitry. Nature 466, 622-626 (2010).
23. Kravitz, A. V., Tye, L. D. & Kreitzer, A. C. Distinct roles for direct and indirect pathway striatal neurons in reinforcement. Nat. Neurosci. 15, 816-818 (2012).
24. Tai, L.-H., Lee, A. M., Benavidez, N., Bonci, A. & Wilbrecht, L. Transient stimulation of distinct subpopulations of striatal neurons mimics changes in action value. Nat. Neurosci. 15, 1281-1289 (2012).
25. Porter-stransky, K. A., Seiler, J. L., Day, J. J. & Aragona, B. J. Development of behavioral preferences for the optimal choice following unexpected reward omission is mediated by a reduction of D2-like receptor tone in the nucleus accumbens. Eur. J. Neurosci. 38, 2572-2588 (2013).
26. Collins, A. G. E. & Frank, M. J. Opponent Actor Learning (OpAL): modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive. Psychol. Rev. 121, 337-366 (2014).
27. Frank, M. J., Doll, B. B., Oas-Terpstra, J. & Moreno, F. Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation. Nat. Neurosci. 12, 1062-1068 (2009).
28. Doll, B. B., Hutchison, K. E. & Frank, M. J. Dopaminergic genes predict individual differences in susceptibility to confirmation bias. J. Neurosci. 31, 6188-6198 (2011).
29. Cockburn, J., Collins, A. G. E. & Frank, M. J. A reinforcement learning mechanism responsible for the valuation of free choice. Neuron 83, 551-557 (2014).
30. Frank, M. J. & O'Reilly, R. C. A mechanistic account of striatal dopamine function in human cognition: psychopharmacological studies with cabergoline and haloperidol. Behav. Neurosci. 120, 497-517 (2006).
31. Santesso, D. L. et al. Single dose of a dopamine agonist impairs reinforcement learning in humans: evidence from event-related potentials and computational modeling of striatal-cortical function. Hum. Brain Mapp. 30, 1963-1976 (2009).
32. Cools, R. et al. Striatal dopamine predicts outcome-specific reversal learning and its sensitivity to dopaminergic drug administration. J. Neurosci. 29, 1538-1543 (2009).
33. Jocham, G., Klein, T. A. & Ullsperger, M. Dopamine-mediated reinforcement learning signals in the striatum and ventromedial prefrontal cortex underlie value-based choices. J. Neurosci. 31, 1606-1613 (2011).
34. Kleven, S. & Recherche, D. Differential effects of direct and indirect dopamine on eye blink rate in cynomolgus monkeys in blinking. J. Pharmacol. Exp. Ther. 279, 1211-1219 (1996).
35. Slagter, H. A., Davidson, R. J. & Tomer, R. Eye-blink rate predicts individual differences in pseudoneglect. Neuropsychologia 48, 1265-1268 (2010).
36. Taylor, J. R. et al. Spontaneous blink rates correlate with dopamine levels in the caudate nucleus of MPTP-treated monkeys. Exp. Neurol. 158, 214-220 (1999).
37. Elsworth, J. D. et al. D1 and D2 dopamine receptors independently regulate spontaneous blink rate in the vervet monkey. J. Pharmacol. Exp. Ther. 259, 595-600 (1991).
38. Simon, J. R. & Rudell, A. P. Auditory S-R compatibility: the effect of an irrelevant cue on information processing. J. Appl. Psychol. 51, 300-304 (1967).
39. Frank, M. J., Woroch, B. S. & Curran, T. Error-related negativity predicts reinforcement learning and conflict biases. Neuron 47, 495-501 (2005).
40. Cavanagh, J. F., Bismark, A. J., Frank, M. J. & Allen, J. J. B. Larger error signals in major depression are associated with better avoidance learning. Front. Psychol. 2, 331 (2011).
41. Kayser, J. & Tenke, C. E. Principal components analysis of Laplacian waveforms as a generic method for identifying ERP generator patterns: I. Evaluation with auditory oddball tasks. Clin. Neurophysiol. 117, 348-368 (2006).
42. Holroyd, C. B., Pakzad-Vaezi, K. L. & Krigolson, O. E. The feedback correct-related positivity: sensitivity of the event-related brain potential to unexpected positive feedback. Psychophysiology 45, 688-697 (2008).
43. Cavanagh, J. F., Eisenberg, I., Guitart-Masip, M., Huys, Q. & Frank, M. J. Frontal theta overrides pavlovian learning biases. J. Neurosci. 33, 8541-8548 (2013).
44. Collins, A. G. E., Cavanagh, J. F. & Frank, M. J. Human EEG uncovers latent generalizable rule structure during learning. J. Neurosci. 34, 4677-4685 (2014).
45. Frank, M. J., Moustafa, A. A., Haughey, H. M., Curran, T. & Hutchison, K. E. Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning. Proc. Natl Acad. Sci. USA 104, 16311-16316 (2007).
46. Yeung, N., Botvinick, M. M. & Cohen, J. D. The neural basis of error detection: conflict monitoring and the error-related negativity. Psychol. Rev. 111, 931-959 (2004).
47. Botvinick, M. M., Braver, T. S., Barch, D. M., Carter, C. S. & Cohen, J. D. Conflict monitoring and cognitive control. Psychol. Rev. 108, 624-652 (2001).
48. Holroyd, C. B. & Coles, M. G. The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychol. Rev. 109, 679-709 (2002).
49. Moser, J. S., Moran, T. P., Schroder, H. S., Donnellan, M. B. & Yeung, N. On the relationship between anxiety and error monitoring: a meta-analysis and conceptual framework. Front. Hum. Neurosci. 7, 466 (2013).
50. Meyer-lindenberg, A. et al. Genetic evidence implicating DARPP-32 in human frontostriatal structure, function, and cognition. Proc. Natl Acad. Sci. USA 117, 672-682 (2007).
51. Nandam, L. S. et al. Dopamine D2 receptor modulation of human response inhibition and error awareness. J. Cogn. Neurosci. 25, 649-656 (2013).
52. Norbury, A., Manohar, S., Rogers, R. D. & Husain, M. Dopamine modulates risk-taking as a function of baseline sensation-seeking trait. J. Neurosci. 33, 12982-12986 (2013).