Deficient reinforcement learning in medial frontal cortex as a model of dopamine-related motivational deficits in ADHD

Neural Networks

Volumn 46, Issue , 2013, Pages 199-209

  Silvetti, Massimo ^a,b   Wiersema, Jan R ^a   Sonuga Barke, Edmund ^a,c   Verguts, Tom ^a,b  

^a GHENT UNIVERSITY   (Belgium)

^b GHENT UNIVERSITY HOSPITAL   (Belgium)

^c UNIVERSITY OF SOUTHAMPTON   (United Kingdom)

Author keywords

ACC; ADHD; Dopamine; Prediction error; Reinforcement learning; Reward expectation

Indexed keywords

ACC; ADHD; DOPAMINE; PREDICTION ERRORS; REWARD EXPECTATION;

AMINES; COMPLEX NETWORKS; FORECASTING; REINFORCEMENT LEARNING;

NEUROPHYSIOLOGY;

DOPAMINE;

ANTERIOR CINGULATE; ARTICLE; ATTENTION DEFICIT DISORDER; COGNITION; COMPUTER PREDICTION; COMPUTER SIMULATION; DOPAMINERGIC TRANSMISSION; EXECUTIVE FUNCTION; FRONTAL CORTEX; MEDIAL FRONTAL CORTEX; MESOLIMBIC DOPAMINERGIC SYSTEM; MOTIVATION; NERVE CELL NETWORK; PRIORITY JOURNAL; REINFORCEMENT;

ACC; ADHD; DOPAMINE; PREDICTION ERROR; REINFORCEMENT LEARNING; REWARD EXPECTATION;

ATTENTION DEFICIT DISORDER WITH HYPERACTIVITY; DOPAMINE; FRONTAL LOBE; LEARNING; MODELS, NEUROLOGICAL; REINFORCEMENT (PSYCHOLOGY); REWARD;

EID: 84879804200     PISSN: 08936080     EISSN: 18792782     Source Type: Journal    
DOI: 10.1016/j.neunet.2013.05.008     Document Type: Article

References (99)

1. Alexander W.H., Brown J.W. Medial prefrontal cortex as an action-outcome predictor. Nature Neuroscience 2011, 14:1338-1344.
2. Alexander G.E., DeLong M.R., Strick P.L. Parallel organization of functionally segregated circuits linking basal ganglia and cortex. Annual Review of Neuroscience 1986, 9:357-381.
3. Amico F., Stauber J., Koutsouleris N., Frodl T. Anterior cingulate cortex gray matter abnormalities in adults with attention deficit hyperactivity disorder: a voxel-based morphometry study. Psychiatry Research 2011, 191:31-35.
4. Amiez C., Joseph J.P., Procyk E. Anterior cingulate error-related activity is modulated by predicted reward. The European Journal of Neuroscience 2005, 21:3447-3452.
5. Amiez C., Joseph J.P., Procyk E. Reward encoding in the monkey anterior cingulate cortex. Cerebral Cortex 2006, 16:1040-1055.
6. Aston-Jones G., Cohen J.D. Adaptive gain and the role of the locus coeruleus-norepinephrine system in optimal performance. The Journal of Comparative Neurology 2005, 493:99-110.
7. Barber M.A., Milich R., Welsch R. Effects of reinforcement schedule and task difficulty on the performance of attention deficit hyperactivity disordered and control boys. Journal of Clinical Child Psychology 1996, 25:66-76.
8. Botvinick M.M. Conflict monitoring and decision making: reconciling two perspectives on anterior cingulate function. Cognitive, Affective, & Behavioral Neuroscience 2007, 7:356-366.
9. Braver T.S., Cohen I. On the control of control: the role of dopamine in regulating prefrontal function and working memory. Attention and performance 2000, 713-737. MIT Press, Cambridge, MA. S. Monsell, J. Driver (Eds.).
10. Brookes K.J., Xu X., Anney R., Franke B., Zhou K., Chen W., et al. Association of ADHD with genetic variants in the 5'-region of the dopamine transporter gene: evidence for allelic heterogeneity. American Journal of Medical Genetics Part B: Neuropsychiatric Genetics 2008, 147B:1519-1523.
11. Bueti D., Bahrami B., Walsh V., Rees G. Encoding of temporal probabilities in the human brain. Journal of Neuroscience 2010, 30:4343-4352.
12. Bush G., Frazier J.A., Rauch S.L., Seidman L.J., Whalen P.J., Jenike M.A., et al. Anterior cingulate cortex dysfunction in attention-deficit/hyperactivity disorder revealed by fMRI and the counting Stroop. Biological Psychiatry 1999, 45:1542-1552.
13. Carmona S., Hoekzema E., Ramos-Quiroga J.A., Richarte V., Canals C., Bosch R., et al. Response inhibition and reward anticipation in medication-naive adults with attention-deficit/hyperactivity disorder: a within-subject case-control neuroimaging study. Human Brain Mapping 2011.
14. Castellanos F.X., Margulies D.S., Kelly C., Uddin L.Q., Ghaffari M., Kirsch A., et al. Cingulate-precuneus interactions: a new locus of dysfunction in adult attention-deficit/hyperactivity disorder. Biological Psychiatry 2008, 63:332-337.
15. Cockburn J., Holroyd C.B. Focus on the positive: computational simulations implicate asymmetrical reward prediction error signals in childhood attention-deficit/hyperactivity disorder. Brain Research 2010, 1365:18-34.
16. Cubillo A., Halari R., Smith A., Taylor E., Rubia K. A review of fronto-striatal and fronto-cortical brain abnormalities in children and adults with Attention Deficit Hyperactivity Disorder (ADHD) and new evidence for dysfunction in adults with ADHD during motivation and attention. Cortex 2011.
17. Demurie E., Roeyers H., Baeyens D., Sonuga-Barke E. Temporal discounting of monetary rewards in children and adolescents with ADHD and autism spectrum disorders. Developmental Science 2012, 15:791-800.
18. Devinsky O., Morrell M.J., Vogt B.A. Contributions of anterior cingulate cortex to behaviour. Brain 1995, 118(Pt 1):279-306.
19. Douglas V.I., Parry P.A. Effects of reward and nonreward on frustration and attention in attention deficit disorder. Journal of Abnormal Child Psychology 1994, 22:281-302.
20. Durston S., van Belle J., de Zeeuw P. Differentiating frontostriatal and fronto-cerebellar circuits in attention-deficit/hyperactivity disorder. Biological Psychiatry 2011, 69:1178-1184.
21. Falkenstein M., Hohnsbein J., Hoormann J., Blanke L. Effects of crossmodal divided attention on late ERP components. II. Error processing in choice reaction tasks. Electroencephalography and Clinical Neurophysiology 1991, 78:447-455.
22. Frank M.J. Dynamic dopamine modulation in the basal ganglia: a neurocomputational account of cognitive deficits in medicated and nonmedicated Parkinsonism. Journal of Cognitive Neuroscience 2005, 17:51-72.
23. Frank M.J., Santamaria A., O'Reilly R.C., Willcutt E. Testing~computational models of dopamine and noradrenaline dysfunction in attention deficit/hyperactivity disorder. Neuropsychopharmacology 2007, 32:1583-1599.
24. Frank M.J., Scheres A., Sherman S.J. Understanding decision-making deficits in neurological conditions: insights from models of natural action selection. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences 2007, 362:1641-1654.
25. Freibergs V., Douglas V.I. Concept learning in hyperactive and normal children. Journal of Abnormal Psychology 1969, 74:388-395.
26. Gehring W.J., Goss B., Coles M.G.H., Meyer D.E., Donchin E. A neural system for error detection and compensation. Psychological Science 1993, 4:385-390.
27. Geisler S., Derst C., Veh R.W., Zahm D.S. Glutamatergic afferents of the ventral tegmental area in the rat. Journal of Neuroscience 2007, 27:5730-5743.
28. Groen Y., Wijers A.A., Mulder L.J., Waggeveld B., Minderaa R.B., Althaus M. Error and feedback processing in children with ADHD and children with autistic spectrum disorder: an EEG event-related potential study. Clinical Neurophysiology 2008, 119:2476-2493.
29. Hall H., Sedvall G., Magnusson O., Kopp J., Halldin C., Farde L. Distribution of D1- and D2-dopamine receptors, and dopamine and its metabolites in the human brain. Neuropsychopharmacology 1994, 11:245-256.
30. Herrmann M.J., Mader K., Schreppel T., Jacob C., Heine M., Boreatti-Hummer A., et al. Neural correlates of performance monitoring in adult patients with attention deficit hyperactivity disorder (ADHD). The World Journal of Biological Psychiatry 2010, 11:457-464.
31. Herrmann M.J., Rommler J., Ehlis A.C., Heidrich A., Fallgatter A.J. Source localization (LORETA) of the error-related-negativity (ERN/Ne) and positivity (Pe). Brain Research. Cognitive Brain Research 2004, 20:294-299.
32. Holroyd C.B., Coles M.G. The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychological Review 2002, 109:679-709.
33. Hoogman M., Aarts E., Zwiers M., Slaats-Willemse D., Naber M., Onnink M., et al. Nitric oxide synthase genotype modulation of impulsivity and ventral striatal activity in adult ADHD patients and healthy comparison subjects. American Journal of Psychiatry 2011, 168:1099-1106.
34. Ivry R.B. The representation of temporal information in perception and motor control. Current Opinion in Neurobiology 1996, 6:851-857.
35. Jessup R.K., Busemeyer J.R., Brown J.W. Error effects in anterior cingulate cortex reverse when error likelihood is high. Journal of Neuroscience 2010, 30:3467-3472.
36. Jonkman L.M., van Melis J.J., Kemner C., Markus C.R. Methylphenidate improves deficient error evaluation in children with ADHD: an event-related brain potential study. Biological Psychology 2007, 76:217-229.
37. Kennerley S.W., Behrens T.E., Wallis J.D. Double dissociation of value computations in orbitofrontal and anterior cingulate neurons. Nature Neuroscience 2011, 14:1581-1589.
38. Ljungberg T., Apicella P., Schultz W. Responses of monkey dopamine neurons during learning of behavioral reactions. Journal of Neurophysiology 1992, 67:145-163.
39. Luman M., Oosterlaan J., Sergeant J.A. The impact of reinforcement contingencies on AD/HD: a review and theoretical appraisal. Clinical Psychology Review 2005, 25:183-213.
40. Luman M., Tripp G., Scheres A. Identifying the neurobiology of altered reinforcement sensitivity in ADHD: a review and research agenda. Neuroscience & Biobehavioral Reviews 2010, 34:744-754.
41. Luman M., van Noesel S.J., Papanikolau A., Van Oostenbruggen-Scheffer J., Veugelers D., Sergeant J.A., et al. Inhibition, reinforcement sensitivity and temporal information processing in ADHD and ADHD+ODD: evidence of a separate entity?. Journal of Abnormal Child Psychology 2009, 37:1123-1135.
42. Makris N., Biederman J., Valera E.M., Bush G., Kaiser J., Kennedy D.N., et al. Cortical thinning of the attention and executive function networks in adults with attention-deficit/hyperactivity disorder. Cerebral Cortex 2007, 17:1364-1375.
43. Makris N., Seidman L.J., Valera E.M., Biederman J., Monuteaux M.C., Kennedy D.N., et al. Anterior cingulate volumetric alterations in treatment-naive adults with ADHD: a pilot study. Journal of Attention Disorders 2010, 13:407-413.
44. Marco R., Miranda A., Schlotz W., Melia A., Mulligan A., Muller U., et al. Delay and reward choice in ADHD: an experimental test of the role of delay aversion. Neuropsychology 2009, 23:367-380.
45. Martel M.M., Roberts B., Gremillion M., von Eye A., Nigg J.T. External validation of bifactor model of ADHD: explaining heterogeneity in psychiatric comorbidity, cognitive control, and personality trait profiles within DSM-IV ADHD. Journal of Abnormal Child Psychology 2011, 39:1111-1123.
46. Matsumoto M., Matsumoto K., Abe H., Tanaka K. Medial prefrontal cell activity signaling prediction errors of action values. Nature Neuroscience 2007, 10:647-656.
47. Mauk M.D., Buonomano D.V. The neural basis of temporal processing. Annual Review of Neuroscience 2004, 27:307-340.
48. McNab F., Varrone A., Farde L., Jucaite A., Bystritsky P., Forssberg H., et al. Changes in cortical dopamine D1 receptor binding associated with cognitive training. Science 2009, 323:800-802.
49. Metin B., Roeyers H., Wiersema J.R., van der Meere J., Sonuga-Barke E. A meta-analytic study of event rate effects on Go/No-Go performance in attention-deficit/hyperactivity disorder. Biological Psychiatry 2012, 72:990-996.
50. Montague P.R., Dayan P., Sejnowski T.J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. Journal of Neuroscience 1996, 16:1936-1947.
51. Nigg J.T., Casey B.J. An integrative theory of attention-deficit/hyperactivity disorder based on the cognitive and affective neurosciences. Development and Psychopathology 2005, 17:785-806.
52. Oades R.D., Halliday G.M. Ventral tegmental (A10) system: neurobiology. 1. Anatomy and connectivity. Brain Research 1987, 434:117-165.
53. Oliveira F.T., McDonald J.J., Goodman D. Performance monitoring in the anterior cingulate is not all error related: expectancy deviation and the representation of action-outcome associations. Journal of Cognitive Neuroscience 2007, 19:1994-2004.
54. O'Reilly R.C., Frank M.J. Making working memory work: a computational model of learning in the prefrontal cortex and basal ganglia. Neural Computation 2006, 18:283-328.
55. O'Reilly R.C., Frank M.J., Hazy T.E., Watz B. PVLV: the primary value and learned value Pavlovian learning algorithm. Behavioral Neuroscience 2007, 121:31-49.
56. Parry P.A., Douglas V.I. Effects of reinforcement on concept identification in hyperactive children. Journal of Abnormal Child Psychology 1983, 11:327-340.
57. Posner M.I., Dehaene S. Attentional networks. Trends in Neurosciences 1994, 17:75-79.
58. Quilodran R., Rothe M., Procyk E. Behavioral shifts and action valuation in the anterior cingulate cortex. Neuron 2008, 57:314-325.
59. Roger C., Benar C.G., Vidal F., Hasbroucq T., Burle B. Rostral cingulate zone and correct response monitoring: ICA and source localization evidences for the unicity of correct- and error-negativities. NeuroImage 2010, 51:391-403.
60. Rubia K., Overmeyer S., Taylor E., Brammer M., Williams S.C., Simmons A., et al. Hypofrontality in attention deficit hyperactivity disorder during higher-order motor control: a study with functional MRI. American Journal of Psychiatry 1999, 156:891-896.
61. Rushworth M.F. Intention, choice, and the medial frontal cortex. Annals of the New York Academy of Sciences 2008, 1124:181-207.
62. Rushworth M.F., Behrens T.E. Choice, uncertainty and value in prefrontal and cingulate cortex. Nature Neuroscience 2008, 11:389-397.
63. Sagvolden T., Aase H., Zeiner P., Berger D. Altered reinforcement mechanisms in attention-deficit/hyperactivity disorder. Behavioural Brain Research 1998, 94:61-71.
64. Sagvolden T., Hendley E.D., Knardahl S. Behavior of hypertensive and hyperactive rat strains: hyperactivity is not unitarily determined. Physiology & Behavior 1992, 52:49-57.
65. Sagvolden T., Johansen E.B., Aase H., Russell V.A. A dynamic developmental theory of attention-deficit/hyperactivity disorder (ADHD) predominantly hyperactive/impulsive and combined subtypes. The Behavioral and Brain Sciences 2005, 28:397-419. Discussion 419-368.
66. Scheres A., Dijkstra M., Ainslie E., Balkan J., Reynolds B., Sonuga-Barke E., et al. Temporal and probabilistic discounting of rewards in children and adolescents: effects of age and ADHD symptoms. Neuropsychologia 2006, 44:2092-2103.
67. Scheres A., Lee A., Sumiya M. Temporal reward discounting and ADHD: task and symptom specific effects. Journal of Neural Transmission 2008, 115:221-226.
68. Scheres A., Milham M.P., Knutson B., Castellanos F.X. Ventral striatal hyporesponsiveness during reward anticipation in attention-deficit/hyperactivity disorder. Biological Psychiatry 2007, 61:720-724.
69. Scheres A., Tontsch C., Thoeny A.L., Kaczkurkin A. Temporal reward discounting in attention-deficit/hyperactivity disorder: the contribution of symptom domains, reward magnitude, and session length. Biological Psychiatry 2010, 67:641-648.
70. Schultz W. Predictive reward signal of dopamine neurons. Journal of Neurophysiology 1998, 80:1-27.
71. Seidman L.J., Valera E.M., Makris N., Monuteaux M.C., Boriel D.L., Kelkar K., et al. Dorsolateral prefrontal and anterior cingulate cortex volumetric abnormalities in adults with attention-deficit/hyperactivity disorder identified by magnetic resonance imaging. Biological Psychiatry 2006, 60:1071-1080.
72. Sikstrom S., Soderlund G. Stimulus-dependent dopamine release in attention-deficit/hyperactivity disorder. Psychological Review 2007, 114:1047-1075.
73. Silvetti M., Seurinck R., Verguts T. Value and prediction error in the medial frontal cortex: integrating the single-unit and systems levels of analysis. Frontiers in Human Neuroscience 2011, 5.
74. Silvetti M., Seurinck R., Verguts T. Value and prediction error in the medial frontal cortex: integrating the single-unit and systems levels of analysis. Frontiers in Human Neuroscience 2011, 5(75).
75. Silvetti M., Seurinck R., Verguts T. Value and prediction error estimation account for volatility effects in ACC: a model-based fMRI study. Cortex 2012, http://dx.doi.org/10.1016/j.cortex.2012.05.008.
76. Sonuga-Barke E.J. Psychological heterogeneity in AD/HD-a dual pathway model of behaviour and cognition. Behavioural Brain Research 2002, 130:29-36.
77. Sonuga-Barke E.J. The dual pathway model of AD/HD: an elaboration of neuro-developmental characteristics. Neuroscience & Biobehavioral Reviews 2003, 27:593-604.
78. Sonuga-Barke E.J. Editorial: ADHD as a reinforcement disorder-moving from general effects to identifying (six) specific models to test. Journal of Child Psychology and Psychiatry 2011, 52:917-918.
79. Sonuga-Barke E.J., Bitsakou P., Thompson M. Beyond the dual pathway model: evidence for the dissociation of timing, inhibitory, and delay-related impairments in attention-deficit/hyperactivity disorder. Journal of the American Academy of Child & Adolescent Psychiatry 2010, 49:345-355.
80. Sonuga-Barke E.J., Fairchild G. Neuroeconomics of attention-deficit/hyperactivity disorder: differential influences of medial, dorsal, and ventral prefrontal brain networks on suboptimal decision making?. Biological Psychiatry 2012, 72:126-133.
81. Sonuga-Barke E.J., Sergeant J.A., Nigg J., Willcutt E. Executive dysfunction and delay aversion in attention deficit hyperactivity disorder: nosologic and diagnostic implications. Child & Adolescent Psychiatric Clinics of North America 2008, 17:367-384. ix.
82. Strohle A., Stoy M., Wrase J., Schwarzer S., Schlagenhauf F., Huss M., et al. Reward anticipation and outcomes in adult males with attention-deficit/hyperactivity disorder. NeuroImage 2008, 39:966-972.
83. Stuber G.D., Klanker M., de Ridder B., Bowers M.S., Joosten R.N., Feenstra M.G., et al. Reward-predictive cues enhance excitatory synaptic strength onto midbrain dopamine neurons. Science 2008, 321:1690-1692.
84. Sutton R.S., Barto A.G. Reinforcement learning: an introduction 1998, MIT Press, Cambridge, MA.
85. Swanson J.M., Kinsbourne M., Nigg J., Lanphear B., Stefanatos G.A., Volkow N., et al. Etiologic subtypes of attention-deficit/hyperactivity disorder: brain imaging, molecular genetic and environmental factors and the dopamine hypothesis. Neuropsychology Review 2007, 17:39-59.
86. Takikawa Y., Kawagoe R., Hikosaka O. A possible role of midbrain dopamine neurons in short- and long-term adaptation of saccades to position-reward mapping. Journal of Neurophysiology 2004, 92:2520-2529.
87. Taylor E., Sonuga-Barke E.J. Disorders of attention and activity. Rutter's child and adolescent psychiatry 2008, Blackwell Publishing, Malden, MA, USA. M. Rutter (Ed.).
88. Tripp G., Wickens J.R. Research review: dopamine transfer deficit: a neurobiological theory of altered reinforcement mechanisms in ADHD. Journal of Child Psychology and Psychiatry 2008, 49:691-704.
89. van Meel C.S., Heslenfeld D.J., Oosterlaan J., Luman M., Sergeant J.A. ERPs associated with monitoring and evaluation of monetary reward and punishment in children with ADHD. Journal of Child Psychology and Psychiatry 2011, 52:942-953.
90. van Meel C.S., Oosterlaan J., Heslenfeld D.J., Sergeant J.A. Telling good from bad news: ADHD differentially affects processing of positive and negative feedback during guessing. Neuropsychologia 2005, 43:1946-1954.
91. Vijayraghavan S., Wang M., Birnbaum S.G., Williams G.V., Arnsten A.F. Inverted-U dopamine D1 receptor actions on prefrontal neurons engaged in working memory. Nature Neuroscience 2007, 10:376-384.
92. Volkow N.D., Wang G.J., Baler R.D. Reward, dopamine and the control of food intake: implications for obesity. Trends in Cognitive Sciences 2011, 15:37-46.
93. Volkow N.D., Wang G.J., Kollins S.H., Wigal T.L., Newcorn J.H., Telang F., et al. Evaluating dopamine reward pathway in ADHD: clinical implications. JAMA 2009, 302:1084-1091.
94. Volkow N.D., Wang G.J., Newcorn J., Telang F., Solanto M.V., Fowler J.S., et al. Depressed dopamine activity in caudate and preliminary evidence of limbic involvement in adults with attention-deficit/hyperactivity disorder. Archives of General Psychiatry 2007, 64:932-940.
95. Wiersema J.R., van der Meere J.J., Roeyers H. ERP correlates of error monitoring in adult ADHD. Journal of Neural Transmission 2009, 116:371-379.
96. Williams J., Dayan P. Dopamine, learning, and impulsivity: a biological account of attention-deficit/hyperactivity disorder. Journal of Child and Adolescent Psychopharmacology 2005, 15:160-179. Discussion 157-169.
97. Wise R.A. Brain reward circuitry: insights from unsensed incentives. Neuron 2002, 36:229-240.
98. Wise R.A. Dopamine, learning and motivation. Nature Reviews Neuroscience 2004, 5:483-494.
99. Yu A.J., Dayan P. Uncertainty, neuromodulation, and attention. Neuron 2005, 46:681-692.