Learning from delayed feedback: Neural responses in temporal credit assignment

Cognitive, Affective and Behavioral Neuroscience

Volumn 11, Issue 2, 2011, Pages 131-143

  Walsh, Matthew M ^a   Anderson, John R ^a  

^a CARNEGIE MELLON UNIVERSITY   (United States)

Author keywords

Actor critic; Credit assignment; Eligibility traces; Event related potentials; Q learning; SARSA; Temporal difference learning

Indexed keywords

ADAPTIVE BEHAVIOR; ADOLESCENT; ADULT; ARTICLE; BRAIN MAPPING; COMPUTER SIMULATION; DECISION MAKING; EVOKED RESPONSE; FEMALE; HUMAN; LEARNING; MALE; METHODOLOGY; PHYSIOLOGY; PSYCHOLOGICAL MODEL; PSYCHOMOTOR PERFORMANCE; TIME;

ADOLESCENT; ADULT; BRAIN MAPPING; COMPUTER SIMULATION; DECISION MAKING; EVOKED POTENTIALS; FEEDBACK, PSYCHOLOGICAL; FEMALE; HUMANS; LEARNING; MALE; MODELS, PSYCHOLOGICAL; PSYCHOMOTOR PERFORMANCE; TIME FACTORS;

EID: 80051661786     PISSN: 15307026     EISSN: None     Source Type: Journal    
DOI: 10.3758/s13415-011-0027-0     Document Type: Article

References (65)

1. Baker, T. E., & Holroyd, C. B. (2009). Which way do I go? Neural activation in response to feedback and spatial processing in a virtual T-maze. Cerebral Cortex, 19, 1708-1722.
2. Barto, A. G., Sutton, R. S., & Anderson, C. W. (1983). Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics, 13, 834-846. (Pubitemid 14138646)
3. Bocker, K. B. E., Baas, J. M. P., Kenemans, J. L., & Verbaten, M. N. (2001). Stimulus-preceding negativity induced by fear: A manifestation of affective anticipation. International Journal of Psychophysiology, 43, 77-90. (Pubitemid 33115238)
4. Bogacz, R., McClure, S. M., Li, J., Cohen, J. D., & Montague, P. R. (2007). Short-term memory traces for action bias in human reinforcement learning. Brain Research, 1153, 111-121. (Pubitemid 46802525)
5. Braver, T. S., Barch, D. M., Gray, J. R., Molfese, D. L., & Snyder, A. (2001). Anterior cingulate cortex and response conflict: Effects of frequency, inhibition and errors. Cerebral Cortex, 11, 825-836. (Pubitemid 32800550)
6. Cohen, M. X., & Ranganath, C. (2007). Reinforcement learning signals predict future decisions. The Journal of Neuroscience, 27, 371-378. (Pubitemid 46106011)
7. Dehaene, S., Posner, M. I., & Tucker, D. M. (1994). Localization of a neural system for error detection and compensation. Psychological Science, 5, 303-305.
8. Delgado, M. R., Miller, M. M., Inati, S., & Phelps, E. A. (2005). An fMRI study of reward-related probability learning. Neuroimage, 24, 862-873. (Pubitemid 40094568)
9. Delorme, A., & Makeig, S. (2004). EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. Journal of Neuroscience Methods, 134, 9-21. (Pubitemid 38224183)
10. Donkers, F. C. L., Nieuwenhuis, S., & van Boxtel, G. J. M. (2005). Mediofrontal negativities in the absence of responding. Cognitive Brain Research, 25, 777-787. (Pubitemid 41774443)
11. Duncan-Johnson, C. C., & Donchin, E. (1977). On quantifying surprise: The variation of event-related potentials with subjective probability. Psychophysiology, 14, 456-467. (Pubitemid 8190202)
12. Falkenstein, M., Hohnsbein, J., Hoormann, J., & Blanke, L. (1991). Effects of crossmodal divided attention on late ERP components: II. Error processing in choice reaction tasks. Electroencephalography and Clinical Neurophysiology, 78, 447-455.
13. Fu, W. T., & Anderson, J. R. (2006). From recurrent choice to skill learning: A reinforcement-learning model. Journal of Experimental Psychology: General, 135, 184-206. (Pubitemid 43883042)
14. Gehring, W. J., Goss, B., Coles, M. G. H., Meyer, D. E., & Donchin, E. (1993). A neural system for error detection and compensation. Psychological Science, 4, 385-390.
15. Gehring, W. J., & Willoughby, A. R. (2002). The medial frontal cortex and the rapid processing of monetary gains and losses. Science, 295, 2279-2282. (Pubitemid 34258845)
16. Gehring, W. J., & Willoughby, A. R. (2004). Are all medial frontal negativities created equal? Toward a richer empirical basis for theories of action monitoring. In M. Ullsperger & M. Falkenstein (Eds.), Errors, conflicts, and the brain: Current opinions on performance monitoring (pp. 14-20). Leipzig: Max Planck Institute of Cognitive Neuroscience.
17. Gureckis, T. M., & Love, B. C. (2009). Short-term gains, long-term pains: How cues about state aid learning in dynamic environments. Cognition, 113, 293-313.
18. Hajcak, G., Holroyd, C. B., Moser, J. S., & Simons, R. F. (2005). Brain potentials associated with expected and unexpected good and bad outcomes. Psychophysiology, 42, 161-170. (Pubitemid 40529868)
19. Hajcak, G., Moser, J. S., Holroyd, C. B., & Simons, R. F. (2006). The feedback-related negativity reflects the binary evaluation of good versus bad outcomes. Biological Psychology, 71, 148-154. (Pubitemid 43082242)
20. Hajcak, G., Moser, J. S., Holroyd, C. B., & Simons, R. F. (2007). It's worse than you thought: The feedback negativity and violations of reward prediction in gambling tasks. Psychophysiology, 44, 905-912. (Pubitemid 47609612)
21. Haruno, M., Kuroda, T., Doya, K., Toyama, K., Kimura, M., Samejima, K., et al. (2004). A neural correlate of reward-based behavioral learning in caudate nucleus: A functional magnetic resonance imaging study of a stochastic decision task. The Journal of Neuroscience, 24, 1660-1665. (Pubitemid 38240962)
22. Holroyd, C. B., & Coles, M. G. H. (2002). The neural basis of human error processing: Reinforcement learning, dopamine, and the error-related negativity. Psychological Review, 109, 679-709.
23. Holroyd, C. B., & Krigolson, O. E. (2007). Reward prediction error signals associated with a modified time estimation task. Psychophysiology, 44, 913-917. (Pubitemid 47609608)
24. Holroyd, C. B., Krigolson, O. E., Baker, R., Lee, S., & Gibson, J. (2009). When is an error not a prediction error? An electrophysiological investigation. Cognitive, Affective & Behavioral Neuroscience, 9, 59-70.
25. Holroyd, C. B., Larsen, J. T., & Cohen, J. D. (2004). Context dependence of the event-related brain potential associated with reward and punishment. Psychophysiology, 41, 245-253. (Pubitemid 38524096)
26. Holroyd, C. B., Nieuwenhuis, S., Yeung, N., Nystrom, L., Mars, R. B., Coles, M. G. H., et al. (2004). Dorsal anterior cingulate cortex shows fMRI response to internal and external error signals. Nature Neuroscience, 7, 497-498. (Pubitemid 38544587)
27. Hull, C. L. (1943). Principles of behavior: An introduction to behavioral theory. New York: Appleton-Century-Crofts.
28. Ito, S., Stuphorn, V., Brown, J. W., & Schall, J. D. (2003). Performance monitoring by the anterior cingulate cortex during saccade countermanding. Science, 302, 120-122. (Pubitemid 37210702)
29. Jennings, J. R., & Wood, C. C. (1976). The ε-adjustment procedure for repeated-measures analyses of variance. Psychophysiology, 13, 277-278.
30. Kennerley, S. W., Walton, M. E., Behrens, T. E. J., Buckley, M. J., & Rushworth, M. F. S. (2006). Optimal decision making and the anterior cingulate cortex. Nature Neuroscience, 9, 940-947. (Pubitemid 43980509)
31. Kopp, B., Rist, F., & Mattler, U. (1996). N200 in the flanker task as a neurobehavioral tool for investigating executive control. Psychophysiology, 33, 282-294. (Pubitemid 26127156)
32. Kotani, Y., Hiraku, S., Suda, K., & Aihara, Y. (2001). Effect of positive and negative emotion on stimulus-preceding negativity prior to feedback stimuli. Psychophysiology, 38, 873-878. (Pubitemid 33108353)
33. Krigolson, O. E., Pierce, L. J., Holroyd, C. B., & Tanaka, J. W. (2009). Learning to become an expert: Reinforcement learning and the acquisition of perceptual expertise. Journal of Cognitive Neuroscience, 21, 1834-1841.
34. Luck, S. J. (2005). An introduction to the event-related potential technique. Cambridge: MIT Press.
35. Martin, L. E., Potts, G. F., Burton, P. C., & Montague, P. R. (2009). Electrophysiological and hemodynamic responses to reward prediction violation. NeuroReport, 20, 1140-1143.
36. Michie, D. (1963). Experiments on the mechanization of game learning: Part 1. Characterization of the model and its parameters. Computer Journal, 6, 232-236.
37. Miltner, W. H. R., Braun, C. H., & Coles, M. G. H. (1997). Eventrelated brain potentials following incorrect feedback in a timeestimation task: Evidence for a "generic" neural system for error detection. Journal of Cognitive Neuroscience, 9, 788-798. (Pubitemid 28025194)
38. Minsky, M. (1963). Steps toward artificial intelligence. In E. A. Feigenbaum & J. Feldman (Eds.), Computers and thought (pp. 406-450). New York: McGraw-Hill.
39. Montague, P. R., Dayan, P., & Sejnowski, T. J. (1996). A framework for mesencephalic dopamine systems based on predictive hebbian learning. The Journal of Neuroscience, 16, 1936-1947. (Pubitemid 26145969)
40. Morris, G., Nevet, A., Arkadir, D., Vaadia, E., & Bergman, H. (2006). Midbrain dopamine neurons encode decisions for future action. Nature Neuroscience, 9, 1057-1063. (Pubitemid 44264919)
41. Nelder, J. A., & Mead, R. (1965). A simplex method for function minimization. Computer Journal, 7, 308-313.
42. Nieuwenhuis, S., Holroyd, C. B., Mol, N., & Coles, M. G. H. (2004).Reinforcement-related brain potentials from medial frontal cortex: Origins and functional significance. Neuroscience and Biobehavioral Reviews, 28, 441-448. (Pubitemid 39077932)
43. Nieuwenhuis, S., Ridderinkhof, K. R., Talsma, D., Coles, M. G. H., Holroyd, C. B., Kok, A., et al. (2002). A computational account of altered error processing in older age: Dopamine and the errorrelated negativity. Cognitive, Affective & Behavioral Neuroscience, 2, 19-36. (Pubitemid 38391249)
44. Niki, H., & Watanabe, M. (1979). Prefrontal and cingulate unit activity during timing behavior in the monkey. Brain Research, 171, 213-224. (Pubitemid 10246665)
45. O'Doherty, J. P., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004). Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science, 304, 452-454. (Pubitemid 38495943)
46. Ohgami, Y., Kotani, Y., Hiraku, S., Aihara, Y., & Ishii, M. (2004). Effects of reward and stimulus modality on stimulus-preceding negativity. Psychophysiology, 41, 729-738. (Pubitemid 39295499)
47. Pan, W. X., Schmidt, R., Wickens, J. R., & Hyland, B. I. (2005). Dopamine cells respond to predicted events during classical conditioning: Evidence for eligibility traces in the rewardlearning network. The Journal of Neuroscience, 25, 6235-6242. (Pubitemid 40923490)
48. Potts, G. F., Martin, L. E., Kamp, S. M., & Donchin, E. (2011). Neural response to action and reward prediction errors: Comparing the error-related negativity to behavioral errors and the feedbackrelated negativity to reward prediction violations. Psychophysiology, 48, 218-228.
49. Ridderinkhof, K. R., Ullsperger, M., Crone, E. A., & Nieuwenhuis, S. (2004). The role of the medial frontal cortex in cognitive control. Science, 306, 443-447. (Pubitemid 39372436)
50. Roesch, M. R., Calu, D. J., & Schoenbaum, G. (2007). Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nature Neuroscience, 10, 1615-1624. (Pubitemid 350175882)
51. Rummery, G. A., & Niranjan, M. (1994). On-line Q-learning using connectionist systems (Tech. Rep. CUED/F-INFENG/TR166). Cambridge: Cambridge University.
52. Rushworth, M. F. S., Walton, M. E., Kennerley, S. W., & Bannerman, D. M. (2004). Action sets and decisions in the medial frontal cortex. Trends in Cognitive Sciences, 8, 410-417. (Pubitemid 39179904)
53. Schonberg, T., Daw, N. D., Joel, D., & O'Doherty, J. P. (2007). Reinforcement learning signals in the human striatum distinguish learners from nonlearners during reward-based decision making. The Journal of Neuroscience, 27, 12860-12867. (Pubitemid 350159584)
54. Schultz, W. (1998). Predictive reward signal of dopamine neurons. Journal of Neurophysiology, 80, 1-27. (Pubitemid 28326237)
55. Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275, 1593-1599. (Pubitemid 27120526)
56. Schwarz, G. E. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461-464.
57. Seymour, B., O'Doherty, J. P., Dayan, P., Koltzenburg, M., Jones, A. K., Dolan, R. J., et al. (2004). Temporal difference models describe higher-order learning in humans. Nature, 429, 664-667. (Pubitemid 38812318)
58. Shima, K., & Tanji, J. (1998). Role for cingulate motor area cells in voluntary movement selection based on reward. Science, 282, 1335-1338. (Pubitemid 28524499)
59. Squires, K. C., Wickens, C., Squires, N. K., & Donchin, E. (1976). The effect of stimulus sequence on the waveform of the cortical event-related potential. Science, 193, 1142-1146.
60. Sutton, R. S., & Barto, A. G. (1990). Time-derivative models of Pavlovian reinforcement. In M. Gabriel & J. Moore (Eds.), Learning and computational neuroscience: Foundations of adaptive networks (pp. 497-537). Cambridge: MIT Press.
61. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge: MIT Press.
62. Tesauro, G. J. (1992). Practical issues in temporal difference learning. Machine Learning, 8, 257-277.
63. van Veen, V., & Carter, C. S. (2002). The timing of action-monitoring processes in the anterior cingulate cortex. Journal of Cognitive Neuroscience, 14, 593-602. (Pubitemid 34553655)
64. Watkins, C. J. C. H., & Dayan, P. (1992). Q-learning. Machine Learning, 8, 279-292.
65. Widrow, B., Gupta, N. K., & Maitra, S. (1973). Punish/reward: Learning with a critic in adaptive threshold systems. IEEE Transactions on Systems, Man, and Cybernetics, 5, 455-465.