Integration of reinforcement learning and optimal decision-making theories of the basal ganglia

Neural Computation

Volumn 23, Issue 4, 2011, Pages 817-851

  Bogacz, Rafal ^a   Larsen, Tobias ^a,b  

^a UNIVERSITY OF BRISTOL   (United Kingdom)

^b TRINITY COLLEGE DUBLIN   (Ireland)

Author keywords

[No Author keywords available]

Indexed keywords

ANIMAL; ARTICLE; ARTIFICIAL NEURAL NETWORK; BASAL GANGLION; COMPARATIVE STUDY; DECISION MAKING; DECISION THEORY; LEARNING; PHYSIOLOGY; REINFORCEMENT;

ANIMALS; BASAL GANGLIA; DECISION MAKING; DECISION THEORY; LEARNING; NEURAL NETWORKS (COMPUTER); REINFORCEMENT (PSYCHOLOGY);

EID: 79952300467     PISSN: 08997667     EISSN: 1530888X     Source Type: Journal    
DOI: 10.1162/NECO_a_00103     Document Type: Article

References (76)

1. Alexander, G. E., DeLong, M. R., & Strick, P. L. (1986). Parallel organization of functionally segregated circuits linking basal ganglia and cortex. Annu. Rev. Neurosci., 9, 357-381.
2. Ashby, F. G., Ennis, J. M., & Spiering, B. J. (2007). A neurobiological theory of automaticity in perceptual categorization. Psychological Review, 114, 632-656.
3. Atallah, H. E., Lopez-Paniagua, D., Rudy, J. W., & O'Reilly, R. C. (2007). Separate neural substrates for skill learning and performance in the ventral and dorsal striatum. Nat. Neurosci., 10(1), 126-131.
4. Baum, C. W., & Veeravalli, V. V. (1994). A sequential procedure for multihypothesis testing. IEEE Transactions on Information Theory, 40, 1996-2007.
5. Bogacz, R. (2009). Optimal decision-making theories. In J.-C. Dreher & L. Tremblay (Eds.), Handbook of reward and decision making. Orlando, FL: Academic Press.
6. Bogacz, R., Brown, E., Moehlis, J., Holmes, P., & Cohen, J. D. (2006). The physics of optimal decision making: A formal analysis of models of performance in twoalternative forced choice tasks. Psychological Review, 113, 700-765.
7. Bogacz, R., & Gurney, K. (2007). The basal ganglia and cortex implement optimal decision making between alternative actions. Neural Computation, 19, 442-477.
8. Bogacz, R., McClure, S. M., Li, J., Cohen, J. D., & Montague, P. R. (2007). Short-term memory traces for action bias in human reinforcement learning. Brain Res., 1153, 111-121.
9. Britten, K. H., Shadlen, M. N., Newsome, W. T., & Movshon, J. A. (1993). Responses of neurons in macaque MT to stochastic motion signals. Vis. Neurosci., 10(6), 1157-1169.
10. Brown, J. W., Bullock, D., & Grossberg, S. (2004). How laminar frontal cortex and basal ganglia circuits interact to control planned and reactive saccades. Neural Netw., 17(4), 471-510.
11. Carpenter, R. H., & Williams, M. L. (1995). Neural computation of log likelihood in control of saccadic eye movements. Nature, 377(6544), 59-62.
12. Chevalier, G., Vacher, S., Deniau, J. M., & Desban, M. (1985). Disinhibition as a basic process in the expression of striatal functions. I. The striato-nigral influence on tecto-spinal/tecto-diencephalic neurons. Brain Res., 334(2), 215-226.
13. Daw, N. D., Niv, Y., & Dayan, P. (2005). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci., 8(12), 1704-1711.
14. Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B., & Dolan, R. J. (2006). Cortical substrates for exploratory decisions in humans. Nature, 441, 876-879.
15. Dayan, P., & Daw, N. D. (2008). Decision theory, reinforcement learning, and the brain. Cogn. Affect. Behav. Neurosci., 8(4), 429-453.
16. Deniau, J.M.,&Chevalier, G. (1985).Disinhibition as a basic process in the expression of striatal functions. II. The striato-nigral influence on thalamocortical cells of the ventromedial thalamic nucleus. Brain Res., 334(2), 227-233.
17. Dimperio, E., Jessup, R. K., & Busemeyer, J. R. (2010). Integrating sophisticated choice models with basic learning processes to more fully account for complex choice behavior. Manuscript submitted for publication.
18. Doya, K. (2000). Complementary roles of basal ganglia and cerebellum in learning and motor control. Curr. Opin. Neurobiol., 10(6), 732-739.
19. Dragalin, V. P., Tertakovsky, A. G., & Veeravalli, V. V. (1999). Multihypothesis sequential probability ratio tests-art I: Asymptotic optimality. IEEE Transactions on Information Theory, 45, 2448-2461.
20. Frank, M. J. (2006). Hold your horses: A dynamic computational role for the subthalamic nucleus in decision making. Neural Netw., 19(8), 1120-1136.
21. Frank, M. J., & Claus, E. D. (2006). Anatomy of a decision: Striato-Orbitofrontal interactions in reinforcement learning, decision making, and reversal. Psychol. Rev., 113(2), 300-326.
22. Frank, M. J., Seeberger, L. C., & O'Reilly, R. C. (2004). By carrot or by stick: Cognitive reinforcement learning in Parkinsonism. Science, 306(5703), 1940-1943.
23. Gerfen, C. R. (1992). The neostriatal mosaic: Multiple levels of compartmental organization in the basal ganglia. Annu. Rev. Neurosci., 15, 285-320.
24. Gold, J. I., & Shadlen, M. N. (2001). Neural computations that underlie decisions about sensory stimuli. Trends Cogn. Sci., 5(1), 10-16.
25. Gold, J. I., & Shadlen, M. N. (2002). Banburismus and the brain: Decoding the relationship between sensory stimuli, decisions, and reward. Neuron, 36(2), 299-308.
26. Gold, J. I., & Shadlen, M. N. (2007). The neural basis of decision making. Annu. Rev. Neurosci., 30, 535-574.
27. Gurney, K., Humphries, M., Wood, R., Prescott, T. J., & Redgrave, P. (2004). Testing computational hypotheses of brain systems function: A case study with the basal ganglia. Network, 15(4), 263-290.
28. Gurney, K., Prescott, T. J., & Redgrave, P. (2001). A computational model of action selection in the basal ganglia. I. A new functional anatomy. Biol. Cybern., 84(6), 401-410.
29. Hallworth, N. E., Wilson, C. J., & Bevan, M. D. (2003). Apamin-sensitive small conductance calcium-activated potassium channels, through their selective coupling to voltage-gated calcium channels, are critical determinants of the precision, pace, and pattern of action potential generation in rat subthalamic nucleus neurons in vitro. J. Neurosci., 23(20), 7525-7542.
30. Hanks, T. D., Ditterich, J., & Shadlen, M. N. (2006). Microstimulation of macaque area LIP affects decision-making in a motion discrimination task. Nat. Neurosci., 9(5), 682-689.
31. Humphries,M.D., Stewart, R. D.,&Gurney, K. N. (2006).Aphysiologically plausible model of action selection and oscillatory activity in the basal ganglia. J. Neurosci., 26(50), 12921-12942.
32. Jessup, R. K., Bishara, A. J., & Busemeyer, J. R. (2008). Feedback produces divergence from prospect theory in descriptive choice. Psychological Science, 19, 1015-1022.
33. Kaelbling, L. P., Littman, M. L., & Cassandra, A. C. (1998). Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101, 99-134.
34. Kakade, S., & Dayan, P. (2002). Dopamine: Generalization and bonuses. Neural Networks, 15, 549-559.
35. Kiani, R., & Shadlen, M. N. (2009). Representation of confidence associated with a decision by neurons in the parietal cortex. Science, 324(5928), 759-764.
36. Larsen, T., Leslie, D. S., Collins, E. J., & Bogacz, R. (2010). Posterior weighted reinforcement learning with stimulus uncertainties. Neural Computation, 22, 1149-1179.
37. Law, C. T., & Gold, J. I. (2008). Neural correlates of perceptual learning in a sensorymotor, but not a sensory, cortical area. Nat. Neurosci., 11(4), 505-513.
38. Law, C. T., & Gold, J. I. (2009). Reinforcement learning can account for associative and perceptual learning on a visual-decision task. Nat. Neurosci., 12(5), 655-663.
39. Lo, C. C.,&Wang, X. J. (2006). Cortico-basal ganglia circuitmechanism for a decision threshold in reaction time task. Nat. Neurosci., 9, 956-963.
40. Mazurek, M. E., Roitman, J. D., Ditterich, J.,&Shadlen, M. N. (2003).Arole for neural integrators in perceptual decision making. Cereb. Cortex, 13(11), 1257-1269.
41. McClure, S. M., Daw, N. D., &Montague, P. R. (2003). A computational substrate for incentive salience. Trends Neurosci., 26(8), 423-428.
42. McMillen, T., & Holmes, P. (2006). The dynamics of choice among multiple alternatives. Journal of Mathematical Psychology, 50, 30-57.
43. Montague, P. R., Dayan, P., & Sejnowski, T. J. (1996). A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci., 16(5), 1936-1947.
44. Nambu, A., & Llinas, R. (1994). Electrophysiology of globus pallidus neurons in vitro. J. Neurophysiol., 72(3), 1127-1139.
45. O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004). Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science, 304(5669), 452-454.
46. Parent, A., & Smith, Y. (1987). Organization of efferent projections of the subthalamic nucleus in the squirrelmonkey as revealed by retrograde labelingmethods. Brain Res., 436(2), 296-310.
47. Pasupathy, A., & Miller, E. K. (2005). Different time courses of learning-related activity in the prefrontal cortex and striatum. Nature, 433(7028), 873-876.
48. Platt, M. L., & Glimcher, P. W. (1999). Neural correlates of decision variables in parietal cortex. Nature, 400, 233-238.
49. Ratcliff, R. (1978). A theory of memory retrieval. Psychol. Rev., 83, 59-108.
50. Ratcliff, R. (2006). Modeling response signal and response time data. Cognitive Psychology, 53, 195-237.
51. Ratcliff, R., Gomez, R.,&McKoon,G. (2004).Adiffusionmodel account of the lexical decision task. Psychological Review, 111, 159-182.
52. Ratcliff, R., & Smith, P. L. (2004). Comparison of sequential sampling models for two-choice reaction time. Psychological Review, 111(2), 333-367.
53. Redgrave, P., Prescott, T. J., & Gurney, K. (1999). The basal ganglia: A vertebrate solution to the selection problem? Neuroscience, 89(4), 1009-1023.
54. Reynolds, J. N., Hyland, B. I., & Wickens, J. R. (2001). A cellular mechanism of reward-related learning. Nature, 413(6851), 67-70.
55. Roesch, M. R., & Olson, C. R. (2004). Neuronal activity related to reward value and motivation in primate frontal cortex. Science, 304(5668), 307-310.
56. Roitman, J. D., & Shadlen, M. N. (2002). Response of neurons in the lateral intraparietal area during a combined visual discrimination reaction time task. J. Neurosci., 22(21), 9475-9489.
57. Samejima, K., Ueda, Y., Doya, K., & Kimura, M. (2005). Representation of actionspecific reward values in the striatum. Science, 310(5752), 1337-1340.
58. Schall, J. D. (2001). Neural basis of deciding, choosing and acting. Nat. Rev. Neurosci., 2(1), 33-42.
59. Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275(5306), 1593-1599.
60. Shadlen, M. N.,&Newsome, W. T. (2001).Neural basis of a perceptual decision in the parietal cortex (area LIP) of the rhesus monkey. J. Neurophysiol., 86(4), 1916-1936.
61. Shea-Brown, E., Gilzenrat, M. S., & Cohen, J. D. (2008). Optimization of decision making in multilayer networks: The role of locus coeruleus. Neural Comput., 20(12), 2863-2894.
62. Simen, P. A., Buck, C.,Hu, P.,Holmes, P.,&Cohen, J. D. (2009). Reward rate optimization in two-alternative decision making: empirical tests of theoretical predictions. Journal of Experimental Psychology: Human Performance and Psychophysics, 35, 1865-1897.
63. Simen, P. A., Cohen, J. D., & Holmes, P. (2006). Rapid decision thresholdmodulation by reward rate in a neural network. Neural Networks, 19, 1013-1026.
64. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning. Cambridge, MA: MIT Press.
65. Tobler, P. N., Fiorillo, C. D., & Schultz, W. (2005). Adaptive coding of reward value by dopamine neurons. Science, 307(5715), 1642-1645.
66. Ungless, M. A., Magill, P. J., & Bolam, J. P. (2004). Uniform inhibition of dopamine neurons in the ventral tegmental area by aversive stimuli. Science, 303(5666), 2040-2042.
67. Usher, M.,&McClelland, J. L. (2001). The time course of perceptual choice: The leaky, competing accumulator model. Psychol. Rev., 108(3), 550-592.
68. Vandekerckhove, J., & Tuerlinckx, F. (2007). Fitting the Ratcliff diffusion model to experimental data. Psychonomic Bulletin and Review, 14, 1011-1026.
69. Wald, A. (1947). Sequential analysis. Hoboken, NJ:Wiley.
70. Wald, A., & Wolfowitz, J. (1948). Optimum character of the sequential probability ratio test. Annals of Mathematical Statistics, 19, 326-339.
71. Wallis, J. D. (2007). Orbitofrontal cortex and its contribution to decision-making. Annu. Rev. Neurosci., 30, 31-56.
72. Wallis, J. D., & Miller, E. K. (2003). From rule to response: Neuronal processes in the premotor and prefrontal cortex. J. Neurophysiol., 90(3), 1790-1806.
73. Wang, X. J. (2002). Probabilistic decision making by slow reverberation in cortical circuits. Neuron, 36(5), 955-968.
74. Wilson, C. J., Weyrick, A., Terman, D., Hallworth, N. E., & Bevan, M. D. (2004). A model of reverse spike frequency adaptation and repetitive firing of subthalamic nucleus neurons. J. Neurophysiol., 91(5), 1963-1980.
75. Yu, A. J., & Dayan, P. (2005). Uncertainty, neuromodulation, and attention. Neuron, 46(4), 681-692.
76. Zhang, J., & Bogacz, R. (2010). Optimal decision making on the basis of evidence represented in spike trains. Neural Computation, 22, 1113-1148.