Efficient reinforcement learning: computational theories, neuroscience and robotics

Current Opinion in Neurobiology

Volumn 17, Issue 2, 2007, Pages 205-212

  Kawato, Mitsuo ^a   Samejima, Kazuyuki ^b  

^a ADVANCED TELECOMMUNICATIONS RESEARCH INSTITUTE INTERNATIONAL   (Japan)

^b TAMAGAWA UNIVERSITY   (Japan)

Author keywords

[No Author keywords available]

Indexed keywords

BASAL GANGLION; BRAIN CORTEX; BRAIN REGION; BRAIN STEM; CEREBELLUM; CONCEPTUAL FRAMEWORK; EXPERIMENTAL STUDY; HUMAN; LEARNING; LEARNING THEORY; MATHEMATICAL COMPUTING; NEUROSCIENCE; NONHUMAN; PRIORITY JOURNAL; PSYCHOLOGICAL THEORY; REINFORCEMENT; REVIEW; ROBOTICS;

ANIMALS; HUMANS; NEURAL NETWORKS (COMPUTER); NEUROSCIENCES; REINFORCEMENT (PSYCHOLOGY); ROBOTICS;

EID: 34147191094     PISSN: 09594388     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.conb.2007.03.004     Document Type: Review

References (71)

1. Barto A.G., Sutton R.S., and Anderson C.W. Neuron-like elements that can solve difficult learning control problems. IEEE Trans Syst Man Cybern 13 (1983) 835-846
2. Sutton R.S., and Barto A.G. Reinforcement learning (1998), The MIT Press
3. Schultz W., Dayan P., and Montague P.R. A neural substrate of prediction and reward. Science 275 (1997) 1593-1599
4. Schultz W., and Dickinson A. Neuronal coding of prediction errors. Annu Rev Neurosci 23 (2000) 473-500
5. Houk JC, Adams JL, Barto AG. Models of information processing in the basal ganglia. Edited by Houk JC, Davis JL, Beiser DG. The MIT Press; 1995.
6. Doya K. Complementary roles of basal ganglia and cerebellum in learning and motor control. Curr Opin Neurobiol 10 (2000) 732-739
7. Satoh T., Nakai S., Sato T., and Kimura M. Correlated coding of motivation and outcome of decision by dopamine neurons. J Neurosci 23 (2003) 9913-9923
8. Takikawa Y., Kawagoe R., and Hikosaka O. A possible role of midbrain dopamine neurons in short- and long-term adaptation of saccades to position-reward mapping. J Neurophysiol 92 (2004) 2520-2529
9. Fiorillo C.D., Tobler P.N., and Schultz W. Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299 (2003) 1898-1902
10. Tobler P.N., Fiorillo C.D., and Schultz W. Adaptive coding of reward value by dopamine neurons. Science 307 (2005) 1642-1645. When a monkey receives unexpected reward, phasic change of activity of dopaminergic neurons can be explained as encoding the temporal-difference error. These authors added new findings regarding the scaling of this coding: this error signal is scaled by variance of reward distribution within a given context. Thus, the activity of dopaminergic neurons is scaled by the range of reward distribution, and could represent expected 'risk' of reward in the context.
11. Morris G., Nevet A., Arkadir D., Vaadia E., and Bergman H. Midbrain dopamine neurons encode decisions for future action. Nat Neurosci 9 (2006) 1057-1063
12. Wickens J.R., Reynolds J.N., and Hyland B.I. Neural mechanisms of reward-related motor learning. Curr Opin Neurobiol 13 (2003) 685-690
13. Kawagoe R., Takikawa Y., and Hikosaka O. Reward-predicting activity of dopamine and caudate neurons - a possible mechanism of motivational control of saccadic eye movement. J Neurophysiol 91 (2004) 1013-1024
14. 2 receptors enhanced it. The dopamine-dependent plasticity in corticostriatal synapses could modulate sensorimotor learning dependent on reward.
15. Cromwell H.C., Hassani O.K., and Schultz W. Relative reward processing in primate striatum. Exp. Brain Res 162 (2005) 520-525
16. Samejima K., Ueda Y., Doya K., and Kimura M. Representation of action-specific reward values in the striatum. Science 310 (2005) 1337-1340. This study demonstrates that a considerable proportion of dorsal striatal neural activity represents the 'action value' predicted by Doya [6]. The authors then directly compared striatal neuronal activities with dynamically changing model parameters in a trial-by-trial manner, using a sophisticated estimation method, and found a good fit.
17. Morris G., Arkadir D., Nevet A., Vaadia E., and Bergman H. Coincident but distinct messages of midbrain dopamine and striatal tonically active neurons. Neuron 43 (2004) 133-143
18. Williams Z.M. Eskandar EN: Selective enhancement of associative learning by microstimulation of the anterior caudate. Nat Neurosci 9 (2006) 562-568. This paper provides direct evidence that the striatum contributes to stimulus-action-reward association learning. The authors recorded caudate neuronal discharge that is temporarily reinforced during learning. They also applied electrical microstimulation to the caudate nucleus while the monkey learned arbitrary association between a visual stimulus and movement guided by reward feedback, and the microstimulation selectively enhanced the association learning.
19. Pasupathy A., and Miller E.K. Different time courses of learning-related activity in the prefrontal cortex and striatum. Nature 433 (2005) 873-876
20. Matsumoto K., Suzuki W., and Tanaka K. Neuronal correlates of goal-based motor selection in the prefrontal cortex. Science 301 (2003) 229-232
21. Barraclough D.J., Conroy M.L., and Lee D. Prefrontal cortex and decision making in a mixed-strategy game. Nat Neurosci 7 (2004) 404-410
22. Sugrue L.P., Corrado G.S., and Newsome W.T. Matching behavior and the representation of value in the parietal cortex. Science 304 (2004) 1782-1787
23. Sugrue L.P., Corrado G.S., and Newsome W.T. Choosing the greater of two goods: neural currencies for valuation and decision making. Nat Rev Neurosci 6 (2005) 363-375
24. Lau B., and Glimcher P.W. Dynamic response-by-response models of matching behavior in rhesus monkeys. J Exp Anal Behav 84 (2005) 555-579
25. Corrado G.S., Sugrue L.P., Seung H.S., and Newsome W.T. Linear-nonlinear-Poisson models of primate choice dynamics. J Exp Anal Behav 84 (2005) 581-617
26. Sakai Y., Okamoto H., and Fukai T. Computational algorithms and neuronal network models underlying decision processes. Neural Netw 19 (2006) 1091-1105. This review provides computational explanations of perceptual decision and matching behavior, which have been studied in the field of behavioral psychology, and their relationship to optimal learning theories and algorithms of reinforcement learning.
27. Montague P.R., King-Casas B., and Cohen J.D. Imaging valuation models in human choice. Annu Rev Neurosci 29 (2006) 417-448
28. Pessiglione M., Seymour B., Flandin G., Dolan R.J., and Frith C.D. Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature 442 (2006) 1042-1045. Using combinations of pharmacological, imaging and computational modeling techniques, the authors directly demonstrate that fMRI signals in the striatum and orbitofrontal cortex are modulated by dopamine.
29. Seymour B., O'Doherty J.P., Dayan P., Koltzenburg M., Jones A.K., Dolan R.J., Friston K.J., and Frackowiak R.S. Temporal difference models describe higher-order learning in humans. Nature 429 (2004) 664-667
30. O'Doherty J., Dayan P., Schultz J., Deichmann R., Friston K., and Dolan R.J. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304 (2004) 452-454
31. Tobler P.N., O'Doherty J.P., Dolan R.J., and Schultz W. Human neural learning depends on reward prediction errors in the blocking paradigm. J Neurophysiol 95 (2006) 301-310
32. Haruno M., Kuroda T., Doya K., Toyama K., Kimura M., Samejima K., Imamizu H., and Kawato M. A neural correlate of reward-based behavioral learning in caudate nucleus: a functional magnetic resonance imaging study of a stochastic decision task. J Neurosci 24 (2004) 1660-1665
33. Tanaka S.C., Doya K., Okada G., Ueda K., Okamoto Y., and Yamawaki S. Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nat Neurosci 7 (2004) 887-893
34. Tanaka S.C., Samejima K., Okada G., Ueda K., Okamoto Y., Yamawaki S., and Doya K. Brain mechanism of reward prediction under predictable and unpredictable environmental dynamics. Neural Netw 19 (2006) 1233-1241
35. Preuschoff K., Bossaerts P., and Quartz S.R. Neural differentiation of expected reward and risk in human subcortical structures. Neuron 51 (2006) 381-390
36. Hsu M., Bhatt M., Adolphs R., Tranel D., and Camerer C.F. Neural systems responding to degrees of uncertainty in human decision-making. Science 310 (2005) 1680-1683
37. Haruno M., and Kawato M. Different neural correlates of reward expectation and reward expectation error in the putamen and caudate nucleus during stimulus-action-reward association learning. J Neurophysiol 95 (2006) 948-959. This study is an example of recent research trends of computational-model-based neuroimaging. Subjects' choice behavior was modeled by a simple reinforcement learning algorithm (Q-learning) while the model was given the same sensory stimuli and received the same rewards. The model reproduced subjects' behavior reasonably well. Internal representations within the model, such as the reward expectation and the reward expectation error, were found to correlate differentially with activities in the putamen and caudate, respectively.
38. Hampton A.N., Bossaerts P., and O'Doherty J.P. The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. J Neurosci 26 (2006) 8360-8367
39. Schweighofer N., and Doya K. Meta-learning in reinforcement learning. Neural Netw 16 (2003) 5-9
40. Doya K. Metalearning and neuromodulation. Neural Netw 15 (2002) 495-506
41. Kennerley S.W., Walton M.E., Behrens T.E., Buckley M.J., and Rushworth M.F. Optimal decision making and the anterior cingulate cortex. Nat Neurosci 9 (2006) 940-947. This study shows that the anterior cingulate cortex has an important role in utilizing integrated past history of action and reward experiences for reward-based decision making. The authors demonstrate that lesion of the monkey anterior cingulate cortex did not impair performance just after error trials, but it rendered the monkeys unable to maintain optimal choice. The authors also found that in a matching task, monkeys that had lesions of the anterior cingulate cortex took longer to attain the optimum choice ratio.
42. Ishii S., Yoshida W., and Yoshimoto J. Control of exploitation-exploration meta-parameter in reinforcement learning. Neural Netw 15 (2002) 665-687
43. Nakamura K. Neural representation of information measure in the primate premotor cortex. J Neurophysiol 96 (2006) 478-485
44. Yoshida W., and Ishii S. Resolution of uncertainty in prefrontal cortex. Neuron 50 (2006) 781-789. In many situations, we face difficulty finding ascertaining where we are from a limited set of available sensory inputs. The problem of finding an optimal strategy with only partial information of a current state is called a 'partially observable Markov decision process'. The authors modeled how humans resolve this problem in a task of pathfinding through a maze of which there was only a limited view. The anterior prefrontal BOLD signal was found to correlate with the uncertainty of a current belief state.
45. Daw N.D., O'Doherty J.P., Dayan P., Seymour B., and Dolan R.J. Cortical substrates for exploratory decisions in humans. Nature 441 (2006) 876-879
46. Daw N.D., Niv Y., and Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 8 (2005) 1704-1711. The authors propose an interesting computational model consisting of two parallel reinforcement learning modules that could be implemented in prefrontal and subcortical basal ganglia circuits. The prefrontal circuit implements 'model-based reinforcement learning', whereas the corticobasal ganglia circuit including dorsolateral striatum implements 'model-free reinforcement learning'. The former achieves goal-directed behaviors using a tree-search algorithm by simulating the action and state transitions even before actual execution of the action. By contrast, the latter maintains a 'cash' of action value for each state, and it is updated by feedback of reward, step by step through actual action execution.
47. Redish A.D. Addiction as a computational process gone awry. Science 306 (2004) 1944-1947
48. Schoenbaum G., Roesch M.R., and Stalnaker T.A. Orbitofrontal cortex, decision-making and drug addiction. Trends Neurosci 29 (2006) 116-124
49. Roesch M.R., Taylor A.R., and Schoenbaum G. Encoding of time-discounted rewards in orbitofrontal cortex is independent of value representation. Neuron 51 (2006) 509-520
50. Rudebeck P.H., Walton M.E., Smyth A.N., Bannerman D.M., and Rushworth M.F. Separate neural pathways process different decision costs. Nat Neurosci 9 (2006) 1161-1168. This rat lesion study shows that two different types of decision costs - the effort cost and the temporal cost in waiting reward - are represented by different prefrontal circuits - the anterior cingulate cortex and orbitofrontal cortex, respectively. A T-maze with an obstacle in the goal arm and temporal delay for reward delivery were used to examine the effort and temporal costs, respectively. Anterior cingulate cortex lesions affected how much effort rats decided to invest for reward. Orbitofrontal cortex lesions affected how long rats decided to wait for reward. This study is very important in understanding how decision is made depending not only on reward but also on multiple types of cost.
51. Lee D., Conroy M.L., McGreevy B.P., and Barraclough D.J. Reinforcement learning and decision making in monkeys during a competitive game. Brain Res Cogn Brain Res 22 (2004) 45-58
52. Soltani A., Lee D., and Wang X.J. Neural mechanism for stochastic behaviour during a competitive game. Neural Netw 19 (2006) 1075-1090
53. Dorris M.C., and Glimcher P.W. Activity in posterior parietal cortex is correlated with the relative subjective desirability of action. Neuron 44 (2004) 365-378
54. Padoa-Schioppa C., and Assad J.A. Neurons in the orbitofrontal cortex encode economic value. Nature 441 (2006) 223-226
55. Singh S. Transfer of learning by composing solutions of elemental sequential tasks. Mach Learn 8 (1992) 323-339
56. Wiering M., and Schmidhuber J. HQ-learning. Adapt Behav 6 (1997) 219-246
57. Parr R., and Russell S. Reinforcement learning with hierarchies of machines. Advances in Neural Information Processing Systems vol 10 (1997), MIT press 1043-1049
58. Sutton R.S. Planning by incremental dynamic programming. Eighteenth International Workshop on Machine Learning; San Mateo, CA (1991), Morgan Kaufmann 353-357
59. Sutton R.S., Precup D., and Singh S. Between MDPs and semi-MDPs: a framework for temporal abstraction in reinforcement learning. Artif Intell 112 (1999) 181-211
60. Morimoto J., and Doya K. Hierarchical reinforcement learning for motion learning: learning 'stand up' trajectories. Adv Robot 13 (1999) 267-268
61. Samejima K., Doya K., and Kawato M. Inter-module credit assignment in modular reinforcement learning. Neural Netw 16 (2003) 985-994
62. Wolpert D.M., and Kawato M. Multiple paired forward and inverse models for motor control. Neural Netw 11 (1998) 1317-1329
63. Kawato M. Internal models for motor control and trajectory planning. Curr Opin Neurobiol 9 (1999) 718-727
64. Haruno M., Wolpert D.M., and Kawato M. Mosaic model for sensorimotor learning and control. Neural Comput 13 (2001) 2201-2220
65. Doya K., Samejima K., Katagiri K., and Kawato M. Multiple model-based reinforcement learning. Neural Comput 14 (2002) 1347-1369
66. Mena-Segovia J., Bolam J.P., and Magill P.J. Pedunculopontine nucleus and basal ganglia: distant relatives or part of the same family?. Trends Neurosci 27 (2004) 585-588
67. Kobayashi Y, Okada K, Inoue Y. Reward predicting activity of pedunculopontine tegmental nucleus neurons during visually guided saccade tasks. In 2002 Abstract Viewer and Itinerary Planner Online (http://sfn.scholarone.com/). Society for Neuroscience; 2002: Program No. 890.5.
68. Haruno M., and Kawato M. Heterarchical reinforcement-learning model for integration of multiple cortico-striatal loops: fMRI examination in stimulus-action-reward association learning. Neural Netw 19 (2006) 1242-1254. In this study, a heterarchical reinforcement learning model is proposed and supporting fMRI data are presented. The interplay between the model and the experiments could resolve theoretical difficulties of the plain reinforcement learning algorithm.
69. Haber S.N. The primate basal ganglia: parallel and integrative networks. J Chem Neuroanat 26 (2003) 317-330
70. Kobayashi Y., Inoue Y., Yamamoto M., Isa T., and Aizawa H. Contribution of pedunculopontine tegmental nucleus neurons to performance of visually guided saccade tasks in monkeys. J Neurophysiol 88 (2002) 715-731
71. Ng A.Y., Harada D., and Russell S. Policy invariance under reward transformations: theory and application to reward shaping. Proceedings of the Sixteenth International Conference on Machine Learning: 1999 (1999), Morgan Kaufmann 278-287