Temporal difference based actor critic learning - Convergence and neural implementation

Advances in Neural Information Processing Systems 21 - Proceedings of the 2008 Conference

Volumn , Issue , 2009, Pages 385-392

  Di Castro, Dotan ^a   Volkinshtein, Dmitry ^a   Meir, Ron ^a  

^a TECHNION ISRAEL INSTITUTE OF TECHNOLOGY   (Israel)

Author keywords

[No Author keywords available]

Indexed keywords

AMINES; BIOINFORMATICS; LEARNING ALGORITHMS; NEUROPHYSIOLOGY;

ACTOR CRITIC; ACTOR-CRITIC ALGORITHM; ACTOR-CRITIC LEARNING; CONVERGENCE PROPERTIES; DOPAMINE; FUNCTIONS APPROXIMATIONS; LEARNING CONVERGENCE; NEURAL IMPLEMENTATIONS; REINFORCEMENT LEARNINGS; TEMPORAL DIFFERENCES;

REINFORCEMENT LEARNING;

EID: 79959855306     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (24)

1. D. Baras and R. Meir. Reinforcement learning, spike time dependent plasticity and the bcm rule. Neural Comput., 19(8):22452279, 2007
2. J. Baxter and P.L. Bartlett. Hebbian synaptic modifications in spiking neurons that learn. (Technical rep.). Canberra: Research School of Information Sciences and Engineering, Australian National University, 1999.
3. J. Baxter and P.L. Bartlett. Infinite-Horizon Policy-Gradient Estimation. J. of Artificial Intelligence Research, 15:319-350, 2001.
4. D.P. Bertsekas. Dynamic Programming and Optimal Control, Vol I., 3rd Ed. Athena Scinetific, 2006.
5. S. Bhatnagar, R. Sutton, M. Ghavamzadeh, and M. Lee. Incremental natural actor-critic algorithms. In J.C. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems 20, pages 105-112. MIT Press, Cambridge, MA, 2008.
6. S. Bhatnagar, R.S. Sutton, M. Ghavamzadeh, and M. Lee. Natural actor-critic algorithms. Automatica, To appear, 2008.
7. V.S. Borkar. Stochastic approximation with two time scales. Syst. Control Lett., 29(5):291294, 1997.
8. P. Bremaud. Markov Chains: Gibbs Fields, Monte Carlo Simulation, and Queues. Springer, 1999.
9. R.V. Florian. Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural Computation, 19:14681502, 2007.
10. R.G. Gallager. Discrete Stochastic Processes. Kluwer Academic Publishers, 1995.
11. W. Gerstner and W.M. Kistler. Spinking Neuron Models. Cambridge University Press, Cambridge, 2002.
12. E.M. Izhikevich. Solving the Distal Reward Problem through Linkage of STDP and Dopamine Signaling. Cerebral Cortex, 17(10):2443-52, 2007. (Pubitemid 47517479)
13. V.R. Konda and J. Tsitsiklis. On actor critic algorithms. SIAM J. Control Optim., 42(4):11431166, 2003.
14. H.J. Kushner and G.G. Yin. Stochastic Approximation Algorithms and Applications. Springer, 1997.
15. P. Marbach and J. Tsitsiklis. Simulation-Based Optimization of Markov Reward Processes. IEEE. Trans. Auto. Cont., 46:191-209, 1998.
16. P.R. Montague, P. Dayan, and T.J. Sejnowski. A framework for mesencephalic dopamine systems based on predictive hebbian learning. Journal of Neuroscience, 16:19361947, 1996.
17. J. ODoherty, P. Dayan, J. Schultz, R. Deichmann, K. Friston, and R.J. Dolan. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science, 304:452454, 2004.
18. J.N.J. Reynolds and J.R.Wickens. Dopamine-dependent plasticity of corticostriatal synapses. Neural Networks, 15(4-6):507521, 2002.
19. S. Marom and G. Shahaf. Development, learning and memory in large random networks of cortical neurons: lessons beyond anatomy. Quarterly Reviews of Biophysics, 35:6387, 2002.
20. W. Schultz. Multiple reward signals in the brain. Nature Reviews Neuroscience, 1:199207, Dec. 2000.
21. S. Singh and P. Dayan. Analytical mean squared error curves for temporal difference learning. Machine Learning, 32:540, 1998.
22. R. S. Sutton and A. G. Barto. Reinforcement Learning. MIT Press, 1998.
23. R. Sutton, D. McAllester, S. Singh and Y. Mansour. Policy-Gradient Methods for Reinforcement Learning with Function Approximation. Advances in Neural Information Processing Systems, 12:1057-1063, 2000.
24. E.M. Tricomi, M.R. Delgado, and J.A. Fiez. Modulation of caudate activity by action contingency. Neuron, 41(2):281292, 2004.