Autonomous reinforcement learning with experience replay

Neural Networks

Volumn 41, Issue , 2013, Pages 156-167

  Wawrzyński, Paweł ^a   Tanwani, Ajay Kumar ^b  

^a WARSAW UNIVERSITY OF TECHNOLOGY   (Poland)

^b EPFL   (Switzerland)

Author keywords

Actor critic; Autonomous learning; Reinforcement learning; Step size estimation

Indexed keywords

ACTOR CRITIC; AUTONOMOUS LEARNING; CONTROL POLICY; CONTROL TASK; FIXED-POINT ALGORITHMS; LEARNING CONTROL; ON-LINE NEURAL NETWORKS; STEP SIZE;

LEARNING ALGORITHMS; NEURAL NETWORKS;

REINFORCEMENT LEARNING;

ARTICLE; ARTIFICIAL NEURAL NETWORK; CALCULATION; LEARNING ALGORITHM; MATHEMATICAL COMPUTING; PRIORITY JOURNAL; PROBABILITY; PROCESS OPTIMIZATION; SIMULATION;

ALGORITHMS; ANIMALS; ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; MARKOV CHAINS; MODELS, THEORETICAL; MOVEMENT; NEURAL NETWORKS (COMPUTER); OCTOPODIFORMES; PROBLEM SOLVING; REINFORCEMENT (PSYCHOLOGY); STOCHASTIC PROCESSES;

EID: 84875884428     PISSN: 08936080     EISSN: 18792782     Source Type: Journal    
DOI: 10.1016/j.neunet.2012.11.007     Document Type: Article

References (39)

1. Abbeel P., Ng A.Y. Exploration and apprenticeship learning in reinforcement learning. Proc. of the 22nd ICML 2005, 1-8. ACM.
2. Adam S., Busoniu L., Babuska R. Experience replay for real-time reinforcement learning control. IEEE Transactions on Systems, Man, and Cybernetics, Part C 2012, 42(2):201-212.
3. Barto A.G., Sutton R.S., Anderson C.W. Neuronlike adaptive elements that can learn difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics 1983, 13:834-846.
4. Behera L., Kumar S., Patnaik A. On adaptive learning rate that guarantees convergence in feedforward networks. IEEE Transactions on Neural Networks 2006, 17(5):1116-1125.
5. Bhatnagar S., Sutton R., Ghavamzadeh M., Lee M. Natural actor-critic algorithms. Automatica 2009, 45:2471-2482.
6. Chiaverini S., Oriolo G., Walker I.D. Kinematically redundant manipulators. Springer handbook of robotics 2008, 245-268.
7. Cichosz P. An analysis of experience replay in temporal difference learning. Cybernetics and Systems 1999, 30:341-363.
8. Engel, Y., Szabó, P., & Volkinshtein, D. (2005). Learning to control an octopus arm with gaussian process temporal difference methods. In NIPS.
9. George A.P., Powell W.B. Adaptive stepsizes for recursive estimation with applications in approximate dynamic programming. Machine Learning 2006, 65(1):167-198.
10. Hachiya H., Peters J., Sugiyama M. Reward-weighted regression with sample reuse for direct policy search in reinforcement learning. Neural Computation 2011, 23(11):2798-2832.
11. Jacobs R.A. Increased rates of convergence through learning rate adaptation. Neural Networks 1988, 1(4):295-308.
12. Kathirvalavakumar T., Subavathi S.J. Neighborhood based modified backpropagation algorithm using adaptive learning parameters for training feedforward neural networks. Neurocomputing 2009, 72:3915-3921.
13. Kimura, H., & Kobayashi, S. (1998). An analysis of actor/critic algorithm using eligibility traces: reinforcement learning with imperfect value functions. In Proc. of the 15th ICML (pp. 278-286).
14. Kober J., Peters J. Policy search for motor primitives in robotics. Machine Learning 2011, 84(1-2):171-203.
15. Konda V., Tsitsiklis J. Actor-critic algorithms. SIAM Journal on Control and Optimization 2003, 42(4):1143-1166.
16. Kormushev, P., Ugurlu, B., Calinon, S., Tsagarakis, N., & Caldwell, D.G. (2011). Bipedal walking energy minimization by reinforcement learning with evolving policy parameterization. In Proc. IEEE/RSJ intl conf. on intelligent robots and systems, IROS (pp. 318-324). San Francisco, USA.
17. Koutnik, J., Gomez, F., & Schmidhuber, J. (2010). Evolving neural networks in compressed weight space. In Proceedings of the conference on genetic and evolutionary computation, GECCO-10 (pp. 619-626).
18. Kushner H.J., Yin G. Stochastic approximation algorithms and applications 1997, Springer-Verlag.
19. McMahan, W., Chitrakaran, V.K., Csencsits, M.A., Dawson, D.M., Walker, I.D., & Jones, B.A. et al. (2006). Field trials and testing of the octarm continuum manipulator. In ICRA (pp. 2336-2341).
20. Noda I. Recursive adaptation of stepsize parameter for non-stationary environments. Principles of practice in multi-agent systems 2009, 525-533.
21. Octopus-sources (2006). http://www.cs.mcgill.ca/~dprecup/workshops/ICML06/Octopus/octopus-code-distribution.zip.
22. Peshkin, L., & Mukherjee, S. (2001). Bounds on sample size for policy evaluation in Markov environments. In Proc. of the 14th annual conf. on computational learning theory, vol. 2111 (pp. 616-629). Berlin: Springer.
23. Peters J., Vijayakumar S., Schaal S. Natural actor-critic. Proc. of ECML 2005, 280-291. Springer-Verlag, Berlin Heidelberg.
24. Rubinstein R.Y. Simulation and the Monte Carlo method 1981, John Wiley & Sons, Inc.
25. Rumelhart D.E., Hinton G.E., Williams R.J. Neurocomputing: foundations of research 1988, MIT Press, (pp. 696-699) (Chap. Learning representations by back-propagating errors).
26. Schraudolph, N.N., & Giannakopoulos, X. (2000). Online independent component analysis with local learning rate adaptation. In Advances in NIPS, vol. 12 (pp. 789-795).
27. Schraudolph, N.N., Yu, J., & Aberdeen, D. (2006). Fast online policy gradient learning with SMD gain vector adaptation. In Advances in NIPS, vol. 18 (pp. 1185-1192).
28. Silva, F.M., & Almeida, L.B. (1990). Acceleration techniques for the backpropagation algorithm. In Neural networks EURASIP workshop. Sesim.
29. Sutton R.S. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. Proc. of the 7th ICML 1990, 216-224. Morgan Kaufmann.
30. Sutton, R.R. (1992a). Gain adaptation beats least squares? In YWALS (pp. 161-166).
31. Sutton, R.S. (1992b). Adapting bias by gradient descent: an incremental version of delta-bar-delta. In Proc. of the 10th NCAI (pp. 171-176).
32. Sutton R.S., Barto A.G. Reinforcement learning: an introduction 1998, MIT Press.
33. Sutton R.S., McAllester D., Singh S., Mansour Y. Policy gradient methods for reinforcement learning with function approximation. Advances in NIPS, vol. 12 2000, 1057-1063. MIT Press.
34. Wawrzyński P. Real-time reinforcement learning by sequential actor-critics and experience replay. Neural Networks 2009, 22:1484-1497.
35. Wawrzyński, P. (2010). Fixed point method of step-size estimation for on-line neural network training. In IJCNN (pp. 2012-2017).
36. Wawrzyński P., Papis B. Fixed point method for autonomous on-line neural network training. Neurocomputing 2011, 74:2893-2905.
37. Williams R.J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 1992, 229-256.
38. Woolley B.G., Stanley K.O. Evolving a single scalable controller for an octopus arm with a variable number of segments. Proceedings of the 11th international conference on parallel problem solving from nature 2010, Springer.
39. Yekutieli Y., Sagiv-Zohar R., Aharonov R., Engel Y., Hochner B., Flash T. Dynamic model of the octopus arm. i. biomechanics of the octopus reaching movement. Journal of Neurophysiology 2005, 94(2):1443-1458.