Recurrent policy gradients

Logic Journal of the IGPL

Volumn 18, Issue 5, 2009, Pages 620-634

  Wierstra, Daan ^a   Förster, Alexander ^a   Peters, Jan ^b   Schmidhuber, Jürgen ^a,c,d  

^a DALLE MOLLE INSTITUTE FOR ARTIFICIAL INTELLIGENCE IDSIA   (Switzerland)

^b MAX PLANCK INSTITUTE FOR BIOLOGICAL CYBERNETICS   (Germany)

^c TECHNICAL UNIVERSITY OF MUNICH   (Germany)

^d UNIVERSITY OF LUGANO   (Switzerland)

Author keywords

Partially Observable Markov Decision Problems (POMDPs); Policy gradient methods; Recurrent neural networks; Reinforcement learning

Indexed keywords

GRADIENT METHODS; RECURRENT NEURAL NETWORKS; STOCHASTIC SYSTEMS;

INTERNAL STATE; MODEL STATE; PARTIALLY OBSERVABLE MARKOV DECISION PROBLEM; PARTIALLY OBSERVABLE MARKOV DECISION PROBLEMS; POLICY GRADIENT; POLICY GRADIENT METHODS; PROBLEM DOMAIN; STATE ESTIMATORS; SYSTEM MODELS; TRADITIONAL APPROACHES;

REINFORCEMENT LEARNING;

EID: 77957283019     PISSN: 13670751     EISSN: 13689894     Source Type: Journal    
DOI: 10.1093/jigpal/jzp049     Document Type: Article

References (31)

1. R. Sutton, A. Barto, Reinforcement learning: An introduction, Cambridge, MA, MIT Press, 1998.
2. S. P. Singh, T. Jaakkola, M. I. Jordan, Learning without state-estimation in partially observable Markovian decision processes. In: International Conference on Machine Learning (ICML 1994), Morgan Kaufmann Publishers, San Francisco, CA, 1994, pp. 284-292.
3. J. Baxter, P. Bartlett, L. Weaver, Experiments with infinite-horizon, policy-gradient estimation, Journal of Artificial Intelligence Research 15 (2001) 351-381.
4. R. Sutton, D. McAllester, S. Singh, Y. Mansour, Policy gradient methods for reinforcement learning with function approximation. In: Advances in Neural Information Processing Systems (NIPS), MIT Press, 2001, pp. 1057-1063.
5. S. Bhatnagar, R. Sutton, M. Ghavamzadeh, M. Lee, Incremental natural actor-critic algorithms. In: Advances in Neural Information Processing Systems (NIPS), MIT Press, 2007, pp. 105-112.
6. D. Aberdeen, Policy-gradient algorithms for partially observable Markov decision processes, Ph.D. thesis, Australian National University, Australia (2003).
7. J. Peters, S. Schaal, Policy gradient methods for robotics. In: Proceedings of the IEEE International Conference on Intelligent Robots and Systems (IROS), Beijing, China, 2006, pp. 2219 - 2225.
8. N. Kohl, P. Stone, Policy gradient reinforcement learning for fast quadrupedal locomotion. In: Proceedings of the IEEE International Conference on Robotics and Automation, Vol. 3, 2004, pp. 2619-2624.
9. J. Peters, S. Schaal, Reinforcement learning of motor skills with policy gradients, Neural Networks 21 (4) (2008) 682-97.
10. H. Benbrahim, J. Franklin, Biped dynamic walking using reinforcement learning, Robotics and Autonomous Systems 22 (3-4) (1997) 283-302.
11. J. Moody, M. Saffell, Learning to Trade via Direct Reinforcement, IEEE Transactions on Neural Networks 12 (4) (2001) 875-889.
12. D. Prokhorov, Toward effective combination of off-line and on-line training in ADP framework. In: Proceedings of the IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning (ADPRL), 2007, pp. 268-271.
13. J. Peters, S. Vijayakumar, S. Schaal, Natural Actor Critic. In: Proceedings of the 16th European Conference on Machine Learning (ECML 2005), 2005, pp. 280-291.
14. J. Peters, S. Schaal, Natural Actor Critic, Neurocomputing 71 (7-9) (2008) 1180-1190.
15. V. Gullapalli, A stochastic reinforcement learning algorithm for learning real-valued functions, Neural Networks 3 (6) (1990) 671-692.
16. V. Gullapalli, Reinforcement learning and its application to control, Ph.D. thesis, University of Massachusetts, Amherst, MA., USA (1992).
17. P. Werbos, Back propagation through time: What it does and how to do it. In: Proceedings of the IEEE, Vol. 78, 1990, pp. 1550-1560.
18. N. Meuleau, L. Peshkin, K.-E. Kim, L. P. Kaelbling, Learning finite-state controllers for partially observable environments. In: Proc. Fifteenth Conference on Uncertainty in Artificial Intelligence (UAI '99). Morgan Kaufmann Publishers, 1999, pp. 427-436.
19. S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural Computation 9 (8) (1997) 1735-1780.
20. B. Bakker, Reinforcement learning with Long Short-Term Memory. In: Advances in Neural Information Processing Systems 14, MIT Press, 2002, pp. 1475-1482.
21. H. T. Siegelmann, E. D. Sontag, Turing computability with neural nets, Applied Mathematics Letters 4 (6) (1991) 77-80.
22. R. J. Williams, D. Zipser, A learning algorithm for continually running fully recurrent networks, Neural Computation 1 (2) (1989) 270-280.
23. Y. Bengio, P. Simard, P. Frasconi, Learning long-term dependencies with gradient descent is difficult, IEEE Transactions on Neural Networks 5 (2) (1994) 157-166.
24. S. Hochreiter, Y. Bengio, P. Frasconi, J. Schmidhuber, Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. In: S. C. Kremer, J. F. Kolen (Eds.), A Field Guide to Dynamical Recurrent Neural Networks, IEEE Press, 2001, pp. 237-244.
25. F. A. Gers, N. Schraudolph, J. Schmidhuber, Learning precise timing with LSTM recurrent networks, Journal of Machine Learning Research 3 (2002) 115-143.
26. J. Schmidhuber, RNN overview, http://www.idsia.ch/~juergen/rnn.html (2004).
27. R. J. Williams, Simple statistical gradient-following algorithms for connectionist reinforcement learning, Machine Learning 8 (1992) 229-256.
28. A. Wieland, Evolving neural network controllers for unstable systems. In: Proceedings of the International Joint Conference on Neural Networks (Seattle, WA), Piscataway, NJ: IEEE, 1991, pp. 667-673.
29. M. Littman, A. Cassandra, L. Kaelbling, Learning policies for partially observable environments: Scaling up. In: A. Prieditis, S. Russell (Eds.), Machine Learning. Proceedings of the Twelfth International Conference. Morgan Kaufmann Publishers, San Francisco, CA, 1995, pp. 362-370.
30. B. Bakker, The state of mind:: Reinforcement learning with recurrent neural networks, Ph.D. thesis, Leiden University, the Netherlands (2004).
31. Torcs, The open racing car simulator, http://torcs.sourceforge.net/ (2007).