Benchmarking deep reinforcement learning for continuous control

33rd International Conference on Machine Learning, ICML 2016

Volumn 3, Issue , 2016, Pages 2001-2014

  Duan, Yan ^a   Chen, Xi ^a   Houthooft, Rein ^a,b   Schulman, John ^a,c   Abbeel, Pieter ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b GHENT UNIVERSITY   (Belgium)

^c Open AI   (Belgium)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; LEARNING ALGORITHMS; LEARNING SYSTEMS; PERSONNEL TRAINING;

BENCHMARK SUITES; CONTINUOUS CONTROL; FEATURE REPRESENTATION; HIERARCHICAL STRUCTURES; PARTIAL OBSERVATION; REFERENCE IMPLEMENTATION; REPRODUCIBILITIES; SYSTEMATIC EVALUATION;

REINFORCEMENT LEARNING;

EID: 84999018287     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (84)

1. Abeyruwan, S. RLLib: Lightweight standard and on/off policy reinforcement learning library (C++), http://web.cs.miami. edu/home/saminda/rilib.html, 2013.
2. Bagnell, J. A. and Schneider, J. Covariant policy search, pp. 1019-1024. IJCAI, 2003.
3. Bakker, B. Reinforcement learning with long short-term memory. In NIPS, pp. 1475-1482, 2001.
4. Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The Arcade Learning Environment: An evaluation platform for general agents. J. Artif. Intell. Res., 47:253-279, 2013.
5. Bellman, R. Dynamic Programming. Princeton University Press, 1957.
6. Bertsekas, Dimitri P and Tsitsiklis, John N. Neuro-dynamic programming: an overview. In CDC, pp. 560-564, 1995.
7. Busoniu, L. ApproxRL: A Matlab toolbox for approximate RL and DP. http://busoniu.net/files/repository/readme-approxrl.html, 2010.
8. Catto, E. Box2D: A 2D physics engine for games, 2011.
9. Coulom, Remi. Reinforcement learning using neural networks, with applications to motor control. PhD thesis, Institut National Polytechnique de Grenoble-INPG, 2002.
10. Dann, C, Neumann, G., and Peters, J. Policy evaluation with temporal differences: A survey and comparison. J. Mach. Learn. Res., 15(1):809-883, 2014.
11. Degris, T., Bechu, J., White, A., Modayil, J., Pilarski, P. M., and Denk, C. RLPark. http://rlpark.github.io, 2013.
12. Deisenroth, M. P., Neumann, G., and Peters, J. A survey on policy search for robotics, foundations and trends in robotics. Found. Trends Robotics, 2(1-2):1-142, 2013.
13. DeJong, G. and Spong, M. W. Swinging up the Acrobot: An example of intelligent control. In ACC, pp. 2158-2162, 1994.
14. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. In CVPR, pp. 248-255, 2009.
15. Dietterich, T. G. Hierarchical reinforcement learning with the MAXQ value function decomposition. J. Artif. Intell. Res, 13:227-303, 2000.
16. Dimitrakakis, C, Tziortziotis, N., and Tossou, A. Beliefbox: A framework for statistical methods in sequential decision making. http://code.google.eom/p/beliefbox/, 2007.
17. Dimitrakakis, Christos, Li, Guangliang, and Tziortziotis, Nikoalos. The reinforcement learning competition 2014. AI Magazine, 35(3):61-65, 2014.
18. Donaldson, P. E. K. Error decorrelation: a technique for matching a class of functions. In Proc. 3th Intl. Conf. Medical Electronics, pp. 173-178, 1960.
19. Doya, K. Reinforcement learning in continuous time and space. Neural Comput., 12(1):219-245, 2000.
20. Dutech, Alain, Edmunds, Timothy, Kok, Jelle, Lagoudakis, Michail, Littman, Michael, Riedmiller, Martin, Russell, Bryan, Scherrer, Bruno, Sutton, Richard, Timmer, Stephan, et al. Reinforcement learning benchmarks and bake-offs ii. Advances in Neural Information Processing Systems (NIPS), 17, 2005.
21. Erez, Tom, Tassa, Yuval, and Todorov. Emanuel. Infinite horizon model predictive control for nonlinear periodic tasks. Manuscript under review, 4, 2011.
22. Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., and Zisserman, A. The pascal visual object classes (VOC) challenge. Int. J. Comput. Vision, 88(2):303-338, 2010.
23. Fei-Fei, L., Fergus, R., and Perona, P. One-shot learning of object categories. IEEE Trans. Pattern Anal. Mach. Intell, 28(4):594-611, 2006.
24. Furuta, K., Okutani, T., and Sone, H. Computer control of a double inverted pendulum. Comput. Electr. Eng., 5(1):67-84, 1978.
25. Garofolo, J. S., Lamel, L. R, Fisher, W. M., Fiscus, J. G., and Pallett, D. S. DARPA TIMIT acoustic-phonetic continuous speech corpus CD-ROM. NIST speech disc 1-1.1. NASA STI/Recon Technical Report N, 93, 1993.
26. Godfrey, J. J., Holliman, E. C, and McDaniel, J. SWITCH-BOARD: Telephone speech corpus for research and development. In ICASSP, pp. 517-520, 1992.
27. Gomez, F. and Miikkulainen, R. 2-d pole balancing with recurrent evolutionary networks. In ICANN, pp. 425-430. 1998.
28. Guo, X., Singh, S., Lee, H., Lewis, R. L., and Wang, X. Deep learning for real-time Atari game play using offline montecarlo tree search planning. In NIPS, pp. 3338-3346. 2014.
29. Hansen, N. and Ostermeier, A. Completely derandomized self-adaptation in evolution strategies. Evol. Comput., 9(2):159-195, 2001.
30. Heess, N., Hunt, J., Lillicrap, T., and Silver, D. Memory-based control with recurrent neural networks. arXiv: 1512.04455, 2015a.
31. Heess, N., Wayne, G., Silver, D., Lillicrap, T., Erez, T, and Tassa, T. Learning continuous control policies by stochastic value gradients. In NIPS, pp. 2926-2934. 2015b.
32. Hester, T. and Stone, P. The open-source TEXPLORE code release for reinforcement learning on robots. In RoboCup 2013: Robot World Cup XVII, pp. 536-543. 2013.
33. Hinton, G., Deng, L., Yu, D., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Dahl, T. S. G., and Kingsbury, B. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Process. Mag, 29(6):82-97, 2012.
34. Hirsch, H.-G. and Pearce, D. The Aurora experimental framework for the performance evaluation of speech recognition systems under noisy conditions. In ASR2000-Automatic Speech Recognition: Challenges for the new Millenium ISCA Tutorial and Research Workshop (ITRW), 2000.
35. Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural Comput., 9(8):1735-1780, 1997.
36. Kakade, S. M. A natural policy gradient. In NIPS, pp. 1531-1538. 2002.
37. Kimura, H. and Kobayashi, S. Stochastic real-valued reinforcement learning to solve a nonlinear control problem. In IEEE SMC, pp. 510-515, 1999.
38. Kober, J. and Peters, J. Policy search for motor primitives in robotics. In NIPS, pp. 849-856, 2009.
39. Kochenderfer, M. JRLF: Java reinforcement learning framework. http://mykel.kochenderfer.com/jrlf, 2006.
40. Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. Technical report, 2009.
41. Krizhevsky, A., Sutskever, I., and Hinton, G. ImageNet classification with deep convolutional neural networks. In NIPS, pp. 1097-1105. 2012.
42. LeCun, Y, Cortes, C, and Burges, C. The MNIST database of handwritten digits, 1998.
43. Levine, S. and Koltun, V. Guided policy search. In ICML, pp. 1-9, 2013.
44. Levine, S., Finn, C, Darrell, T, and Abbeel, P. End-to-end training of deep visuomotor policies. arXiv: 1504.00702, 2015.
45. Lillicrap, T, Hunt, J., Pritzel, A., Heess, N., Erez, T, Tassa, Y, Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. arXiv: 1509.02971, 2015.
46. Martin, D., C. Fowlkes, D. Tal, and Malik, J. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV, pp. 416-423, 2001.
47. Metzen, J. M. and Edgington, M. Maja machine learning framework. http://mloss.org/software/view/220/, 2011.
48. Michie, D. and Chambers, R. A. BOXES: An experiment in adaptive control. Machine Intelligence, 2:137-152, 1968.
49. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C, Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.
50. Moore, A. Efficient memory-based learning for robot control. Technical report, University of Cambridge, Computer Laboratory, 1990.
51. Murray, R. M. and Hauser, J. A case study in approximate linearization: The Acrobot example. Technical report, UC Berkeley, EECS Department, 1991.
52. Murthy, S. S. and Raibert, M. H. 3D balance in legged locomotion: modeling and simulation for the one-legged case. ACM SIGGRAPH Computer Graphics, 18(1):27-27, 1984.
53. Neumann, G. A reinforcement learning toolbox and RL benchmarks for the control of dynamical systems. Dynamical principles for neuroscience and intelligent biomimetic devices, pp. 113, 2006.
54. Papis, B. and Wawrzynski, P. dotrl: A platform for rapid reinforcement learning methods development and validation. In FedCSIS, pp. pages 129-136., 2013.
55. Parr, Ronald and Russell, Stuart. Reinforcement learning with hierarchies of machines. Advances in neural information processing systems, pp. 1043-1049, 1998.
56. Peters, J. Policy Gradient Toolbox. http://www.ausy. tu-darmstadt.de/Research/PolicyGradientToolbox, 2002.
57. Peters, J. and Schaal, S. Reinforcement learning by rewardweighted regression for operational space control. In ICML, pp. 745-750, 2007.
58. Peters, J. and Schaal, S. Reinforcement learning of motor skills with policy gradients. Neural networks, 21(4):682-697, 2008.
59. Peters, J., Vijaykumar, S., and Schaal, S. Policy gradient methods for robot control. Technical report, 2003.
60. Peters, J., Mulling, K., and Altiin, Y. Relative entropy policy search. In AAAI, pp. 1607-1612, 2010.
61. Purcell, E. M. Life at low Reynolds number. Am. J. Phys, 45(1):3-11, 1977.
62. Raibert, M. H. and Hodgins, J. K. Animation of dynamic legged locomotion. In ACM SIGGRAPH Computer Graphics, volume 25, pp. 349-358, 1991.
63. Riedmiller, M., Blum, M., and Lampe, T. CLS2: Closed loop simulation system, http://ml.inforniatik.uni-freiburg.de/research/clsquare, 2012.
64. Rubinstein, R. The cross-entropy method for combinatorial and continuous optimization. Methodol. Comput. Appl. Probab., 1(2):127-190, 1999.
65. Schäfer, A. M. and Udluft, S. Solving partially observable reinforcement learning problems with recurrent neural networks. In ECML Workshops, pp. 71-81, 2005.
66. Schaul, T, Bayer, J., Wierstra, D., Sun, Y., Felder, M., Sehnke, F., Rückstieß, T, and Schmidhuber, J. PyBrain. J Mach. Learn. Res., 11:743-746, 2010.
67. Schulman, J., Levine, S., Abbeel, P., Jordan, M. I., and Moritz, P. Trust region policy optimization. In ICML, pp. 1889-1897, 2015a.
68. Schulman, J., Moritz, P., Levine, S., Jordan, M. I., and Abbeel, P. High-dimensional continuous control using generalized advantage estimation. arXiv: 1506.02438, 2015b.
69. Stephenson, A. On induced stability. Philos. Mag., 15(86):233-236, 1908.
70. Stone, Peter, Kuhlmann, Gregory, Taylor, Matthew E, and Liu, Yaxin. Keepaway soccer: From machine learning testbed to benchmark. In RoboCup 2005: Robot Soccer World Cup IX, pp. 93-105. Springer, 2005.
71. Sutton, Richard S, Precup, Doina, and Singh, Satinder. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1):181-211, 1999.
72. Szita, I. and Lorincz, A. Learning Tetris using the noisy crossentropy method. Neural Comput., 18(12):2936-2941, 2006.
73. Szita, I., Takacs, B., and Lorincz, A. ϵ-MDPs: Learning in varying environments. J. Mach. Learn. Res., 3:145-174, 2003.
74. Tassa, Yuval, Erez, Tom, and Todorov, Emanuel. Synthesis and stabilization of complex behaviors through online trajectory optimization. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pp. 4906-4913. IEEE, 2012.
75. Tesauro, G. Temporal difference learning and TD-Gammon. Commun. ACM, 38(3):58-68, 1995.
76. Todorov, E., Erez, T, and Tassa, Y. MuJoCo: A physics engine for model-based control. In IROS, pp. 5026-5033, 2012.
77. van Hoof, H., Peters, J., and Neumann, G. Learning of nonparametric control policies with high-dimensional state features. In AISTATS, pp. 995-1003, 2015.
78. Watter, M., Springenberg, J., Boedecker, J., and Riedmiller, M. Embed to control: A locally linear latent dynamics model for control from raw images. In NIPS, pp. 2728-2736, 2015.
79. Wawrzynski, P. Learning to control a 6-degree-of-freedom walking robot. In IEEE EUROCON, pp. 698-705, 2007.
80. Widrow, B. Pattern recognition and adaptive control. IEEE Trans. Ind. Appl, 83(74):269-277, 1964.
81. Wierstra, D., Foerster, A., Peters, J., and Schmidhuber, J. Solving deep memory POMDPs with recurrent policy gradients. In ICANN, pp. 697-706. 2007.
82. Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn., 8:229-256, 1992.
83. Yamaguchi, A. and Ogasawara, T. SkyAI: Highly modularized reinforcement learning library. In IEEE-RAS Humanoids, pp. 118-123, 2010.
84. Yu, D., Ju, Y.-C, Wang, Y.-Y, Zweig, G., and Acero, A. Automated directory assistance system - from theory to practice. In Interspeech, pp. 2709-2712, 2007.