System Design Perspective for Human-Level Agents Using Deep Reinforcement Learning: A Survey

IEEE Access

Volumn 5, Issue , 2017, Pages 27091-27102

  Nguyen, Ngoc Duy ^a   Nguyen, Thanh ^a   Nahavandi, Saeid ^a  

^a DEAKIN UNIVERSITY   (Australia)

Author keywords

Deep learning; human level agents; reinforcement learning; robotics; survey; system design

Indexed keywords

ARTIFICIAL INTELLIGENCE; DEEP LEARNING; INTELLIGENT ROBOTS; LEARNING SYSTEMS; MACHINE DESIGN; MATHEMATICAL MODELS; MONTE CARLO METHODS; ROBOTICS; ROBOTS; SURVEYING; SURVEYS; SYSTEMS ANALYSIS;

ARTIFICIAL AGENTS; AUTONOMOUS SYSTEMS; FUTURE RESEARCH DIRECTIONS; HUMAN LEVELS; INTELLIGENT ACTION; INTERACTIVE ENVIRONMENTS; LEARNING (ARTIFICIAL INTELLIGENCE); LEARNING METHODS;

REINFORCEMENT LEARNING;

EID: 85035762670     PISSN: None     EISSN: 21693536     Source Type: Journal    
DOI: 10.1109/ACCESS.2017.2777827     Document Type: Review

References (75)

1. R. S. Sutton and A. G. Barto, Reinforcement LearningfiAn Introduction. Cambridge, MA, USA: MIT Press, 2012.
2. J. Kober, J. A. Bagnell, J. Peters, "Reinforcement learning in robotics: A survey, " Int. J. Robot. Res., vol. 32, no. 11, pp. 1238-1274, 2013.
3. S. Mahadevan and J. Connell, "Automatic programming of behaviorbased robots using reinforcement learning, " Artif. Intell., vol. 55, nos. 2-3, pp. 311-365, Jun. 1992.
4. S. Schaal, "Learning from demonstration, " in Proc. Adv. Neural Inf. Process. Syst., 1997, pp. 1040-1046.
5. J.-L. Lin, K.-S. Hwang, W.-C. Jiang, Y.-J. Chen, "Gait balance and acceleration of a biped robot based on Q-learning, " IEEE Access, vol. 4, pp. 2439-2449, 2016.
6. K. Mülling, J. Kober, O. Kroemer, J. Peters, "Learning to select and generalize striking movements in robot table tennis, " Int. J. Robot. Res., vol. 32, no. 3, pp. 263-279, 2013.
7. M. Riedmiller, T. Gabel, R. Hafner, S. Lange, "Reinforcement learning for robot soccer, " Auton. Robots, vol. 27, no. 1, pp. 55-73, 2009.
8. E. L. Thorndike, "Animal intelligence: An experimental study of the associate processes in animals, " Amer. Psychol., vol. 53, no. 10, pp. 1125-1127, 1998.
9. W. Schultz, P. Dayan, P. R. Montague, "Aneural substrate of prediction and reward, " Science, vol. 275, no. 5306, pp. 1593-1599, 1997.
10. R. Bellman, Dynamic Programming. Princeton, NJ, USA: Princeton Univ. Press, 2010.
11. L. Deng and D. Yu, "Deep learning: Methods and applications, " Found. Trends Signal Process., vol. 7, nos. 3-4, pp. 197-387, Jun. 2014.
12. L. Fei-Fei, J. Deng, K. Li, "ImageNet: Constructing a large-scale image database, " J. Vis., vol. 9, no. 8, p. 1037, 2009.
13. A. Krizhevsky, I. Sutskever, G. E. Hinton, "ImageNet classification with deep convolutional neural networks, " in Proc. Adv. Neural Inf. Pro-cess. Syst., Dec. 2012, pp. 1097-1105.
14. K. Simonyan and A. Zisserman. (Sep. 2014). "Very deep convolutional networks for large-scale image recognition." [Online]. Available: Https://arxiv.org/abs/1409.1556
15. A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, L. Fei-Fei, "Large-scale video classification with convolutional neural networks, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2014, pp. 1725-1732.
16. O. Abdel-Hamid, A. Mohamed, H. Jiang, G. Penn, "Applying convolutional neural networks concepts to hybrid NN-HMM model for speech recognition, " in Proc. IEEE Conf. Acoust. Speech Signal Process., Mar. 2012, pp. 4277-4280.
17. M.-T. Luong, M. Kayser, C. D. Manning, "Deep neural language models for machine translation, " in Proc. Conf. Comput. Nat. Lang. Learn., Jul. 2015, pp. 305-309.
18. H. A. Pierson and M. S. Gashler, "Deep learning in robotics: A review of recent research, " Adv. Robot., vol. 31, no. 16, pp. 821-835, 2017.
19. C. J. C. H. Watkins and P. Dayan, "Q-learning, " Mach. Learn., vol. 8, nos. 3-4, pp. 279-292, 1992.
20. V. Mnih, et al., "Human-level control through deep reinforcement learning, " Nature, vol. 518, no. 7540, pp. 529-533, 2015.
21. M. G. Bellemare, Y. Naddaf, J. Veness, M. Bowling, "The arcade learning environment: An evaluation platform for general agents, " J. Artif. Intell. Res., vol. 47, pp. 253-279, May 2013.
22. D. Silver, et al., "Mastering the game of Go with deep neural networks and tree search, " Nature, vol. 529, no. 7578, pp. 484-489, Jan. 2016.
23. V. R. Konda and J. N. Tsitsiklis, "Actor-critic algorithms, " in Proc. Adv. Neural Inf. Process. Syst., 2000, pp. 1008-1014.
24. J. N. Tsitsiklis and B. Van Roy, "An analysis of temporal-difference learning with function approximation, " in Proc. Adv. Neural Inf. Process. Syst., 1997, pp. 1075-1081.
25. M. G. Bellemare, J. Veness, M. Bowling, "Investigating contingency awareness using Atari 2600 games, " in Proc. AAAI Conf. Artif. Intell., Jul. 2012, pp. 864-871.
26. M. Campbell, A. J. Hoane, Jr., F.-H. Hsu, "Deep blue, " Artif. Intell., vol. 134, nos. 1-2, pp. 57-83, Jan. 2002.
27. R. Coulom, "Efficient selectivity and backup operators in Monte-Carlo tree search, " in Proc. Int. Conf. Comput. Games, 2006, pp. 72-83.
28. C. B. Browne, et al., "A survey of Monte Carlo tree search methods, " IEEE Trans. Comput. Intell. AI Games, vol. 4, no. 1, pp. 1-43, Mar. 2012.
29. L. Kocsis and C. Szepesvári, "Bandit based Monte-Carlo planning, " in Proc. Eur. Conf. Mach. Learn., vol. 6. 2006, pp. 282-293.
30. G. Tesauro and G. R. Galperin, "On-line policy improvement using Monte-Carlo search, " in Proc. Adv. Neural Inf. Process. Syst., 1997, pp. 1068-1074.
31. G. Tesauro, "Temporal difference learning and TD-Gammon, " Commun. ACM, vol. 38, no. 3, pp. 58-68, Mar. 1995.
32. R. M. French, "Catastrophic forgetting in connectionist networks, " Trends Cognit. Sci., vol. 3, no. 4, pp. 128-135, 1999.
33. D. Kumaran, D. Hassabis, J. L. McClelland, "What learning systems do intelligent agents need Complementary learning systems theory updated, " Trends Cognit. Sci., vol. 20, no. 7, pp. 512-534, 2016.
34. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, "Dropout: A simple way to prevent neural networks from overfitting, " J. Mach. Learn. Res., vol. 15, no. 1, pp. 1929-1958, 2014.
35. F. Girosi, M. Jones, T. Poggio, "Regularization theory and neural networks architectures, " Neural Comput., vol. 7, no. 2, pp. 219-269, Mar. 1995.
36. I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, Y. Bengio. (Dec. 2013). "An empirical investigation of catastrophic forgetting in gradient-based neural networks." [Online]. Available: Https://arxiv.org/abs/1312.6211
37. S. Thrun and L. Pratt, Learning to Learn. Boston, MA, USA: Kluwer, 1998.
38. A. A. Rusu, et al. (2016.). "Progressive neural networks." [Online]. Available: Https://arxiv.org/abs/1606.04671
39. A. A. Rusu, et al. (Nov. 2015). "Policy distillation." [Online]. Available: Https://arxiv.org/abs/1511.06295
40. E. Parisotto, J. L. Ba, R. Salakhutdinov. (2015). "Actor-mimic: Deep multitask and transfer reinforcement learning." [Online]. Available: Https://arxiv.org/abs/1511.06342
41. H. Yin and S. J. Pan, "Knowledge transfer for deep reinforcement learning with hierarchical experience replay, " in Proc. AAAI Conf. Artif. Intell., Jan. 2017, pp. 1640-1646.
42. J. Kirkpatrick, et al., "Overcoming catastrophic forgetting in neural networks, " in Proc. Nat. Acad. Sci. USA, vol. 114, no. 3, pp. 3521-3526, 2017.
43. C. Clopath, "Synaptic consolidation: An approach to long-term learning, " Cognit. Neurodyn., vol. 6, no. 3, pp. 251-257, Jun. 2012.
44. H. J. Sussmann, "Uniqueness of the weights for minimal feedforward nets with a given input-output map, " J. Neural Netw., vol. 5, no. 4, pp. 589-593, Jul./Aug. 1992.
45. R. Olfati-Saber, "Flocking for multi-agent dynamic systems: Algorithms and theory, " IEEE Trans. Autom. Control, vol. 51, no. 3, pp. 401-420, Mar. 2006.
46. M. L. Littman, "Markov games as a framework for multi-agent reinforcement learning, " in Proc. Int. Conf. Mach. Learn., 1994, pp. 157-163.
47. A. Tampuu, et al., "Multiagent cooperation and competition with deep reinforcement learning, " PLoS ONE, vol. 12, no. 4, p. e0172395, Apr. 2017.
48. L. Kraemer and B. Banerjee, "Multi-agent reinforcement learning as a rehearsal for decentralized planning, " Neurocomputing, vol. 190, pp. 82-94, May 2016.
49. J. Foerster, Y. A. M. Assael, N. de Freitas, S. Whiteson, "Learning to communicate with deep multi-agent reinforcement learning, " in Proc. Adv. Neural Inf. Process. Syst., 2016, pp. 2137-2145.
50. S. Sukhbaatar, A. Szlam, R. Fergus, "Learning multiagent communication with backpropagation, " in Proc. Adv. Neural Inf. Process. Syst., 2016, pp. 2244-2252.
51. H. He, J. Boyd-Graber, K. Kwok, H. Daumé, III, "Opponent modeling in deep reinforcement learning, " in Proc. Int. Conf. Mach. Learn., 2016, pp. 1804-1813.
52. A. Nair, et al. (2015). "Massively parallel methods for deep reinforcement learning." [Online]. Available: Https://arxiv.org/abs/1507.04296
53. T. Schaul, J. Quan, I. Antonoglou, D. Silver. (2015). "Prioritized experience replay." [Online]. Available: Https://arxiv.org/abs/1511.05952
54. Z. Wang, et al. (2015). "Dueling network architectures for deep reinforcement learning." [Online]. Available: Https://arxiv.org/abs/1511.06581
55. V. Mnih, et al., "Asynchronous methods for deep reinforcement learning, " in Proc. Int. Conf. Mach. Learn., 2016, pp. 1928-1937.
56. M. Hausknecht and P. Stone, "Deep recurrent Q-learning for partially observable MDPs, " in Proc. AAAI Symp. Seq. Decis. Mak. Intell. Agents, Nov. 2015. [Online]. Available: Https://www.cs.utexas.edu/~pstone/Papers/bib2html/b2hd-SDMIA15-Hausknecht.html
57. I. Sorokin, A. Seleznev, M. Pavlov, A. Fedorov, A. Ignateva. (2015). "Deep attention recurrent Q-network." [Online]. Available: Https://arxiv.org/abs/1512.01693
58. T. P. Lillicrap, et al. (2015). "Continuous control with deep reinforcement learning." [Online]. Available: Https://arxiv.org/abs/1509.02971
59. H. van Hasselt, A. Guez, D. Silver, "Deep reinforcement learning with double Q-learning, " in Proc. AAAI Conf. Artif. Intell., 2016, pp. 2094-2100.
60. A. G. Barto and S. Mahadevan, "Recent advances in hierarchical reinforcement learning, " Discrete Event Dyn. Syst., vol. 13, no. 4, pp. 341-379, Oct. 2003.
61. R. S. Sutton, D. Precup, S. Singh, "Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning, " Artif. Intell., vol. 112, nos. 1-2, pp. 181-211, Aug. 1999.
62. C. Guestrin, D. Koller, R. Parr, S. Venkataraman, "Efficient solution algorithms for factored MDPs, " J. Artif. Intell. Res., vol. 19, pp. 399-468, Jul./Dec. 2003. [Online]. Available: Http://www.jair.org/contents.html
63. T. G. Dietterich, "Hierarchical reinforcement learning with the MAXQ value function decomposition, " J. Artif. Intell. Res., vol. 13, pp. 227-303, Nov. 2000.
64. N. Chentanez, A. G. Barto, S. P. Singh, "Intrinsically motivated reinforcement learning, " in Proc. Adv. Neural Inf. Process. Syst., 2005, pp. 1281-1288.
65. T. D. Kulkarni, K. R. Narasimhan, A. Saeedi, J. B. Tenenbaum, "Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation, " in Proc. Adv. Neural Inf. Process. Syst., 2016, pp. 3675-3683.
66. G. Brockman, et al. (2016). "OpenAI gym." [Online]. Available: Https://arxiv.org/abs/1606.01540
67. M. Abadi, et al. (2016). "TensorFlow: Large-scale machine learning on heterogeneous distributed systems." [Online]. Available: Https://arxiv.org/abs/1603.04467
68. J. Bergstra, et al., "Theano: Deep learning on GPUs with Python, " in Proc. BigLearn Workshop NIPS, 2011.
69. F. Chollet. (2015). Keras. [Online]. Available: Http://keras.io
70. F. Pedregosa, et al., "Scikit-learn: Machine learning in Python, " J. Mach. Learn. Res., vol. 12, pp. 2825-2830, Oct. 2011.
71. O. Klimov and J. Schulman. (2017). Roboschool. [Online]. Available: Https://blog.openai.com/roboschool/
72. J. MacGlashan. (2016). Brown-UMBC Reinforcement Learning and Plan-ning (BURLAP). [Online]. Available: Http://burlap.cs.brown.edu/
73. B. Tanner and A. White, "RL-glue: Language-independent software for reinforcement-learning experiments, " J. Mach. Learn. Res., vol. 10, pp. 2133-2136, Sep. 2009.
74. A. Makmal, A. A. Melnikov, V. Dunjko, H. J. Briegel, "Meta-learning within projective simulation, " IEEE Access, vol. 4, pp. 2110-2122, 2016.
75. P. Abbeel and A. Y. Ng, "Inverse reinforcement learning, " in Encyclopedia of Machine Learning. New York, NY, USA: Springer, 2011, pp. 554-558.