Experience replay for real-time reinforcement learning control

IEEE Transactions on Systems, Man and Cybernetics Part C: Applications and Reviews

Volumn 42, Issue 2, 2012, Pages 201-212

  Adam, Sander ^a   Buşoniu, Lucian ^b   Babuška, Robert ^b  

^a ABN AMRO Corporate Research   (Netherlands)

^b DELFT UNIVERSITY OF TECHNOLOGY   (Netherlands)

Author keywords

Experience replay (ER); Q learning; real time control; reinforcement learning (RL); robotics; SARSA

Indexed keywords

EXPERIENCE REPLAY (ER); OPTIMAL CONTROL STRATEGY; PENDULUM SWING-UP PROBLEM; Q-LEARNING; REAL-TIME EXPERIMENT; REAL-TIME LEARNING; REINFORCEMENT LEARNING CONTROL; SARSA; SARSA ALGORITHM; SIMULATED SYSTEM; SIMULATION STUDIES; VISION BASED CONTROL;

ALGORITHMS; REAL TIME CONTROL; ROBOTICS;

REINFORCEMENT LEARNING;

EID: 84857501996     PISSN: 10946977     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSMCC.2011.2106494     Document Type: Article

References (37)

1. L. Baird, "Residual algorithms: Reinforcement learning with function approximation," in Proc. 12th Int. Conf. Mach. Learning, Tahoe City, CA, Jul. 9-12, 1995, pp. 30-37.
2. F. Bernardo, R. Agustí, J. Pérez-Romero, and O. Sallent, "An application of reinforcement learning for efficient spectrum usage in next-generation mobile cellular networks," IEEE Trans. Syst., Man, Cybern. C: Appl. Rev., vol. 40, no. 4, pp. 477-484, Jul. 2010.
3. D. P. Bertsekas, Dynamic Programming and Optimal Control, 3rd ed. vol. 2, Belmont, MA: Athena Sci., 2007.
4. D. P. Bertsekas and S. E. Shreve, Stochastic Optimal Control: TheDiscrete Time Case. New York: Academic, 1978.
5. D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming. Belmont, MA: Athena Sci., 1996.
6. L. Buşoniu, R. Babuška, B. De Schutter, and D. Ernst, Reinforcement Learning and Dynamic Programming Using Function Approximators (Automation and Control Engineering). New York/Boca Raton, FL: Taylor & Francis/CRC, 2010.
7. L. Buşoniu, D. Ernst, B. De Schutter, and R. Babuška, "Online leastsquares policy iteration for reinforcement learning control," in Proc. Amer. Control Conf., Baltimore, MD, Jun. 30-Jul. 2, 2010, pp. 486-491.
8. P. Cichosz, "An analysis of experience replay in temporal difference learning," Cybern. Syst., vol. 30, pp. 341-363, 1999.
9. L. T. Dung, T. Komeda, and M. Takagi, "Efficient experience reuse in non-Markovian environments," in Proc. Int. Conf. Instrum., Control Inf. Technol., Tokyo, Japan, Aug. 20-22, 2008, pp. 3327-3332.
10. D. Ernst, "Near optimal closed-loop control. Application to electric power systems," Ph.D. dissertation, University of Liége, Liége, Belgium, Mar. 2003.
11. D. Ernst, P. Geurts, and L. Wehenkel, "Tree-based batch mode reinforcement learning," J. Mach. Learning Res., vol. 6, pp. 503-556, 2005.
12. D. Ernst, G.-B. Stan, J. Gonçalves, and L.Wehenkel, "Clinical data based optimal STI strategies for HIV: A reinforcement learning approach," in Proc. 45th IEEE Conf. Decis. Control, San Diego, CA, Dec. 13-15, 2006, pp. 667-672.
13. D. Gu and H. Hu, "Integration of coordination architecture and behavior fuzzy learning in quadruped walking robots," IEEE Trans. Syst., Man, Cybern. C: Appl. Rev., vol. 37, no. 4, pp. 670-681, Jul. 2007.
14. T. Jaakkola, M. I. Jordan, and S. P. Singh, "On the convergence of stochastic iterative dynamic programming algorithms," Neural Comput., vol. 6, no. 6, pp. 1185-1201, 1994.
15. L. Jouffe, "Fuzzy inference system learning by reinforcement methods," IEEE Trans. Syst.,Man, Cybern. C: Appl. Rev., vol. 28, no. 3, pp. 338-355, Aug. 1998.
16. S. Kalyanakrishnan and P. Stone, "Batch reinforcement learning in a complex domain," in Proc. 6th Int. Conf. Auton. Agents Multi-Agent Syst., Honolulu, HI, May 14-18, 2007, pp. 650-657.
17. M. G. Lagoudakis and R. Parr, "Least-squares policy iteration," J. Mach. Learning Res., vol. 4, pp. 1107-1149, 2003.
18. L. Li, M. L. Littman, and C. R. Mansley, "Online exploration in leastsquares policy iteration," in Proc. 8th Int. Joint Conf. Auton. Agents Multiagent Syst., Budapest, Hungary, May 10-15, 2009, vol. 2, pp. 733-739.
19. L.-J. Lin, "Self-improving reactive agents based on reinforcement learning, planning and teaching," Mach. Learning, vol. 8, no. 3/4, (Special issue on reinforcement learning), pp. 293-321, Aug. 1992.
20. F. S. Melo, S. P. Meyn, and M. I. Ribeiro, "An analysis of reinforcement learning with function approximation," in Proc. 25th Int. Conf. Mach. Learning, Helsinki, Finland, Jul. 5-9, 2008, pp. 664-671.
21. A. W. Moore and C. G. Atkeson, "Prioritized sweeping: Reinforcement learning with less data and less time," Mach. Learning, vol. 13, pp. 103-130, 1993.
22. K. Park, Y. Kim, and J. Kim, "Modular Q-learning based multi-agent cooperation for robot soccer," Robot. Auton. Syst., vol. 35, pp. 109-122, 2001.
23. T. J. Perkins and D. Precup, "A convergent form of approximate policy iteration," in Advances in Neural Information Processing Systems, vol. 15, S. Becker, S. Thrun, and K. Obermayer, Eds. Cambridge,MA: MIT Press, 2003, pp. 1595-1602.
24. J. Peters and S. Schaal, "Reinforcement learning of motor skills with policy gradients," Neural Netw., vol. 21, pp. 682-697, 2008.
25. G. A. Rummery and M. Niranjan. (1994, Sep.). On-line Q-learning using connectionist systems. Tech. Rep. CUED/F-INFENG/TR166, Engineering Department, Cambridge University, U.K. [Online]. Available at http://mi.eng.cam.ac.uk/ reports/svr-ftp/rummery-tr166.ps.Z.
26. S. Singh and R. Sutton, "Reinforcement learning with replacing eligibility traces," Mach. Learning, vol. 22, pp. 123-158, 1996.
27. S. P. Singh, T. Jaakkola, and M. I. Jordan, "Reinforcement learning with soft state aggregation," in Advances in Neural Information Processing Systems, vol. 7, G. Tesauro, D. S. Touretzky, and T. K. Leen, Eds. Cambridge, MA: MIT Press, 1995, pp. 361-368.
28. W. Smart, "Making reinforcement learning work on real robots," Ph.D. dissertation, Brown University, Providence, RI, 2002.
29. W. D. Smart and L. P. Kaelbling, "Practical reinforcement learning in continuous spaces," in Proc. 17th Int. Conf. Mach. Learning, Stanford University, Stanford, CA, Jun. 29-Jul. 2, 2000, pp. 903-910.
30. P. Stone, R. Sutton, and G. Kuhlmann, "Reinforcement learning for RoboCup soccer keepaway," Adaptive Behav., vol. 13, pp. 165-188, 2005.
31. R. S. Sutton, "Integrated architectures for learning, planning, and reacting based on approximating dynamic programming," in Proc. 7th Int. Conf. Mach. Learning, Austin, TX, Jun. 21-23, 1990, pp. 216-224.
32. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press, 1998.
33. R. S. Sutton, Cs. Szepesvári, A. Geramifard, and M. H. Bowling, "Dynastyle planning with linear function approximation and prioritized sweeping," in Proc. 24th Conf. Uncertainty Artif. Intell., Helsinki, Finland, Jul. 9-12, 2008, pp. 528-536.
34. Cs. Szepesvári and W. D. Smart, "Interpolation-based Q-learning," in Proc. 21st Int. Conf. Mach. Learning, Bannf, AB, Canada, Jul. 4-8, 2004, pp. 791-798.
35. J. Van Ast and R. Babuska, "Dynamic exploration in Q(λ)-learning," in Proc. Int. Joint Conf. Neural Netw., Vancouver, BC, Canada, Jul. 16-21, 2006, pp. 41-46.
36. C. J. C. H. Watkins and P. Dayan, "Q-learning," Mach. Learning, vol. 8, pp. 279-292, 1992.
37. P. Wawrzynski, "Real-time reinforcement learning by sequential actor-critics and experience replay," Neural Netw., vol. 22, no. 10, pp. 1484-1497, 2009.