Approximate reinforcement learning: An overview

IEEE SSCI 2011: Symposium Series on Computational Intelligence - ADPRL 2011: 2011 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning

Volumn , Issue , 2011, Pages 1-8

  Buşoniu, Lucian ^a   Ernst, Damien ^b   De Schutter, Bart ^a   Babuška, Robert ^a  

^a DELFT UNIVERSITY OF TECHNOLOGY   (Netherlands)

^b UNIVERSITY OF LIÈGE   (Belgium)

Author keywords

function approximation; policy iteration; policy search; reinforcement learning; value iteration

Indexed keywords

COMPLEX ENVIRONMENTS; FUNCTION APPROXIMATION; OFFLINE; ON-LINE ALGORITHMS; POLICY GRADIENT METHODS; POLICY ITERATION; POLICY SEARCH; SIMULATION-BASED; VALUE ITERATION;

APPROXIMATION ALGORITHMS; ARTIFICIAL INTELLIGENCE; GRADIENT METHODS; REINFORCEMENT LEARNING; ROBOTICS;

DYNAMIC PROGRAMMING;

EID: 80052220856     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ADPRL.2011.5967353     Document Type: Conference Paper

References (82)

1. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. MIT Press, 1998.
2. Cs. Szepesvári, Algorithms for Reinforcement Learning. Morgan & Claypool Publishers, 2010.
3. O. Sigaud and O. Buffet, Eds., Markov Decision Processes in Artificial Intelligence. Wiley, 2010.
4. D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming. Athena Scientific, 1996.
5. J. Si, A. G. Barto, and W. B. Powell, Eds., Handbook of Learning and Approximate Dynamic Programming. Wiley, 2004.
6. D. P. Bertsekas, Dynamic Programming and Optimal Control, 3rd ed. Athena Scientific, 2007, vol. 2.
7. L. Buşoniu, R. Babuška, B. De Schutter, and D. Ernst, Reinforcement Learning and Dynamic Programming Using Function Approximators, ser. Automation and Control Engineering. Taylor & Francis CRC Press, 2010.
8. J. Peters and S. Schaal, "Reinforcement learning of motor skills with policy gradients," Neural Networks, vol. 21, pp. 682-697, 2008.
9. P. Abbeel, A. Coates, M. Quigley, and A. Y. Ng, "An application of reinforcement learning to aerobatic helicopter flight," in Advances in Neural Information Processing Systems 19, B. Schölkopf, J. C. Platt, and T. Hoffman, Eds. MIT Press, 2007, pp. 1-8.
10. J. DiGiovanna, B. Mahmoudi, J. Fortes, J. C. Principe, and J. C. Sanchez, "Coadaptive brain-machine interface via reinforcement learning," IEEE Transactions on Biomedical Engineering, vol. 56, no. 1, pp. 54-64, 2009.
11. D. Ernst, G.-B. Stan, J. Gonçalves, and L. Wehenkel, "Clinical data based optimal STI strategies for HIV: A reinforcement learning approach," in Proceedings 45th IEEE Conference on Decision & Control, San Diego, US, 13-15 December 2006, pp. 667-672. (Pubitemid 351283311)
12. D. P. Bertsekas and S. E. Shreve, Stochastic Optimal Control: The Discrete Time Case. Academic Press, 1978.
13. L. P. Kaelbling, M. L. Littman, and A. W. Moore, "Reinforcement learning: A survey," Journal of Artificial Intelligence Research, vol. 4, pp. 237-285, 1996. (Pubitemid 126646155)
14. C. J. C. H. Watkins and P. Dayan, "Q-learning," Machine Learning, vol. 8, pp. 279-292, 1992.
15. G. A. Rummery and M. Niranjan, "On-line Q-learning using connectionist systems," Engineering Department, Cambridge University, UK, Tech. Rep. CUED/F-INFENG/TR166, September 1994, available at http://mi.eng.cam.ac.uk/reports/svr-ftp/rummery tr166.ps.Z.
16. D. Ernst, P. Geurts, and L. Wehenkel, "Tree-based batch mode reinforcement learning," Journal of Machine Learning Research, vol. 6, pp. 503-556, 2005.
17. M. Riedmiller, "Neural fitted Q-iteration - first experiences with a data efficient neural reinforcement learning method," in Proceedings 16th European Conference on Machine Learning (ECML-05), ser. Lecture Notes in Computer Science, vol. 3720, Porto, Portugal, 3-7 October 2005, pp. 317-328.
18. A. Antos, R. Munos, and Cs. Szepesvári, "Fitted Q-iteration in continuous action-space MDPs," in Advances in Neural Information Processing Systems 20, J. C. Platt, D. Koller, Y. Singer, and S. T. Roweis, Eds. MIT Press, 2008, pp. 9-16.
19. S. P. Singh, T. Jaakkola, and M. I. Jordan, "Reinforcement learning with soft state aggregation," in Advances in Neural Information Processing Systems 7, G. Tesauro, D. S. Touretzky, and T. K. Leen, Eds. MIT Press, 1995, pp. 361-368.
20. Cs. Szepesvári and W. D. Smart, "Interpolation-based Q-learning," in Proceedings 21st International Conference on Machine Learning (ICML- 04), Bannf, Canada, 4-8 July 2004, pp. 791-798. (Pubitemid 40290882)
21. P. Y. Glorennec, "Reinforcement learning: An overview," in Proceedings European Symposium on Intelligent Techniques (ESIT-00), Aachen, Germany, 14-15 September 2000, pp. 17-35.
22. L.-J. Lin, "Self-improving reactive agents based on reinforcement learning, planning and teaching," Machine Learning, vol. 8, no. 3-4, pp. 293-321, Aug. 1992, special issue on reinforcement learning.
23. G. Neumann and J. Peters, "Fitted Q-iteration by advantage weighted regression," in Advances in Neural Information Processing Systems 21, D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, Eds. MIT Press, 2009, pp. 1177-1184.
24. D. Ormoneit and S. Sen, "Kernel-based reinforcement learning," Machine Learning, vol. 49, no. 2-3, pp. 161-178, 2002. (Pubitemid 34325684)
25. R. Munos and Cs. Szepesvári, "Finite time bounds for fitted value iteration," Journal of Machine Learning Research, vol. 9, pp. 815-857, 2008.
26. A. M. Farahmand, M. Ghavamzadeh, Cs. Szepesvári, and S. Mannor, "Regularized fitted Q-iteration for planning in continuous-space Markovian decision problems," in Proceedings 2009 American Control Conference (ACC-09), St. Louis, US, 10-12 June 2009, pp. 725-730.
27. F. S. Melo, S. P. Meyn, and M. I. Ribeiro, "An analysis of reinforcement learning with function approximation," in Proceedings 25th International Conference on Machine Learning (ICML-08), Helsinki, Finland., 5-9 July 2008, pp. 664-671.
28. S. Jodogne, C. Briquet, and J. H. Piater, "Approximate policy iteration for closed-loop learning of visual tasks," in Proceedings 17th European Conference on Machine Learning (ECML-06), ser. Lecture Notes in Computer Science, vol. 4212, Berlin, Germany, 18-22 September 2006, pp. 210-221. (Pubitemid 44618833)
29. A. Antos, Cs. Szepesvári, and R. Munos, "Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path," Machine Learning, vol. 71, no. 1, pp. 89-129, 2008.
30. A. M. Farahmand, M. Ghavamzadeh, Cs. Szepesvári, and S. Mannor, "Regularized policy iteration," in Advances in Neural Information Processing Systems 21, D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, Eds. MIT Press, 2009, pp. 441-448.
31. M. G. Lagoudakis and R. Parr, "Least-squares policy iteration," Journal of Machine Learning Research, vol. 4, pp. 1107-1149, 2003.
32. S. J. Bradtke and A. G. Barto, "Linear least-squares algorithms for temporal difference learning," Machine Learning, vol. 22, no. 1-3, pp. 33-57, 1996. (Pubitemid 126724362)
33. C. Thiery and B. Scherrer, "Least-squares λ policy iteration: Biasvariance trade-off in control problems," in Proceedings 27th International Conference on Machine Learning (ICML-10), Haifa, Israel, 21-24 June 2010, pp. 1071-1078.
34. D. P. Bertsekas and S. Ioffe, "Temporal differences-based policy iteration and applications in neuro-dynamic programming," Massachusetts Institute of Technology, Cambridge, US, Tech. Rep. LIDS-P-2349, 1996, available at http://web.mit.edu/dimitrib/www/Tempdif.pdf.
35. D. P. Bertsekas, "Approximate policy iteration: A survey and some new methods," Massachusetts Institute of Technology, Cambridge, US, Tech. Rep. LIDS 2833, July 2010.
36. M. G. Lagoudakis and R. Parr, "Reinforcement learning as classification: Leveraging modern classifiers," in Proceedings 20th International Conference on Machine Learning (ICML-03). Washington, US, 21-24 August 2003, pp. 424-431.
37. C. Dimitrakakis and M. Lagoudakis, "Rollout sampling approximate policy iteration," Machine Learning, vol. 72, no. 3, pp. 157-171, 2008.
38. T. Jung and D. Polani, "Learning RoboCup-keepaway with kernels," in Gaussian Processes in Practice, ser. JMLR Workshop and Conference Proceedings, vol. 1, 2007, pp. 33-57.
39. L. Buşoniu, D. Ernst, B. De Schutter, and R. Babuška, "Online leastsquares policy iteration for reinforcement learning control," in Proceedings 2010 American Control Conference (ACC-10), Baltimore, US, 30 June - 2 July 2010, pp. 486-491.
40. R. S. Sutton, "Generalization in reinforcement learning: Successful examples using sparse coarse coding," in Advances in Neural Information Processing Systems 8, D. S. Touretzky, M. C. Mozer, and M. E. Hasselmo, Eds. MIT Press, 1996, pp. 1038-1044.
41. L. Li, M. L. Littman, and C. R. Mansley, "Online exploration in least-squares policy iteration," in Proceedings 8th International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS-09), vol. 2, Budapest, Hungary, 10-15 May 2009, pp. 733-739.
42. R. Sutton, H. Maei, D. Precup, S. Bhatnagar, D. Silver, Cs. Szepesvari, and E. Wiewiora, "Fast gradient-descent methods for temporal-difference learning with linear function approximation," in Proceedings 26th International Conference on Machine Learning (ICML-09), Montreal, Canada, 14-18 June 2009, pp. 993-1000.
43. H. R. Maei, C. Szepesvári, S. Bhatnagar, and R. S. Sutton, "Toward offpolicy learning control with function approximation," in Proceedings 27th International Conference on Machine Learning (ICML-10), Haifa, Israel, 21-24 June 2010, pp. 719-726.
44. A. Lazaric, M. Ghavamzadeh, and R. Munos, "Analysis of a classification-based policy iteration algorithm," in Proceedings 27th International Conference on Machine Learning (ICML-10), Haifa, Israel, 21-24 June 2010, pp. 607-614.
45. H. Yu, "Convergence of least squares temporal difference methods under general conditions," in Proceedings 27th International Conference on Machine Learning (ICML-10), Haifa, Israel, 21-24 June 2010, pp. 1207-1214.
46. B. Scherrer, "Should one compute the Temporal Difference fix point or minimize the Bellman Residual? the unified oblique projection view," in Proceedings 27th International Conference on Machine Learning (ICML- 10), Haifa, Israel, 21-24 June 2010, pp. 959-966.
47. A. Lazaric, M. Ghavamzadeh, and R. Munos, "Finite-sample analysis of LSTD," in Proceedings 27th International Conference on Machine Learning (ICML-10), Haifa, Israel, 21-24 June 2010, pp. 615-622.
48. P. Marbach and J. N. Tsitsiklis, "Approximate gradient methods in policy-space optimization of Markov reward processes," Discrete Event Dynamic Systems: Theory and Applications, vol. 13, no. 1-2, pp. 111- 148, 2003.
49. A. G. Barto, R. S. Sutton, and C. W. Anderson, "Neuronlike adaptive elements that can solve difficult learning control problems," IEEE Transactions on Systems, Man, and Cybernetics, vol. 13, no. 5, pp. 833-846, 1983.
50. R. S. Sutton, D. A. McAllester, S. P. Singh, and Y. Mansour, "Policy gradient methods for reinforcement learning with function approximation," in Advances in Neural Information Processing Systems 12, S. A. Solla, T. K. Leen, and K.-R. Müller, Eds. MIT Press, 2000, pp. 1057-1063.
51. V. R. Konda and J. N. Tsitsiklis, "On actor-critic algorithms," SIAM Journal on Control and Optimization, vol. 42, no. 4, pp. 1143-1166, 2003.
52. S. Kakade, "A natural policy gradient," in Advances in Neural Information Processing Systems 14, T. G. Dietterich, S. Becker, and Z. Ghahramani, Eds. MIT Press, 2001, pp. 1531-1538.
53. J. Peters and S. Schaal, "Natural actor-critic," Neurocomputing, vol. 71, no. 7-9, pp. 1180-1190, 2008.
54. S. Bhatnagar, R. Sutton, M. Ghavamzadeh, and M. Lee, "Natural actorcritic algorithms," Automatica, vol. 45, no. 11, pp. 2471-2482, 2009.
55. F. J. Gomez, J. Schmidhuber, and R. Miikkulainen, "Efficient nonlinear control through neuroevolution," in Proceedings 17th European Conference on Machine Learning (ECML-06), ser. Lecture Notes in Computer Science, vol. 4212, Berlin, Germany, 18-22 September 2006, pp. 654-662. (Pubitemid 44618874)
56. H. S. Chang, M. C. Fu, J. Hu, and S. I. Marcus, Simulation-Based Algorithms for Markov Decision Processes. Springer, 2007.
57. S. Mannor, R. Y. Rubinstein, and Y. Gat, "The cross-entropy method for fast policy search," in Proceedings 20th International Conference on Machine Learning (ICML-03), Washington, US, 21-24 August 2003, pp. 512-519.
58. L. Buşoniu, D. Ernst, B. De Schutter, and R. Babuška, "Cross-entropy optimization of control policies with adaptive basis functions," IEEE Transactions on Systems, Man, and Cybernetics-Part B: Cybernetics, vol. 41, no. 1, 2011, accepted for publication, available online.
59. J. N. Tsitsiklis and B. Van Roy, "Feature-based methods for large scale dynamic programming," Machine Learning, vol. 22, no. 1-3, pp. 59-94, 1996. (Pubitemid 126724363)
60. S. Whiteson and P. Stone, "Evolutionary function approximation for reinforcement learning," Journal of Machine Learning Research, vol. 7, pp. 877-917, 2006. (Pubitemid 43736560)
61. P. W. Keller, S. Mannor, and D. Precup, "Automatic basis function construction for approximate dynamic programming and reinforcement learning," in Proceedings 23rd International Conference on Machine Learning (ICML-06), Pittsburgh, US, 25-29 June 2006, pp. 449-456.
62. P. Preux, S. Girgin, and M. Loth, "Feature discovery in approximate dynamic programming," in Proceedings IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL-09), 30 March - 2 April 2009, pp. 109-116.
63. Y. Engel, S. Mannor, and R. Meir, "Reinforcement learning with Gaussian processes," in Proceedings 22nd International Conference on Machine Learning (ICML-05), Bonn, Germany, 7-11 August 2005, pp. 201-208. (Pubitemid 43183334)
64. G. Taylor and R. Parr, "Kernelized value function approximation for reinforcement learning," in Proceedings 26th International Conference on Machine Learning (ICML-09), Montreal, Canada, 14-18 June 2009, pp. 1017-1024.
65. I. Menache, S. Mannor, and N. Shimkin, "Basis function adaptation in temporal difference reinforcement learning," Annals of Operations Research, vol. 134, no. 1, pp. 215-238, 2005. (Pubitemid 40550047)
66. S. Mahadevan and M. Maggioni, "Proto-value functions: A Laplacian framework for learning representation and control in Markov decision processes," Journal of Machine Learning Research, vol. 8, pp. 2169-2231, 2007. (Pubitemid 350046199)
67. J. Z. Kolter and A. Ng, "Regularization and feature selection in leastsquares temporal difference learning," in Proceedings 26th International Conference on Machine Learning (ICML-09), Montreal, Canada, 14-18 June 2009, pp. 521-528.
68. R. Parr, L. Li, G. Taylor, C. Painter-Wakefield, and M. Littman, "An analysis of linear models, linear value-function approximation, and feature selection for reinforcement learning," in Proceedings 25th Annual International Conference on Machine Learning (ICML-08), Helsinki, Finland, 5-9 July 2008, pp. 752-759.
69. H. van Hasselt and M. Wiering, "Reinforcement learning in continuous action spaces," in Proceedings IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning (ADPRL-07), Honolulu, US, 1-5 April 2007, pp. 272-279. (Pubitemid 47431396)
70. J. Pazis and M. Lagoudakis, "Binary action search for learning continuous-action control policies," in Proceedings of the 26th International Conference on Machine Learning (ICML-09), Montreal, Canada, 14-18 June 2009, pp. 793-800.
71. R. S. Sutton, "Integrated architectures for learning, planning, and reacting based on approximating dynamic programming," in Proceedings 7th International Conference on Machine Learning (ICML-90), Austin, US, 21-23 June 1990, pp. 216-224.
72. M. van Otterlo, The Logic Of Adaptive Behavior. IOS Press, 2009.
73. J.-Y. Audibert, R. Munos, and Cs. Szepesvári, "Tuning bandit algorithms in stochastic environments," in Proceedings 18th International Conference on Algorithmic Learning Theory (ALT-07), Sendai, Japan, 1-4 October 2007, pp. 150-165.
74. S. Bubeck, R. Munos, G. Stoltz, and C. Szepesvári, "Online optimization in X-armed bandits," in Advances in Neural Information Processing Systems 21, D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, Eds. MIT Press, 2009, pp. 201-208.
75. B. Kveton, M. Hauskrecht, and C. Guestrin, "Solving factored MDPs with hybrid state and action variables," Journal of Artificial Intelligence Research, vol. 27, pp. 153-201, 2006. (Pubitemid 44681376)
76. F. Lewis, D. Liu, and G. Lendaris, "Guest editorial - special issue on adaptive dynamic programming and reinforcement learning in feedback control," IEEE Transactions on Systems, Man, and Cybernetics-Part B: Cybernetics, vol. 38, no. 4, pp. 896-897, 2008.
77. W. B. Powell, Approximate Dynamic Programming: Solving the Curses of Dimensionality. Wiley, 2007.
78. X.-R. Cao, Stochastic Learning and Optimization: A Sensitivity-Based Approach. Springer, 2007.
79. D. P. Bertsekas, "Approximate dynamic programming," 20 November 2010, update of Chapter 6 in volume 2 of the book Dynamic Programming and Optimal Control. Available at http://web.mit.edu/dimitrib/www/dpchapter.html.
80. V. Heidrich-Meisner, M. Lauer, C. Igel, and M. Riedmiller, "Reinforcement learning in a nutshell," in Proceedings 15th European Symposium on Artificial Neural Networks (ESANN-07), Bruges, Belgium, 25-27 April 2007, pp. 277-288.
81. D. P. Bertsekas, "Dynamic programming and suboptimal control: A survey from ADP to MPC," European Journal of Control, vol. 11, no. 4-5, pp. 310-334, 2005, special issue for the CDC-ECC-05 in Seville, Spain.
82. J. Rust, "Numerical dynamic programming in economics," in Handbook of Computational Economics, H. M. Amman, D. A. Kendrick, and J. Rust, Eds. Elsevier, 1996, vol. 1, ch. 14, pp. 619-729.