Reinforcement learning and optimal adaptive control: An overview and implementation examples

Annual Reviews in Control

Volumn 36, Issue 1, 2012, Pages 42-59

  Khan, Said G ^a   Herrmann, Guido ^b   Lewis, Frank L ^c   Pipe, Tony ^a   Melhuish, Chris ^b  

^a UNIVERSITY OF THE WEST OF ENGLAND   (United Kingdom)

^b UNIVERSITY OF BRISTOL   (United Kingdom)

^c UNIVERSITY OF TEXAS AT ARLINGTON   (United States)

Author keywords

ADP; Optimal adaptive control; Q learning; Reinforcement learning

Indexed keywords

ADAPTIVE CONTROL SYSTEMS; ADMINISTRATIVE DATA PROCESSING; ANTHROPOMORPHIC ROBOTS; CONTROL THEORY; CONTROLLERS; COST FUNCTIONS; DYNAMIC PROGRAMMING; ROBOTIC ARMS; ROBOTICS;

ADAPTIVE CONTROLLERS; ADAPTIVE DYNAMIC PROGRAMMING; HUMANOID ROBOT ARM; LEARNING TECHNIQUES; OPTIMAL ADAPTIVE CONTROLS; OPTIMAL CONTROL THEORY; Q-LEARNING; TRACKING CONTROLLER;

REINFORCEMENT LEARNING;

EID: 84860519689     PISSN: 13675788     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.arcontrol.2012.03.004     Document Type: Article

References (115)

1. ∞ constrained feedback control: A practical approach using neural networks 2006 Springer-Verlag UK
2. M. Abu-Khalaf, and F. Lewis Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach Automatica 41 5 2005 779 791
3. S. Adam, L. Busoniu, and R. Babuska Experience replay for real-time reinforcement learning control IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews 99 2011 1 12
4. A. Al-Tamimi, F. Lewis, and F. Abu-Khalaf Discrete-time nonlinear hjb solution using approximate dynamic programming: Convergence proof IEEE Transaction on System, Man and Cybernetics - Part B: Cybernetics 38 4 2008 943 949
5. ∞ control Automatica 43 3 2007 47 481
6. K. Althoefer, B. Krekelberg, D. Husmeier, and L. Seneviratne Reinforcement learning in a rule-based navigator for robotic manipulators Neurocomputing 37 1-4 2001 51 70
7. Atkeson, C. G.; Schaal, S. (1997). Learning tasks from a single demonstration. In IEEE international conference on robotics and automation (icra97) (pp. 1706-1712).
8. S. Balakrishnan, J. Ding, and F. Lewis Issues on stability of ADP feedback controllers for dynamical systems IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 38 4 2008 913 917
9. A. Barto, S.J. Bradtke, and S.P. Singh Learning to act using real-time dynamic programming Artificial Intelligence 72 1995 81 138
10. A. Barto, and R. Sutton Neuron-like adaptive elements that can solve difficult learning control problems IEEE Transactions on Systems, Man, and Cybernetics 5 1983 834 846
11. Barto, A.; Sutton, R. S.; Werbos, P. J. (1989). Connectionist learning for control: An overview.
12. A.G. Barto Reinforcement learning and adaptive critic methods White Sofge Handbook of intelligent control 1992 V.N. Reinhold New York 69 91
13. D. Bertsekas Neuro-dynamic programming: An overview and recent results K.-H. Waldmann, U.M. Stocker, Operations research proceedings 2006 2006 Springer Berlin Heidelberg 71 72
14. Bertsekas, D. (Ed.) (2007). Dynamic programming and optimal control. Vol. II, 3rd ed. Springer.
15. Bertsekas, D. (2009). Neuro-dynamic programming. In Encyclopedia of optimization (pp. 2555-2560).
16. Bertsekas, D. (2010a). Approximate policy iteration: A survey and some new methods. < http://stuff.mit.edu/people/dimitrib/API-Survey.pdf >.
17. D. Bertsekas Temporal difference methods for general projected equations IEEE Transactions on Automatic Control 99 2011 1
18. D. Bertsekas, and J. Tsitsiklis Neuro-dynamic programming 1996 Athena Scientific
19. Bertsekas, D.; Yu, H. (2010). Q-learning and enhanced policy iteration in discounted dynamic programming. In Proceedings of the 49th IEEE conference on decision and control, CDC (pp. 1409-1416).
20. D.P. Bertsekas Dynamic programming and suboptimal control: A survey from ADP to MPC European Journal of Control 11 4-5 2005 310 334
21. Bertsekas, D. P. (2005b). Dynamic programming and suboptimal control: A survey from ADP to MPC. In CDC proceedings.
22. Bertsekas, D. P. (2010b). Pathologies of temporal difference methods in approximate dynamic programming. In Proceedings of the 49th IEEE conference on decision and control, CDC 2010 (pp. 3034-3039).
23. S. Bhatnagar, R.S. Sutton, M. Ghavamzadeh, and M. Lee Natural actor-critic algorithms Automatica 45 2009 2471 2482
24. J.A. Boyan Technical update: Least-squares temporal difference learning Machine Learning 49 2 2002 233 246
25. S.J. Bradtke Reinforcement learning applied to linear quadratic regulation Advances in neural information processing systems 5 1993 N. Morgan Kaufman 295 302
26. Bucak, I.; Zohdy, M. (1999). Application of reinforcement learning control to a nonlinear dexterous robot. In Proceedings of the 38th IEEE conference on decision and control, 1999 (Vol. 5, pp. 5108 -5113).
27. O. Bucak, and M. Zohdy Reinforcement learning control of nonlinear multi-link system Engineering Applications of Artificial Intelligence 14 5 2001 563 575
28. Buchli, J.; Theodorou, E.; Stulp, F.; Schaal, S. (2010). variable impedance control - A reinforcement learning approach. In Robotics science and systems.
29. Busoniu, L.; Babuska, R.; De Schutter, B. (2006). Multi-agent reinforcement learning: A survey. In 9th International conference on control, automation, robotics and vision, 2006 (ICARCV '06) (pp. 1-6).
30. L. Busoniu, R. Babuska, and B. De Schutter A comprehensive survey of multiagent reinforcement learning IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews 38 2 2008 156 172
31. L. Busoniu, R. Babuska, B. Schutter, and D. Ernst Reinforcement learning and dynamic programming using function approximators 2010 CRS Press Taylor and Francis Group, USA
32. Busoniu, L.; De Schutter, B.; Babuska, R.; Ernst, D. (2010b). Using prior knowledge to accelerate online least-squares policy iteration. In 2010 IEEE international conference on automation quality and testing robotics (AQTR) (Vol. 1, pp. 1-6).
33. Busoniu, L.; Ernst, D.; De Schutter, B.; Babuska, R. (2010c). Online least-squares policy iteration for reinforcement learning control. In American control conference (ACC), 2010 (pp. 486-491).
34. D. Chang, and M.J. Er Real-time dynamic fuzzy Q-learning and control of mobile robots Proceedings of the 2nd WSEAS international conference on electronics, control and signal processing 2003 World Scientific and Engineering Academy and Society (WSEAS) Stevens Point, Wisconsin, USA 82:1 82:9
35. Digney, B. (1996). Nested Q-learning of hierarchical control structures. In IEEE international conference on neural networks, 1996 (Vol. 1, pp. 161-166).
36. D. Ernst, P. Geurts, L. Wehenkel, and L. Littman Tree-based batch mode reinforcement learning Journal of Machine Learning Research 6 2005 503 556
37. D. Ernst, M. Glavic, P. Geurts, and L. Wehenkel Approximate value iteration in the reinforcement learning context: Application to electrical power system control International Journal of Emerging Electric Power Systems 3 1 2005 1066.1 1066.37
38. Gaskett, C. (2002). Q learning for robot control: Phd thesis.
39. M. Geist, and O. Pietquin Kalman temporal differences Journal of Artificial Intelligence Research 39 2010 483 532
40. M. Geist, and O. Pietquin Revisiting natural actor-critics with value function approximation V. Torra, Y. Narukawa, M. Daumas, Proceedings of 7th international conference in modeling decisions for artificial intelligence (MDAI 2010) Lecture notes in artificial intelligence LNAI Vol. 6408 2010 Springer Verlag Heidelberg, Berlin, Perpinya (France) 207 218
41. M. Geist, O. Pietquin, and G. Fricout Tracking in reinforcement learning Proceedings of the 16th international conference on neural information processing: Part I (ICONIP '09) 2009 Springer-Verlag Berlin, Heidelberg 502 511
42. S. Girgin, and P. Preux Recent advances in reinforcement learning 2008 Springer-Verlag Berlin, Heidelberg Chapter: Basis expansion in natural actor critic methods (pp. 110-123)
43. Hafner, R.; Riedmiller, M. (2007). Neural reinforcement learning controllers for a real robot application. In IEEE international conference on robotics and automation, 2007 (pp. 2098-2103).
44. R. Hafner, and M. Riedmiller Reinforcement learning in feedback control - Challenges and benchmarks from technical process control Machine Learning 84 1-2 2011 137 169
45. P. He, and S. Jagannathan Reinforcement learning neural-network-based controller for nonlinear discrete-time systems with input constraints IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 37 2 2007 425 436
46. Heidrich-Meisner, V.; Lauer, M.; Igel, C.; Riedmiller, M. A. (2007). Reinforcement learning in a nutshell. In ESANN (pp. 277-288).
47. Hoffmann, H.; Theodorou, E.; Schaal, S. (2008). Behavioral experiments on reinforcement learning in human motor control. In Abstracts of the eighteenth annual meeting of neural control of movement.
48. Izawa, J.; Kondo, T.; Ito, K. (2002). Biological robot arm motion through reinforcement learning. In SICE 2002: Proceedings of the 41st SICE annual conference (Vol. 1, pp. 413-418).
49. L. Kael, M. Littman, and A. Moore Reinforcement learning: A survey Journal of Artificial Intelligence Research 4 0 1997 237 285
50. R.E. Kalman A new approach to linear filtering and prediction problems Journal of Basic Engineering 82 Series D 1960 35 45
51. Kampen, E.; Chu, Q.; Mulder, J. (2006). Online adaptive critic flight control using approximated plant dynamics. In International conference on machine learning and cybernetics, 2006 (pp. 256-261).
52. H. Kappen Path integrals and symmetry breaking for optimal control theory Journal of Statistical Mechanics: Theory and Experiment 2005
53. Kappen, H. (2008). An introduction to stochastic control theory, path integrals and reinforcement learning. < http://www.snn.ru.nl >.
54. Khan, S.; Herrmann, G.; Lewis, F.; Pipe, T.; Melhuish, C. (2011a). A novel Q-learning based adaptive optimal controller implementation for a humanoid robotic arm. In IFAC 2011 world congress. Milan, Italy.
55. Khan, S.; Herrmann, G.; Lewis, F.; Pipe, T.; Melhuish, C. (2011b). Q-learning based cartesian model reference compliance controller implementation for a humanoid robot arm. In IEEE international conference CIS/RAM 2011. China.
56. Khan, S.; Herrmann, G.; Pipe, T.; Melhuish, C. (2010a). Adaptive multi-dimensional compliance control of a humanoid robotic arm with anti-windup compensation. In IEEE/RSJ international conference on intelligent robots and systems (IROS), 2010 (pp. 2218-2223).
57. S. Khan, G. Herrmann, T. Pipe, C. Melhuish, and A. Spiers Safe adaptive compliance control of a humanoid robotic arm with anti-windup compensation and posture control International Journal of Social Robotics 2 2010 305 319
58. Khan, S.; Lenz, A.; Herrmann, G.; Pipe, T.; Melhuish, C. (2011c). Towards safe human robot interaction: Integration of compliant control, an anthropomorphic hand and verbal communication. In FIRA 2011 conference: ICAHRR. Taiwan.
59. S.G. Khan, W. Naeem, R. Sutton, and S. Sharma Application of soft computing techniques to a LQG controller design X.-T. Yan, C. Jiang, B. Eynard, Advanced design and manufacture to gain a competitive edge 2008 Springer London 137 146
60. Kim, B.; Kang, B.; Park, S.; Kang, S. (2008). Learning robot stiffness for contact tasks using the natural actor-critic. In IEEE international conference on robotics and automation, 2008 (ICRA 2008) (pp. 3832-3837).
61. B. Kim, J. Park, S. Park, and S. Kang Impedance learning for robotic contact task using natural actor-critic algorithm IEEE Transaction on System, Man and Cybernetics, part B: Cybernetics 4 2 2010
62. C. Kuan, and K. Young Reinforcement learning and robust control for robot compliance tasks Journal of Intelligent and Robotic Systems 23 1998 165 182
63. V. Kuzmin Connectionist Q-learning in robot control task 2002
64. M.G. Lagoudakis, and R. Parr Least-squares policy iteration Journal of Machine Learning Research 4 2003 1107 1149
65. F. Lewis, D. Dawson, and C. Abdallah Robot manipulator control: Theory and practice 2003 Marcel Dekker Inc New York, USA
66. F. Lewis, and K. Vamvoudakis Reinforcement learning for partially observable dynamic processes: Adaptive dynamic programming using measured output data IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 41 1 2011 14 25
67. F.L. Lewis, and D. Vrabie Reinforcement learning and adaptive dynamic programming for feedback control Circuits and Systems Magazine 09 3 2009 32 50
68. Li, L.; Littman, M. L.; Mansley, C. (2009). Online exploration in least-squares policy iteration. In Proceedings of 8th international conference on autonomous agents and multiagent systems (AAMAS 2009).
69. Lin, W.; Hui, P.; Hua-yong, Z.; Lin-cheng, S. (2009). A survey of approximate dynamic programming. In International conference on intelligent human-machine systems and cybernetics, 2009 (IHMSC '09) (Vol. 2, pp. 396-399).
70. Liu, W.; Tan, Y.; Qiu, Q. (2010). Enhanced Q-learning algorithm for dynamic power management with performance constraint. In Design, automation test in Europe conference exhibition (DATE), 2010 (pp. 602-605).
71. H.R. Maei, and R.S. Sutton Gq(λ): A general gradient algorithm for temporal-difference prediction learning with eligibility traces Proceedings of the 3d conference on artificial general intelligence, AGI10 2010 Atlantis Press 1 6
72. J. Peng, and R.J. Williams Incremental multi-step Q-learning Machine learning 1996 M. Kaufmann 226 232
73. J. Peters, and S. Schaal Learning to control in operational space International Journal of Robotics Research 2008 197 212
74. J. Peters, and S. Schaal Natural actor-critic Neurocomputing 71 7-9 2008 1180 1190
75. Peters, J. Vijayakumar, S.; Schaal, S. (2003). Reinfrocement learning for humanoid robotics. In Third IEEE-RAS international conference on humanoid robots. Karlsruhe, Germany.
76. Pietquin, O.; Geist, M.; Chandramohan, S.; Frezza-Buet, H. (2011). Sample efficient on-line learning of optimal dialogue policies with Kalman temporal differences. In International joint conference on artificial intelligence (IJCAI 2011) (pp. 1878-1883). Barcelona, Spain.
77. A.G. Pipe An architecture for learning potential field cognitive maps with an application to mobile robot navigation Adaptive Behavior 8 2 2000 173 204
78. D. Prokhorov, and D. Wunsch Adaptive critic designs IEEE Transactions on Neural Networks 8 5 1997 997 1007
79. M. Riedmiller Neural fitted Q iteration first experiences with a data efficient neural reinforcement learning method In 16th European conference on machine learning 2005 Springer 317 328
80. M. Riedmiller, T. Gabel, R. Hafner, and S. Lange Reinforcement learning for robot soccer Autonomous Robots 27 1 2009 55 73
81. Riedmiller, M.; Peters, J.; Schaal, S. (2007). Evaluation of policy gradient methods and variants on the cart-pole benchmark. In IEEE international symposium on approximate dynamic programming and reinforcement learning, 2007 (ADPRL 2007) (pp. 254-261).
82. Rudowsky, I.; Kulyba, M.; Kunin, M.; Parsons, S.; Raphan, T. (2006). Reinforcement learning interfaces for biomedical database systems. In Proceedings of the 28th IEEE EMBS annual international conference (pp. 6269-6272). New York City, USA.
83. J. Sarangapani Neural network control of nonlinear discrete-time systems 2006 Taylor & Francis USA
84. S. Schaal Learning by demontration Advances in Neural Information Processing Systems 9 1997 1040 1046
85. Shah, H.; Gopal, M. (2009). Reinforcement learning control of robot manipulators in uncertain environments. In IEEE international conference on industrial technology, 2009 (ICIT 2009) (pp. 1-6).
86. Shoham, Y.; Powers, R.; Grenager, T. (2003). Multi-agent reinforcement learning: A critical survey. Tech. rep.; Stanford University. < http://citeseerx.ist.psu.edu/viewdoc/summary? >.
87. J. Si, A. Barto, W. Powell, and D. Wunsch Handbook of learning and approximate dynamic programming 2004 IEEE Press/Wiley
88. Smart, W.; Kaelbling, L. (2002). Reinforcement learning for robot control. In D. Gage, & H. Choset (Eds.), Society of photo-optical instrumentation engineers (SPIE) conference series (Vol. 4573, pp. 92-103).
89. P. Stingu, and F. Lewis Adaptive dynamic programming applied to a 6dof quadrotor B. Igelnik, Computational modeling and simulation of intellect: Current state and future perspectives 2010 IGI Global
90. Sutton, R. (1984). Temporal credit assigning in reinforcement learning: Phd thesis.
91. Sutton, R.; Barto, A. G.; Williams, R. J. (1992). Reinforcement learning is direct adaptive optimal control. In Proceedings of the American control conference (pp. 2143-2146).
92. R.S. Sutton Reinforcement learning: Past, present and future Selected papers from the second Asia-Pacific conference on simulated evolution and learning (SEAL'98) 1999 Springer-Verlag London, UK 195 197
93. R.S. Sutton, and A.G. Barto Reinforcement learning: An introduction 1998 The MIT Press Cambridge, Massachusetts London, England
94. Sutton, R. S.; Maei, H. R.; Precup, D.; Bhatnagar, S.; Silver, D.; Szepesvri, C.; et al. (2009). Fast gradient-descent methods for temporal-difference learning with linear function approximation. In Proceedings of the 26th international conference on machine learning.
95. Theodorou, E.; Buchli, J.; Schaal, S. (2010). Reinforcement learning of motor skills in high dimensions: A path integral approach. In IEEE international conference on robotics and automation (ICRA), 2010 (pp. 2397-2403).
96. Theodorou, E.; Peters, J.; Schaal, S. (2007). Reinforcement learning for optimal control of arm movements. In Abstracts of the 37st meeting of the society of neuroscience.
97. S. Thrun, and M. Littman A review of reinforcement learning AI Magazine 21 2000 103 105
98. K. Vamvoudakis, and F. Lewis Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem Automatica 46 5 2010 878 888
99. Van Roy, B.; Bertsekas, D.; Lee, Y.; Tsitsiklis, J. (1997). A neuro-dynamic programming approach to retailer inventory management. In Proceedings of the 36th IEEE conference on decision and control, 1997 (Vol. 4, pp. 4052-4057).
100. D. Vrabie, and F. Lewis Neural network approach to continuous-time direct adaptive optimal control for partially unknown nonlinear systems Neural Networks 22 3 2009 237 246 goal-directed neural system
101. D. Vrabie, O. Pastravanu, M. Abu-Khalaf, and F. Lewis Adaptive optimal control for continuous-time linear systems based on policy iteration Automatica 45 2 2009 477 484
102. Watkins, C. (1989). Learning from delayed rewards: Phd thesis.
103. C. Watkins, and P. Dayan Technical note: Q-learning Machine Learning 8 1992 279 292
104. G. Welch, and G. Bishop An introduction to the Kalman filter Design 7 1 2001 1 16
105. P. Werbos Approximate dynamic programming for real-time control and neural modeling White Sofge Handbook of intelligent control 1992 V.N. Reinhold New York, USA 493 525
106. P. Werbos Neurocontrol and supervised learning: An overview and evaluationg White Sofge Handbook of intelligent control 1992 V.N. Reinhold New York, USA 65 89
107. P. Werbos 1 ADP: Goals, opportunities and principles J. SiBarto, W. Powell, D. Wunsch, Handbook of learning and approximate dynamic programming 2004 Wiley-IEEE Press Piscataway, NJ, USA 3 44
108. P. Werbos Using ADP to understand and replicate brain intelligence: The next level design L. Perlovsky, R. Kozma, Neurodynamics of cognition and consciousness Understanding complex systems Vol. 25 2007 Springer Berlin/Heidelberg 109 123
109. P. Werbos Intelligence in the brain: A theory of how it works and how to build it Neural Networks 22 3 2009 200 212 goal-directed neural system
110. P.J. Werbos Consistency of HDP applied to a simple reinforcement learning problem Neural Networks 3 2 1990 179 189
111. P.J. Werbos Foreword - ADP: The key direction for future research in intelligent control and understanding brain intelligence IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 38 4 2008 898 900
112. D.A. White, and D.A. Sorge Sofge Handbook of intelligent control 1992 Van Nostrant Reinhold
113. J.K. Williams Reinforcement learning of optimal controls S.E. Haupt, A. Pasini, C. Marzban, Artificial intelligence methods in the environmental sciences 2009 Springer Netherlands 297 327
114. Xin, L.; Balakrishnan, S. (2000). Convergence analysis of adaptive critic based optimal control. In Proceedings of the American control conference, 2000 (Vol. 3. pp. 1929-1933).
115. X. Xu, D. Hu, and X. Lu Kernel-based least squares policy iteration for reinforcement learning IEEE Transactions on Neural Networks 18 4 2007 973 992