Optimal adaptive control and differential games by reinforcement learning principles

Optimal Adaptive Control and Differential Games by Reinforcement Learning Principles

Volumn , Issue , 2012, Pages 1-289

  Vrabie, Draguna ^a   Vamvoudakis, Kyriakos G ^b   Lewis, Frank L ^c  

^a UNITED TECHNOLOGIES RESEARCH CENTER   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c UNIVERSITY OF TEXAS AT ARLINGTON   (United States)

Author keywords

Adaptive control; Differential games; Dynamic feedback control systems; Feedback; Learning (artificial intelligence); Online differential games; Optimal adaptive control; Optimal control; Reinforcement learning principles

Indexed keywords

ARTIFICIAL INTELLIGENCE; AUTOMATION; CONTINUOUS TIME SYSTEMS; CONTROL SYSTEMS; CONTROLLERS; DISTURBANCE REJECTION; DYNAMIC PROGRAMMING; DYNAMICS; FEEDBACK; FEEDBACK CONTROL; GAME THEORY; HAMILTONIANS; LEARNING SYSTEMS; MULTI AGENT SYSTEMS; ONLINE SYSTEMS; OPTIMAL CONTROL SYSTEMS; REAL TIME SYSTEMS; REINFORCEMENT LEARNING; ROBUST CONTROL;

ADAPTIVE CONTROL; DIFFERENTIAL GAMES; DYNAMIC FEEDBACK; LEARNING (ARTIFICIAL INTELLIGENCE); OPTIMAL ADAPTIVE CONTROLS; OPTIMAL CONTROLS;

ADAPTIVE CONTROL SYSTEMS;

EID: 85013129810     PISSN: None     EISSN: None     Source Type: Book    
DOI: None     Document Type: Book

References (121)

1. Abu-Khalaf M., Lewis F. L. ‘Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach’, Automatica. 2005;41(5):779-791.
2. Abu-Khalaf M., Lewis F. L. ‘Neurodynamic programming and zero-sum games for constrained control systems’, IEEE Transactions on Neural Networks. 2008;19(7):1243-1252.
3. Abu-KhalafM., Lewis F. L., Huang J. ‘Policy iterations and the Hamilton-Jacobi-Isaacs equation for H-infinity state feedback control with input saturation’, IEEE Transactions on Automatic Control. 2006;51(12):1989-1995.
4. Adams R., Fournier J. Sobolev Spaces. New York: Academic Press; 2003.
5. Al-Tamimi A., Abu-Khalaf M., Lewis F. L. ‘Model-free Q-learning designs for discrete-time zero-sum games with application to H-infinity control’, Automatica. 2007a;43(3):473-482.
6. Al-Tamimi A., Abu-Khalaf M., Lewis F. L. ‘Adaptive critic designs for discretetime zero-sum games with application to H-infinity control’, IEEE Transactions on Systems, Man and Cybernetics-B. 2007b;37(1).
7. Al-Tamimi A., Lewis F. L., Abu-Khalaf M. ‘Discrete-time nonlinear HJB solution using approximate dynamic programming: convergence proof’, IEEE Transactions on Systems, Man, Cybernetics, Part B. 2008;38(4):943-949, special issue on ADP/RL.
8. Astrom K. J., Wittenmark B. Adaptive Control. Addison-Wesley; 1995.
9. Baird L. C. III. ‘Reinforcement learning in continuous time: Advantage updating’, Proceedings of the ICNN. 1994, Orlando FL.
10. Balakrishnan S. N., Ding J., Lewis F. L. ‘Issues on stability of ADP feedback controllers for dynamical systems’, IEEE Transactions on Systems, Man, Cybernetics, Part B. 2008;38(4):913-917, special issue on ADP/RL, invited survey paper.
11. Ball J., Helton W. ‘Viscosity solutions of Hamilton-Jacobi equations arising in nonlinear Hx-control’, Journal of Mathematical Systems, Estimation and Control. 1996;6(1):1-22.
12. Balzer L. A. ‘Accelerated convergence of the matrix sign function method of solving Lyapunov, Riccati and other equations’, International Journal of Control. 1980;32(6):1076-1078.
13. Banks H. T., Ito K. ‘A numerical algorithm for optimal feedback gains in high dimensional linear quadratic regulator problems’, SIAM Journal on Control Optimal. 1991;29(3):499-515.
14. Bardi M., Capuzzo-Dolcetta I. Optimal Control and Viscosity Solutions of Hamilton-Jacobi-Bellman Equations. Boston, MA: Birkhâuser; 1997.
15. Barto A. G., Sutton R. S., Anderson C. ‘Neuron-like adaptive elements that can solve difficult learning control problems’, IEEE Transactions on Systems, Man, and Cybernetics. SMC-1983;13:834-846.
16. Başar T., Bernard P. Optimal Control and Related Minimax Design Problems. Boston, MA: Birkhâuser; 1995.
17. Başar T., Olsder G. J. Dynamic Noncooperative Game Theory, 2nd ed. (Classics in Applied Mathematics; 23), SIAM; 1999.
18. Beard R., Saridis G., Wen J. ‘Galerkin approximations of the generalized Hamilton-Jacobi-Bellman equation’, Automatica. 1997;33(12):2159-2177.
19. Bellman R. E. Dynamic Programming. Princeton, NJ: Princeton University Press; 1957.
20. Bertsekas D. P., Tsitsiklis J. N. Neuro-Dynamic Programming. MA: Athena Scientific; 1996.
21. Bhasin S., Johnson M., Dixon W. E. ‘A model free robust policy iteration algorithm for optimal control of nonlinear systems’, Proceedings of the 49th IEEE Conference on Decision and Control. 2010, 3060-3065.
22. Bradtke S. J., Ydestie B. E., Barto A. G. ‘Adaptive linear quadratic control using policy iteration’, Proceedings of the ACC. 1994, 3475-3476.
23. Brewer J. W. ‘Kronecker products and matrix calculus in system theory’, IEEE Transactions on Circuit and System. 1978;25(9):772-781.
24. Busoniu L., Babuska R., De Schutter B., Ernst D. Reinforcement Learning and Dynamic Programming Using Function Approximators. Boca Raton, FL: CRC Press; 2009.
25. Byers R. ‘Solving the algebraic Riccati equation with the matrix sign’, Linear Algebra and its Applications. 1987;85:267-279.
26. Callier F. M., Desoer C. A. Linear Systems Theory. New York: Springer-Verlag; 1991.
27. Cao X. Stochastic Learning and Optimization. Berlin: Springer-Verlag; 2007.
28. Chen B. M., Lin Z., Shamash Y. Linear Systems Theory: A Structural Decomposition Approach. Boston, MA: Birkhauser; 2004.
29. Cherfi L., Abou-Kandil H., Bourles H. ‘Iterative method for general algebraic Riccati equation’, Proceedings of the ACSE’05. 2005.
30. Damm T. Rational Matrix Equations in Stochastic Control. Berlin, Germany: Springer-Verlag; 2004.
31. Darwin C. On the Origin of Species by Means of Natural Selection. London: John Murray; 1859.
32. Dierks T., Jagannathan S. ‘Optimal control of affine nonlinear continuous-time systems using an online Hamilton-Jacobi-Isaacs formulation’, Proceedings of the IEEE Conference on Decision and Control. 2010, 3048-3053.
33. Doya K. ‘Reinforcement learning in continuous time and space’, Neural Computation. 2000;12(1):219-245.
34. Doya K., Kimura H., Kawato M. ‘Neural mechanisms of learning and control’, IEEE Control Systems Magazine. 2001;21(4):42-54.
35. Doyle J., Glover K., Khargonekar P., Francis B. ‘State-space solutions to standard H2 and H-infinity control problems’, IEEE Transactions on Automatic Control. 1989;34:831-847.
36. Feng Y., Anderson B. D., Rotkowitz M. ‘A game theoretic algorithm to compute local stabilizing solutions to HJBI equations in nonlinear Hot control’, Automatica. 2009;45(4):881-888.
37. Ferrari S., Stengel R. ‘An adaptive critic global controller’, Proceedings of the American Control Conference. 2002, 2665-2670.
38. Finlayson B. A. The Method of Weighted Residuals and Variational Principles. New York: Academic Press; 1990.
39. Freeman R. A., Kokotovic P. Robust Nonlinear Control Design: State-Space and Lyapunov Techniques. Boston, MA: Birkhauser; 1996.
40. Gajic Z., Li T. Y. ‘Simulation results for two new algorithms for solving coupled algebraic Riccati equations’, Third International Symposium on Differential Games. 1988., Sophia, Antipolis, France.
41. Ge S. S., Wang C. ‘Adaptive neural control of uncertain MIMO nonlinear systems’, IEEE Transactions Neural Networks. 2004;15(3):674-692.
42. Guo C. H., Lancaster P. ‘Analysis and Modification of Newton’s Method for Algebraic Riccati Equations’, Mathematics of Computation. 1998;67(223):1089-1105.
43. Hanselmann T., Noakes L., Zaknich A. ‘Continuous-time adaptive critics’, IEEE Transactions on Neural Networks. 2007;18(3):631-647.
44. Hasan M. A., Yang J. S., Hasan A. A. ‘A method for solving the algebraic Riccati and Lyapunov equations using higher order matrix sign function algorithms’, Proceedings of the ACC. 1999, 2345-2349.
45. Hewer G. ‘An iterative technique for the computation of the steady state gains for the discrete optimal regulator’, IEEE Transactions on Automatic Control. 1971;16(4):382-384.
46. Hornik K., Stinchcombe M., White H. ‘Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks’, Neural Networks. 1990;3:551-560.
47. Howard R. A. Dynamic Programming and Markov Processes. Cambridge, MA: MIT Press; 1960.
48. Huang J., Lin C. F. ‘Numerical approach to computing nonlinear H? control laws’, Journal of Guidance, Control and Dynamics. 1995;18(5):989-994.
49. Jungers M., De Pieri E., Abou-Kandil H. ‘Solving coupled algebraic Riccati equations from closed-loop Nash strategy, by lack of trust approach’, International Journal of Tomography and Statistics. 2007;7(F07):49-54.
50. Ioannou P., Fidan B. Adaptive Control Tutorial, Philadelphia, PA: SIAM, Advances in Design and Control; 2006.
51. Kailath T. ‘Some new algorithms for recursive estimation in constant linear systems’, IEEE Transactions on Information Theory. 1973;19(6):750-760.
52. Khalil H. K. Nonlinear Systems. New Jersey, Prentice-Hall; 1996.
53. Kirk D. E. Optimal Control Theory-an introduction. Mineola, NY: Dover Pub. Inc.; 2004.
54. Kleinman D. ‘On an iterative technique for Riccati equation computations’, IEEE Transactions on Automatic Control. 1968;13(1):114-115.
55. Kolmogorov A. N., Fomin S. V. Elements of the Theory of Functions and Functional Analysis. Mineola, NY: Dover Pub. Inc.; 1999.
56. Krstic M., Deng H. Stabilization of Nonlinear Uncertain Systems. Springer; 1998.
57. Lancaster P., Rodman L. Algebraic Riccati Equations., UK: Oxford University Press; 1995.
58. Landelius T. Reinforcement Learning and Distributed Local Model Synthesis, PhD Dissertation, Linkoping University, Sweden; 1997.
59. Lanzon A., Feng Y., Anderson B. D. O., Rotkowitz M. ‘Computing the positive stabilizing solution to algebraic Riccati equations with an indefinite quadratic term via a recursive method’, IEEE Transactions on Automatic Control. 2008;53(10):2280-2291.
60. Laub A. J. ‘A schur method for solving algebraic Riccati equations’, IEEE Transactions on Automatic Control. 1979;24(6):913-921.
61. Leake R. J., Liu Ruey-Wen. ‘Construction of suboptimal control sequences’, Journal on SIAM Control. 1967;5(1):54-63.
62. Levine D. S., Brown V. R., Shirey V. T. Oscillations in Neural Systems. Mahwah, NJ: Lawrence Erlbaum Associates Publ; 2000.
63. Lewis F. L., Vrabie D., Syrmos V. L. Optimal Control, 3rd ed. New Jersey: John Wiley; 2012.
64. Lewis F. L., Vrabie D. ‘Reinforcement learning and adaptive dynamic programming for feedback control’, IEEE Circuits & Systems Magazine, Third Quarter. 2009, 32-48.
65. Lewis F. L., Lendaris G., Liu Derong. ‘Special issue on approximate dynamic programming and reinforcement learning for feedback control’, IEEE Transactions on Systems, Man, and Cybernetics, Part B. 2008;38(4).
66. Lewis F. L., Jagannathan S., Yesildirek A. Neural Network Control of Robot Manipulators and Nonlinear Systems. London: Taylor & Francis; 1999.
67. Lewis F. L., Liu. K., Yesildirek A. ‘Neural net controller with guaranteed tracking performance’, IEEE Transactions on Neural Networks. 1995;6(3):703-715.
68. Li Z. H., Krstic M. ‘Optimal design of adaptive tracking controllers for nonlinear systems’, Proceedings of the ACC, 1997, 1191-1197.
69. Limebeer D. J. N., Anderson B. D. O., Hendel H. ‘A Nash game approach to mixed control’, IEEE Transactions on Automatic Control. 1994;39(1):69-82.
70. Liu X., Balakrishnan S. N. ‘Convergence analysis of adaptive critic based optimal control’, Proceedings of the American Control Conference. 2000, 1929-1933.
71. Ljung L. System Identification. New Jersey: Prentice-Hall; 1999.
72. Luenberger D. G. Introduction to Dynamic Systems. New York: John Wiley; 1979.
73. MacFarlane A. G. J. ‘An eigenvector solution of the optimal linear regulator problem’, Journal of Electronics and Control. 1963;14:643-654.
74. Mehta P., Meyn S. ‘Q-learning and Pontryagin’s minimum principle’, Proceedings of the IEEE Conference on Decision and Control. 2009, 3598-3605.
75. Mendel J. M., MacLaren R.W. ‘Reinforcement learning control and pattern recognition systems’, in Mendel J. M., Fu K. S. (eds). Adaptive, Learning, and Pattern Recognition Systems: Theory and Applications. New York: Academic Press; 1970. pp. 287-318.
76. Moris K., Navasca C. ‘Iterative solution of algebraic Riccati equations for damped systems’, Proceedings of the IEEE Conference on Decision and Control. 2006, 2436-2440.
77. Moore K. L. Iterative Learning Control for Deterministic Systems. London: Springer-Verlag; 1993.
78. Murray J. J., Cox C. J., Lendaris G. G., Saeks R. ‘Adaptive dynamic programming’, IEEE Transactions on Systems, Man and Cybernetics. 2002;32(2):140-153.
79. Nevistic V., Primbs J. ‘Constrained nonlinear optimal control: A converse HJB approach’. Technical Report 96-021, California Institute of Technology. 1996.
80. Potter J. E. ‘Matrix quadratic solutions’, SIAM Journal on Applied Mathematics. 1966;14:496-501.
81. Powell W. B. Approximate Dynamic Programming. New Jersey: John Wiley; 2007.
82. Prokhorov D., Wunsch D. ‘Adaptive critic designs’, IEEE Transactions on Neural Networks. 1997;8(5):997-1007.
83. Sandberg I. W. ‘Notes on uniform approximation of time-varying systems on finite time intervals’, IEEE Transactions on Circuits and Systems-1: Fundamental Theory and Applications. 1998;45(8):863-865.
84. Schultz W. ‘Neural coding of basic reward terms of animal learning theory, game theory, microeconomics and behavioral ecology’, Current Opinion in Neurobiology. 2004;14:139-147.
85. Schultz W., Dayan P., Read Montague P. ‘A neural substrate of prediction and reward’, Science. 1997;275:1593-1599.
86. Schultz W., Tremblay L., Hollerman J. R. ‘Reward processing in primate orbitofrontal cortex and basal ganglia’, Cerebral Cortex. 2000;10:272-283.
87. Si J., Barto A., Powell W., Wunsch D. Handbook of Learning and Approximate Dynamic Programming. New Jersey: John Wiley; 2004.
88. Sontag E. D., Sussmann H. J. ‘Nonsmooth control-Lyapunov functions’, Proceedings of the IEEE Conference on Decision and Control. 1995, 2799-2805.
89. Stevens B. L., Lewis F. L. Aircraft Control and Simulation, 2nd ed. New Jersey: John Wiley; 2003.
90. Stoorvogel A. A. The H-infinity Control Problem: A State Space Approach. New York: Prentice-Hall; 1992.
91. Sutton R. ‘Learning to predict by the method of temporal differences’, Machine Learning. 1988;3:9-44.
92. Sutton R. S., Barto A. G. Reinforcement Learning-An Introduction. Cambridge, MT: MIT Press; 1998.
93. Sutton R. S., Barto A. G., Williams R. J. ‘Reinforcement learning is direct adaptive optimal control’, IEEE Control Systems Magazine. 1992, 19-22.
94. Tao G. Adaptive Control Design and Analysis. Adaptive and Learning Systems for Signal Processing, Communications and Control Series. Hoboken, NJ: Wiley-Interscience; 2003.
95. Tsitsiklis J. N. ‘On the convergence of optimistic policy iteration’, Journal of Machine Learning Research. 2002;3:59-72.
96. Vamvoudakis K. G., Lewis F. L. ‘Online solution of nonlinear two-player zero-sum games using synchronous policy iteration’, Proceedings of the IEEE Conference on Decision and Control. Dec. 2010b, 3040-3047, Atlanta.
97. Vamvoudakis K. G., Lewis F. L. ‘Multi-player non-zero sum games: Online adaptive learning solution of coupled Hamilton-Jacobi Equations’, Automatica. 2011;47:1556-1569.
98. Vamvoudakis K. G. Online Learning Algorithms for Differential Dynamic Games and Optimal Control, Ph.D. Thesis, School of Electrical Engineering, The University of Texas at Arlington, May 2011.
99. Vamvoudakis K. G., Lewis F. L. ‘Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem’, Automatica. 2010a;46(5):878-888.
100. Van Der Schaft A. J. ‘L2-gain analysis of nonlinear systems and nonlinear state feedback Hot control’, IEEE Transactions on Automatic Control. 1992;37(6):770-784.
101. Vrabie D., Lewis F. L. ‘Neural network approach to continuous-time direct adaptive optimal control for partially-unknown nonlinear systems’, Neural Networks. 2009;22(3):237-246.
102. Vrabie D., Lewis F. L. ‘Adaptive dynamic programming algorithm for finding online the equilibrium solution of the two-player zero-sum differential game’, Proceedings of the International Joint Conference on Neural Networks. July 2010, 1-8, Barcelona.
103. Vrabie D., Lewis F. L., Abu-Khalaf M. ‘Biologically inspired scheme for continuous-time approximate dynamic programming’, Transactions of the Institute of Measurement and Control. 2008;30:207-223.
104. Vrabie D., Pastravanu O., Abu-Khalaf M., Lewis F. L. ‘Adaptive optimal control for continuous-time linear systems based on policy iteration’, Automatica. 2009;45:477-484.
105. Vrabie D. Online Adaptive Optimal Control for Continuous-Time Systems, Ph.D. Thesis, School of Electrical Engineering, The University of Texas at Arlington, Dec. 2009.
106. Wang F. Y., Zhang H., Liu D. ‘Adaptive dynamic programming: An introduction’, IEEE Computational Intelligence Magazine. 2009, 39-47.
107. Wang Y., Zhou R., Wen C. ‘Robust load-frequency controller design for power systems’, IEE Proceedings C. 1993;140(1):11-16.
108. Watkins C. J. C. H. (1989). Learning from Delayed Rewards, PhD. Thesis, University of Cambridge, England.
109. Watkins C. J. C. H., Dayan P. ‘Q-learning’, Machine Learning. 1992;8:279-292.
110. Werbos P. J. Beyond Regression: New Tools for Prediction and Analysis in the Behavior Sciences, Ph.D. Thesis, 1974.
111. Werbos P. ‘Neural networks for control and system identification’, Proceedings of the IEEE Conference on Decision and Control. 1989, 260-265.
112. Werbos P. J. ‘A menu of designs for reinforcement learning over time’, in Miller W. T., Sutton R. S., Werbos P. J. (eds). Neural Networks for Control, Cambridge: MIT Press; 1991. pp. 67-95.
113. Werbos P. J. ‘Approximate dynamic programming for real-time control and neural modeling’, in White D. A., Sofge D. A. (eds). Handbook of Intelligent Control, New York: Van Nostrand Reinhold; 1992. pp. 493-525.
114. Werbos P. ‘Intelligence in the brain: A theory of how it works and how to build it’, Neural Networks-Special Issue: Goal-Directed Neural Systems. 2009;22(3):200-212.
115. Wei Q., Zhang H. ‘A new approach to solve a class of continuous-time onlinear quadratic zero-sum game using ADP’, Proceedings of the IEEE International Conference on Networking, Sensing and Control (ICNSC’08). 2008;6(8):507-512.
116. Wheeler R. M., Narendra K. S. ‘Decentralized learning in finite Markov Chains’, IEEE Transactions on Automatic Control. 1986;31(6).
117. White D. A., Sofge D. A. Handbook of Intelligent Control. New York: Van Nostrand Reinhold, 1992.
118. Zames G. ‘Feedback and optimal sensitivity: Model reference transformations, multiplicative seminorms and approximate inverses’, IEEE Transactions on Automatic Control. 1981;26:301-320.
119. Zhang H., Huang J., Lewis F. L. ‘Algorithm and stability of ATC receding horizon control’, Proceedings of the IEEE Symposium on ADPRL. 2009, 28-35, Nashville.
120. Zhou K., Khargonekar P. P. ‘An algebraic Riccati equation approach to H-infinity optimization’, Systems and Control Letters. 1988;11:85-91.
121. Zhou K., Doyle J. C., Glover K. Robust and Optimal Control. Upper Saddle River, NJ: Prentice Hall; 1996.