Reinforcement learning solution for HJB equation arising in constrained optimal control problem

Neural Networks

Volumn 71, Issue , 2015, Pages 150-158

  Luo, Biao ^a   Wu, Huai Ning ^b   Huang, Tingwen ^c   Liu, Derong ^d  

^a INSTITUTE OF GEOLOGY AND GEOPHYSICS   (China)

^b BEIHANG UNIVERSITY   (China)

^c TEXAS A AND M UNIVERSITY AT QATAR   (Qatar)

^d UNIVERSITY OF SCIENCE AND TECHNOLOGY BEIJING   (China)

Author keywords

Constrained optimal control; Data based; Hamilton Jacobi Bellman equation; Off policy reinforcement learning; The method of weighted residuals

Indexed keywords

CONTROL THEORY; DYNAMIC PROGRAMMING; OPTIMAL CONTROL SYSTEMS; TELECOMMUNICATION NETWORKS;

CONSTRAINED OPTIMAL CONTROL; CONSTRAINED OPTIMAL CONTROL PROBLEMS; DATA-BASED; HAMILTON JACOBI BELLMAN EQUATION; METHOD OF WEIGHTED RESIDUAL; REINFORCEMENT LEARNING METHOD; REINFORCEMENT LEARNING SOLUTION; SUCCESSIVE APPROXIMATION APPROACH;

REINFORCEMENT LEARNING;

ARTICLE; CALCULATION; HAMILTON JACOBI BELLAMAN EQUATION; LEARNING; MATHEMATICAL ANALYSIS; MATHEMATICAL MODEL; MONTE CARLO METHOD; POLICY; PRIORITY JOURNAL; REINFORCEMENT; SIMULATION; ALGORITHM; ARTIFICIAL NEURAL NETWORK; COMPUTER SIMULATION; MACHINE LEARNING; NONLINEAR SYSTEM; PROBLEM SOLVING; THEORETICAL MODEL;

ALGORITHMS; COMPUTER SIMULATION; MACHINE LEARNING; MODELS, THEORETICAL; NEURAL NETWORKS (COMPUTER); NONLINEAR DYNAMICS; PROBLEM SOLVING; REINFORCEMENT (PSYCHOLOGY);

EID: 84941097144     PISSN: 08936080     EISSN: 18792782     Source Type: Journal    
DOI: 10.1016/j.neunet.2015.08.007     Document Type: Article

References (43)

1. Abu-Khalaf M., Lewis F.L. Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach. Automatica 2005, 41(5):779-791.
2. Abu-Khalaf M., Lewis F.L., Huang J. Neurodynamic programming and zero-sum games for constrained control systems. IEEE Transactions on Neural Networks 2008, 19(7):1243-1252.
3. Anderson B.D., Moore J.B. Optimal control: linear quadratic methods 2007, Dover Publications, Mineola, NY.
4. Beard R.W., Saridis G.N., Wen J.T. Galerkin approximations of the generalized Hamilton-Jacobi-Bellman equation. Automatica 1997, 33(12):2159-2177.
5. Bertsekas D.P. Dynamic programming and optimal control. Vol. 1 2005, Athena Scientific, Nashua.
6. Chen Z., Jagannathan S. Generalized Hamilton-Jacobi-Bellman formulation-based neural network control of affine nonlinear discrete-time systems. IEEE Transactions on Neural Networks 2008, 19(1):90-106.
7. Doya K. Reinforcement learning in continuous time and space. Neural Computation 2000, 12(1):219-245.
8. Faust A., Ruymgaart P., Salman M., Fierro R., Tapia L. Continuous action reinforcement learning for control-affine systems with unknown dynamics. IEEE/CAA Journal of Automatica Sinica 2014, 1(3):323-336.
9. He P., Jagannathan S. Reinforcement learning-based output feedback control of nonlinear systems with input constraints. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 2005, 35(1):150-154.
10. He P., Jagannathan S. Reinforcement learning neural-network-based controller for nonlinear discrete-time systems with input constraints. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 2007, 37(2):425-436.
11. Heydari A., Balakrishnan S.N. Finite-horizon control-constrained nonlinear optimal control using single network adaptive critics. IEEE Transactions on Neural Networks and Learning Systems 2013, 24(1):147-157.
12. Hull D.G. Optimal control theory for applications 2003, Springer, Troy, NY.
13. Jiang Y., Jiang Z.-P. Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics. Automatica 2012, 48(10):2699-2704.
14. Jiang Y., Jiang Z.-P. Robust adaptive dynamic programming and feedback stabilization of nonlinear systems. IEEE Transactions on Neural Networks and Learning Systems 2014, 25(5):882-893.
15. Lee J.Y., Park J.B., Choi Y.H. Integral Q-learning and explorized policy iteration for adaptive optimal control of continuous-time linear systems. Automatica 2012, 48(11):2850-2859.
16. Lendaris G.G. Adaptive dynamic programming approach to experience-based systems identification and control. Neural Networks 2009, 22(5-6):822-832.
17. Lewis F.L., Vrabie D., Syrmos V.L. Optimal control 2013, John Wiley & Sons, Hoboken. NJ.
18. Liu D., Wang D., Li H. Decentralized stabilization for a class of continuous-time nonlinear interconnected systems using online learning optimal control approach. IEEE Transactions on Neural Networks and Learning Systems 2014, 25(2):418-428.
19. Liu D., Wang D., Yang X. An iterative adaptive dynamic programming algorithm for optimal control of unknown discrete-time nonlinear systems with constrained inputs. Information Sciences 2013, 220:331-342.
20. Liu D., Wei Q. Policy iteration adaptive dynamic programming algorithm for discrete-time nonlinear systems. IEEE Transactions on Neural Networks and Learning Systems 2014, 25(3):621-634.
21. ∞ control design. IEEE Transactions on Cybernetics 2015, 45(1):65-76.
22. Luo B., Wu H.-N., Huang T., Liu D. Data-based approximate policy iteration for affine nonlinear continuous-time optimal control design. Automatica 2014, 50(12):3281-3290.
23. Lyashevskiy, S. (1996). Constrained optimization and control of nonlinear systems: new results in optimal control. In Proceedings of the 35th IEEE decision and control (pp. 541-546).
24. Lyshevski S.E. Optimal control of nonlinear continuous-time systems: design of bounded controllers via generalized nonquadratic functionals. Proceedings of the 1998 American control conference. Vol. 1 1998, 205-209. IEEE.
25. Maei, H. R., Szepesvári, C., Bhatnagar, S., & Sutton, R. S. (2010). Toward off-policy learning control with function approximation. In Proceedings of the 27th international conference on machine learning (pp. 719-726).
26. Modares H., Lewis F.L. Linear quadratic tracking control of partially-unknown continuous-time systems using reinforcement learning. IEEE Transactions on Automatic Control 2014, 59(11):3051-3056.
27. Modares H., Lewis F.L., Naghibi-Sistani M.-B. Adaptive optimal control of unknown constrained-input systems using policy iteration and neural networks. IEEE Transactions on Neural Networks and Learning Systems 2013, 24(10):1513-1525.
28. Modares H., Lewis F.L., Naghibi-Sistani M.-B. Integral reinforcement learning and experience replay for adaptive optimal control of partially-unknown constrained-input continuous-time systems. Automatica 2014, 50(1):193-202.
29. Murray J.J., Cox C.J., Lendaris G.G., Saeks R. Adaptive dynamic programming. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews 2002, 32(2):140-153.
30. Peter Lepage G. A new algorithm for adaptive multidimensional integration. Journal of Computational Physics 1978, 27(2):192-203.
31. Powell W.B. Approximate dynamic programming: solving the curses of dimensionality 2007, John Wiley & Sons, Hoboken, NJ.
32. Precup, D., Sutton, R. S., & Dasgupta, S. (2001). Off-policy temporal-difference learning with function approximation. In Proceedings of the 18th international conference on machine learning (pp. 417-424).
33. Saridis G.N., Lee C.-S.G. An approximation theory of optimal control for trainable manipulators. IEEE Transactions on Systems, Man and Cybernetics 1979, 9(3):152-159.
34. Sutton R.S., Barto A.G. Reinforcement learning: an introduction 1998, The MIT Press, Cambridge.
35. Vamvoudakis K.G., Lewis F.L. Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem. Automatica 2010, 46(5):878-888.
36. Vrabie D., Lewis F.L. Neural network approach to continuous-time direct adaptive optimal control for partially unknown nonlinear systems. Neural Networks 2009, 22(3):237-246.
37. Vrabie D., Pastravanu O., Abu-Khalaf M., Lewis F.L. Adaptive optimal control for continuous-time linear systems based on policy iteration. Automatica 2009, 45(2):477-484.
38. Wang D., Liu D., Li H. Policy iteration algorithm for online design of robust control for a class of continuous-time nonlinear systems. IEEE Transactions on Automation Science and Engineering 2014, 11(2):627-632.
39. Wei Q., Liu D. An iterative ε-optimal control scheme for a class of discrete-time nonlinear systems with unfixed initial state. Neural Networks 2012, 32:236-244.
40. Yang X., Liu D., Wang D. Reinforcement learning for adaptive optimal control of unknown continuous-time nonlinear systems with input constraints. International Journal of Control 2014, 87(3):553-566.
41. Yang X., Liu D., Wang D., Wei Q. Discrete-time online learning control for a class of unknown nonaffine nonlinear systems using reinforcement learning. Neural Networks 2014, 55:30-41.
42. Zhang H., Luo Y., Liu D. Neural-network-based near-optimal control for a class of discrete-time affine nonlinear systems with control constraints. IEEE Transactions on Neural Networks 2009, 20(9):1490-1503.
43. Zhao Q., Xu H., Jagannathan S. Near optimal output feedback control of nonlinear discrete-time systems based on reinforcement neural network learning. IEEE/CAA Journal of Automatica Sinica 2014, 1(4):372-384.