Off-policy reinforcement learning for H∞ control design

IEEE Transactions on Cybernetics

Volumn 45, Issue 1, 2015, Pages 65-76

  Luo, Biao ^a,b   Wu, Huai Ning ^a   Huang, Tingwen ^c  

^a BEIHANG UNIVERSITY   (China)

^b INSTITUTE OF AUTOMATION   (China)

^c TEXAS A AND M UNIVERSITY AT QATAR   (Qatar)

Author keywords

H control design; Hamilton Jacobi Isaacs equation; Neural network; Off policy learning; Reinforcement learning

Indexed keywords

AIRCRAFT CONTROL; DESIGN; LEAST SQUARES APPROXIMATIONS; NEURAL NETWORKS; NONLINEAR EQUATIONS; PARTIAL DIFFERENTIAL EQUATIONS;

CONTROL DESIGN; HAMILTON-JACOBI-ISAACS; HAMILTON-JACOBI-ISAACS EQUATIONS; MATHEMATICAL SYSTEM MODEL; METHOD OF WEIGHTED RESIDUAL; NONLINEAR PARTIAL DIFFERENTIAL EQUATIONS; POLICY LEARNING; ROTATIONAL/TRANSLATIONAL ACTUATOR;

REINFORCEMENT LEARNING;

EID: 84919730591     PISSN: 21682267     EISSN: None     Source Type: Journal    
DOI: 10.1109/TCYB.2014.2319577     Document Type: Article

References (64)

1. L. P. Kaelbling, M. L. Littman, and A. W. Moore, "Reinforcement learning: A survey," J. Artif. Intell. Res., vol. 4, no. 1, pp. 237-285, Jan. 1996.
2. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, U.K.: Cambridge Univ. Press, 1998.
3. D. P. Bertsekas, Dynamic Programming and Optimal Control. Nashua, NH, USA: Athena Scientific, 2005.
4. W. B. Powell, Approximate Dynamic Programming: Solving the Curses of Dimensionality. Hoboken, NJ, USA: Wiley, 2007.
5. J. J. Murray, C. J. Cox, G. G. Lendaris, and R. Saeks, "Adaptive dynamic programming," IEEE Trans. Syst., Man, Cybern. C, Appl. Rev., vol. 32, no. 2, pp. 140-153, May 2002.
6. V. Yadav, R. Padhi, and S. Balakrishnan, "Robust/optimal temperature profile control of a high-speed aerospace vehicle using neural networks," IEEE Trans. Neural Netw., vol. 18, no. 4, pp. 1115-1128, Jul. 2007.
7. H. Zhang, Q. Wei, and Y. Luo, "A novel infinite-time optimal tracking control scheme for a class of discrete-time nonlinear systems via the greedy HDP iteration algorithm," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 937-942, Aug. 2008.
8. D. Vrabie, O. Pastravanu, M. Abu-Khalaf, and F. L. Lewis, "Adaptive optimal control for continuous-time linear systems based on policy iteration," Automatica, vol. 45, no. 2, pp. 477-484, 2009.
9. D. Vrabie and F. L. Lewis, "Neural network approach to continuoustime direct adaptive optimal control for partially unknown nonlinear systems," Neural Netw., vol. 22, no. 3, pp. 237-246, 2009.
10. H. Zhang, Y. Luo, and D. Liu, "Neural-network-based near-optimal control for a class of discrete-time affine nonlinear systems with control constraints," IEEE Trans. Neural Netw., vol. 20, no. 9, pp. 1490-1503, Sep. 2009.
11. H. Zhang, L. Cui, X. Zhang, and Y. Luo, "Data-driven robust approximate optimal tracking control for unknown general nonlinear systems using adaptive dynamic programming method," IEEE Trans. Neural Netw., vol. 22, no. 12, pp. 2226-2236, Dec. 2011.
12. B. Luo and H.-N. Wu, "Online policy iteration algorithm for optimal control of linear hyperbolic PDE systems," J. Process Control, vol. 22, no. 7, pp. 1161-1170, 2012.
13. D. Vrabie, F. L. Lewis, and K. G. Vamvoudakis, Optimal Adaptive Control and Differential Games by Reinforcement Learning Principles. Stevenage, England: IET Press, 2012.
14. D. Liu, D. Wang, D. Zhao, Q. Wei, and N. Jin, "Neural-network-based optimal control for a class of unknown discrete-time nonlinear systems using globalized dual heuristic programming," IEEE Trans. Autom. Sci. Eng., vol. 9, no. 3, pp. 628-634, Jul. 2012.
15. B. Luo and H.-N. Wu, "Approximate optimal control design for nonlinear one-dimensional parabolic PDE systems using empirical eigenfunctions and neural network," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 42, no. 6, pp. 1538-1549, Dec. 2012.
16. H.-N. Wu and B. Luo, "Heuristic dynamic programming algorithm for optimal control design of linear continuous-time hyperbolic PDE systems," Ind. Eng. Chem. Res., vol. 51, no. 27, pp. 9310-9319, 2012.
17. D. Liu and Q. Wei, "Finite-approximation-error-based optimal control approach for discrete-time nonlinear systems," IEEE Trans. Cybern., vol. 43, no. 2, pp. 779-789, Apr. 2013.
18. Q. Wei and D. Liu, "A novel iterative θ-adaptive dynamic programming for discrete-time nonlinear systems," IEEE Trans. Autom. Sci. Eng., 2013, to be published.
19. D. Liu, D. Wang, and H. Li, "Decentralized stabilization for a class of continuous-time nonlinear interconnected systems using online learning optimal control approach," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 2, pp. 418-428, Feb. 2014.
20. B. Luo, H.-N. Wu, T. Huang, and D. Liu, "Data-based approximate policy iteration for nonlinear continuous-time optimal control design," arXiv preprint arXiv:1311.0396, 2013 [Online]. Available: http://arxiv.org/abs/1311.0396
21. H. Modares, F. L. Lewis, and M.-B. Naghibi-Sistani, "Integral reinforcement learning and experience replay for adaptive optimal control of partially-unknown constrained-input continuous-time systems," Automatica, vol. 50, no. 1, pp. 193-202, 2014.
22. D. Liu and Q. Wei, "Policy iteration adaptive dynamic programming algorithm for discrete-time nonlinear systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 3, pp. 621-634, Mar. 2014.
23. B. Luo, H.-N. Wu, and H.-X. Li, "Data-based suboptimal neuro-control design with reinforcement learning for dissipative spatially distributed processes," Ind. Eng. Chem. Res., 2014, to be published.
24. K. Zhou, J. C. Doyle, and K. Glover, Robust and Optimal Control. Upper Saddle River, NJ, USA: Prentice Hall, 1996.
25. 2-Gain and Passivity in Nonlinear Control. New York, NY, USA: Springer-Verlag, 1996.
26. ∞ Optimal Control and Related Minimax Design Problems: A Dynamic Game Approach. Berlin, Germany: Springer, 2008.
27. ∞ control," IEEE Trans. Autom. Control, vol. 37, no. 6, pp. 770-784, Jun. 1992.
28. ∞ control via measurement feedback for general nonlinear systems," IEEE Trans. Autom. Control, vol. 40, no. 3, pp. 466-472, Mar. 1995.
29. ∞-control via measurement feedback in nonlinear systems," IEEE Trans. Autom. Control, vol. 37, no. 9, pp. 1283-1293, Sep. 1992.
30. R. W. Beard, "Successive Galerkin approximation algorithms for nonlinear optimal and robust control," Int. J. Control, vol. 71, no. 5, pp. 717-743, 1998.
31. ∞ state feedback control with input saturation," IEEE Trans. Autom. Control, vol. 51, no. 12, pp. 1989-1995, Dec. 2006.
32. K. G. Vamvoudakis and F. L. Lewis, "Online solution of nonlinear two-player zero-sum games using synchronous policy iteration," Int. J. Robust Nonlinear Control, vol. 22, no. 13, pp. 1460-1483, 2012.
33. ∞ control," Automatica, vol. 45, no. 4, pp. 881-888, 2009.
34. D. Liu, H. Li, and D. Wang, "Neural-network-based zero-sum game for discrete-time nonlinear systems via iterative adaptive dynamic programming algorithm," Neurocomputing, vol. 110, no. 13, pp. 92-100, 2013.
35. N. Sakamoto and A. V. D. Schaft, "Analytical approximation methods for the stabilizing solution of the Hamilton-Jacobi equation," IEEE Trans. Autom. Control, vol. 53, no. 10, pp. 2335-2350, Nov. 2008.
36. G. N. Saridis and C.-S. G. Lee, "An approximation theory of optimal control for trainable manipulators," IEEE Trans. Syst., Man, Cybern., vol. 9, no. 3, pp. 152-159, Mar. 1979.
37. R. W. Beard, G. N. Saridis, and J. T. Wen, "Galerkin approximations of the generalized Hamilton-Jacobi-Bellman equation," Automatica, vol. 33, no. 12, pp. 2159-2177, 1997.
38. R. W. Beard, G. Saridis, and J. Wen, "Approximate solutions to the time-invariant Hamilton-Jacobi-Bellman equation," J. Optim. Theory Appl., vol. 96, no. 3, pp. 589-626, 1998.
39. S. Mehraeen, T. Dierks, S. Jagannathan, and M. L. Crow, "Zero-sum two-player game theoretic formulation of affine nonlinear discrete-time systems using neural networks," IEEE Trans. Cybern., vol. 43, no. 6, pp. 1641-1655, Dec. 2013.
40. M. Abu-Khalaf, F. L. Lewis, and J. Huang, "Neurodynamic programming and zero-sum games for constrained control systems," IEEE Trans. Neural Netw., vol. 19, no. 7, pp. 1243-1252, Jul. 2008.
41. ∞ control of constrained input systems," Int. J. Adapt. Control, vol. 28, nos. 3-5, pp. 232-254, Mar.-May 2014.
42. H. Zhang, Q. Wei, and D. Liu, "An iterative adaptive dynamic programming method for solving a class of nonlinear zero-sum differential games," Automatica, vol. 47, no. 1, pp. 207-214, 2011.
43. K. G. Vamvoudakis and F. L. Lewis, "Online actor - Critic algorithm to solve the continuous-time infinite horizon optimal control problem," Automatica, vol. 46, no. 5, pp. 878-888, 2010.
44. ∞ state feedback control with Galerkin's method," Int. J. Robust Nonlinear Control, vol. 23, no. 9, pp. 991-1012, 2013.
45. ∞ control laws," J. Guid. Control Dynam., vol. 18, no. 5, pp. 989-994, 1995.
46. ∞ state feedback control," Inf. Sci., vol. 222, pp. 472-485, Feb. 2013.
47. D. Vrabie and F. Lewis, "Adaptive dynamic programming for online solution of a zero-sum differential game," J. Control Theory Appl., vol. 9, no. 3, pp. 353-360, 2011.
48. ∞ control," IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 12, pp. 1884-1895, Dec. 2012.
49. H. Zhang, L. Cui, and Y. Luo, "Near-optimal control for nonzero-sum differential games of continuous-time nonlinear systems using singlenetwork ADP," IEEE Trans. Cybern., vol. 43, no. 1, pp. 206-216, Feb. 2013.
50. S. B. Thrun, "Efficient exploration in reinforcement learning," Carnegie Mellon University, Pittsburgh, PA, USA, Tech. Rep. CMU-CS-92-102, 1992.
51. R. Courant and D. Hilbert, Methods of Mathematical Physics. New York, NY, USA: Wiley, 2004.
52. B. A. Finlayson, The Method of Weighted Residuals and Variational Principles: With Applications in Fluid Mechanics, Heat and Mass Transfer. New York, NY, USA: Academic Press, Inc., 1972.
53. G. Peter Lepage, "A new algorithm for adaptive multidimensional integration," J. Comput. Phys., vol. 27, no. 2, pp. 192-203, 1978.
54. J. A. Farrell and M. M. Polycarpou, Adaptive Approximation Based Control: Unifying Neural, Fuzzy and Traditional Adaptive Approximation Approaches. New York, NY, USA: Wiley, 2006.
55. J.-J. E. Slotine et al., Applied Nonlinear Control. Englewood Cliffs, NJ, USA: Prentice-Hall 1991.
56. D. Kleinman, "On an iterative technique for Riccati equation computations," IEEE Trans. Autom. Control, vol. 13, no. 1, pp. 114-115, Feb. 1968.
57. M. Abu-Khalaf and F. L. Lewis, "Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach," Automatica, vol. 41, no. 5, pp. 779-791, 2005.
58. Z. Chen and S. Jagannathan, "Generalized Hamilton-Jacobi-Bellman formulation-based neural network control of affine nonlinear discretetime systems," IEEE Trans. Neural Netw., vol. 19, no. 1, pp. 90-106, Jan. 2008.
59. K. G. Vamvoudakis and F. L. Lewis, "Multi-player non-zero-sum games: Online adaptive learning solution of coupled Hamilton-Jacobi equations," Automatica, vol. 47, no. 8, pp. 1556-1569, 2011.
60. D. Precup, R. S. Sutton, and S. Dasgupta, "Off-policy temporal-difference learning with function approximation," in Proc. 18th ICML, 2001, pp. 417-424.
61. H. R. Maei, C. Szepesvári, S. Bhatnagar, and R. S. Sutton, "Toward off-policy learning control with function approximation," in Proc. 27th ICML, 2010, pp. 719-726.
62. M. Green and D. J. Limebeer, Linear Robust Control. Englewood Cliffs, NJ, USA: Prentice-Hall, 1995.
63. B. L. Stevens and F. L. Lewis, Aircraft Control and Simulation. Hoboken, NJ, USA: Wiley-Interscience, 2003.
64. 2 disturbance attenuation solution to the nonlinear benchmark problem," Int. J. Robust Nonlinear Control, vol. 8, nos. 4-5, pp. 311-330, 1999.