Discrete-time online learning control for a class of unknown nonaffine nonlinear systems using reinforcement learning

Neural Networks

Volumn 55, Issue , 2014, Pages 30-41

  Yang, Xiong ^a   Liu, Derong ^a   Wang, Ding ^a   Wei, Qinglai ^a  

^a INSTITUTE OF AUTOMATION   (China)

Author keywords

Adaptive critic design; Neural network; Nonaffine nonlinear system; Online learning; Reinforcement learning

Indexed keywords

NEURAL NETWORKS; NONLINEAR SYSTEMS; REINFORCEMENT LEARNING; TELECOMMUNICATION NETWORKS;

ADAPTIVE CRITIC DESIGNS; FEEDBACK LINEARIZATION METHODS; HEURISTIC DYNAMIC PROGRAMMING; IMPLICIT FUNCTION THEOREM; MULTI-INPUT MULTI-OUTPUT; NON-AFFINE NONLINEAR SYSTEMS; ONLINE LEARNING; UNIFORMLY ULTIMATELY BOUNDED;

E-LEARNING;

ARTICLE; ARTIFICIAL NEURAL NETWORK; CALCULATION; COMPUTER PREDICTION; CONTROLLED STUDY; INFORMATION PROCESSING; INFORMATION RETRIEVAL; MATHEMATICAL ANALYSIS; NONLINEAR SYSTEM; ONLINE SYSTEM; PRIORITY JOURNAL; REINFORCEMENT; SIMULATION; ALGORITHM; ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; EDUCATIONAL MODEL; FEEDBACK SYSTEM; PROCEDURES; TEACHING;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; COMPUTER-ASSISTED INSTRUCTION; FEEDBACK; MODELS, EDUCATIONAL; NEURAL NETWORKS (COMPUTER); NONLINEAR DYNAMICS; REINFORCEMENT (PSYCHOLOGY);

EID: 84897950099     PISSN: 08936080     EISSN: 18792782     Source Type: Journal    
DOI: 10.1016/j.neunet.2014.03.008     Document Type: Article

References (52)

1. Al-Tamimi A., Lewis F.L., Abu-Khalaf M. Discrete-time nonlinear HJB solution using approximate dynamic programming: convergence proof. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 2008, 38(4):943-949.
2. Apostol T.M. Mathematical analysis 1974, Addison-Wesley, Cambridge, MA. 2nd ed.
3. Bellman R.E. Dynamic programming 1957, Princeton University Press, Princeton, New Jersey.
4. Bertsekas D.P., Tsitsiklis J.N. Neuro-dynamic programming 1996, Athena Scientific, Cambridge, MA.
5. Bhasin S., Kamalapurkar R., Johnson M., Vamvoudakis K.G., Lewis F.L., Dixon W.E. A novel actor-critic-identifier architecture for approximate optimal control of uncertain nonlinear systems. Automatica 2013, 49(1):82-92.
6. Chemachema M. Output feedback direct adaptive neural network control for uncertain SISO nonlinear systems using a fuzzy estimator of the control error. Neural Networks 2012, 36:25-34.
7. Chen F.C., Khalil H.K. Adaptive control of a class of nonlinear discrete-time systems using neural networks. IEEE Transactions on Automatic Control 1995, 40(5):791-801.
8. Deng H., Li H.X., Wu Y.H. Feedback-linearization-based neural adaptive control for unknown nonaffine nonlinear discrete-time systems. IEEE Transactions on Neural Networks 2008, 19(9):1615-1625.
9. Ge S.S., Hang C.C., Zhang T. Adaptive neural network control of nonlinear systems by state and output feedback. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 1999, 29(6):818-828.
10. Haykin S. Neural networks and learning machines 2008, Prentice Hall, New Jersey. 3rd ed.
11. He P., Jagannathan S. Reinforcement learning-based output feedback control of nonlinear systems with input constraints. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 2005, 35(1):150-154.
12. Horn R.A., Johnson C.R. Matrix analysis 1985, Cambridge University Press, New York.
13. Hovakimyan N., Nardi F., Calise A., Kim N. Adaptive output feedback control of uncertain nonlinear systems using single-hidden-layer neural networks. IEEE Transactions on Neural Networks 2002, 13(6):1420-1431.
14. Igelnik B., Pao Y.H. Stochastic choice of basis functions in adaptive function approximation and the function-link net. IEEE Transactions on Neural Networks 1995, 6(6):1320-1329.
15. Lewis F.L., Jagannathan S., Yesildirek A. Neural network control of robot manipulators and nonlinear systems 1999, Taylor & Francis, London.
16. Lewis F.L., Lendaris G., Liu D. Special issue on approximate dynamic programming and reinforcement learning for feedback control. IEEE Transactions on Systems, Man, Cybernetics, Part B, Cybernetics 2008, 38(4):896-897.
17. Lewis F.L., Vamvoudakis K.G. Reinforcement learning for partially observable dynamic processes: adaptive dynamic programming using measured output data. IEEE Transactions on Systems, Man, Cybernetics, Part B, Cybernetics 2011, 41(1):14-25.
18. Lewis F.L., Vrabie D., Vamvoudakis K.G. Reinforcement learning and feedback control: using natural decision methods to design optimal adaptive controllers. IEEE Control Systems 2012, 32(6):76-105.
19. Lewis F.L., Yesildirek A., Liu K. Multilayer neural-net robot controller with guaranteed tracking performance. IEEE Transactions on Neural Networks 1996, 7(2):388-399.
20. Liu W., Venayagamoorthy G.K., Wunsch D.C. Design of an adaptive neural network based power system stabilizer. Neural Networks 2003, 16(5-6):891-898.
21. Liu D., Wang D., Yang X. An iterative adaptive dynamic programming algorithm for optimal control of unknown discrete-time nonlinear systems with constrained inputs. Information Sciences 2013, 220(20):331-342.
22. Liu D., Wei Q. Finite-approximation-error-based optimal control approach for discrete-time nonlinear systems. IEEE Transactions on Cybernetics 2013, 43(2):779-789.
23. Liu D., Yang X., Li H. Adaptive optimal control for a class of continuous-time affine nonlinear systems with unknown internal dynamics. Neural Computing and Applications 2013, 23(7-8):1843-1850.
24. Liu D., Zhang Y., Zhang H. A self-learning call admission control scheme for CDMA cellular networks. IEEE Transactions on Neural Networks 2005, 16(5):1219-1228.
25. Murray J.J., Cox C.J., Lendaris G.G., Saeks R. Adaptive dynamic programming. IEEE Transactions on Systems, Man and Cybernetics, Part C: Applications and Reviews 2002, 32(2):140-153.
26. Nakanishi J., Schaal S. Feedback error learning and nonlinear adaptive control. Neural Networks 2004, 17(10):1453-1465.
27. Narendra K.S., Mukhopadhyay S. Adaptive control of nonlinear multivariable systems using neural networks. Neural Networks 1994, 7(5):737-752.
28. Noriega J.R., Wang H. A direct adaptive neural-network control for unknown nonlinear systems and its applications. IEEE Transactions on Neural Networks 1998, 9(1):27-34.
29. Padhi R., Unnikrishnan N., Wang X., Balakrishnan S.N. A single network adaptive critic (SNAC) architecture for optimal control synthesis for a class of nonlinear systems. Neural Networks 2006, 19(10):1648-1660.
30. Park J.H., Huh S.H., Kim S.H., Seo S.J., Park G.T. Direct adaptive controller for nonaffine nonlinear systems using self-structuring neural networks. IEEE Transactions on Neural Networks 2005, 16(2):414-422.
31. Prokhorov D.V., Wunsch D.C. Adaptive critic designs. IEEE Transactions on Neural Networks 1997, 8(5):997-1007.
32. Rudin W. Functional analysis 1991, McGraw-Hill, Inc., Singapore. 2nd ed.
33. Si J., Wang Y.T. On-line learning control by association and reinforcement. IEEE Transactions on Neural Networks 2001, 12(2):264-276.
34. Sutton R.S., Barto A.G. Reinforcement learning-an introduction 1998, MIT Press, Cambridge, MA.
35. Vamvoudakis K.G., Lewis F.L. Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem. Automatica 2010, 46(5):878-888.
36. Vamvoudakis K.G., Lewis F.L. Multi-player non-zero-sum games: online adaptive learning solution of coupled Hamilton-Jacobi equations. Automatica 2011, 47(8):1556-1569.
37. Wang D., Liu D., Wei Q. Finite-horizon neuro-optimal tracking control for a class of discrete-time nonlinear systems using adaptive dynamic programming approach. Neurocomputing 2012, 78(1):14-22.
38. Wang D., Liu D., Wei Q., Zhao D., Jin N. Optimal control of unknown nonaffine nonlinear discrete-time systems based on adaptive dynamic programming. Automatica 2012, 48(8):1825-1832.
39. Wang F.Y., Zhang H., Liu D. Adaptive dynamic programming: an introduction. IEEE Computational Intelligence Magazine 2009, 4(2):39-47.
40. Wei Q., Liu D. An iterative ε-optimal control scheme for a class of discrete-time nonlinear systems with unfixed initial state. Neural Networks 2012, 32:236-244.
41. Werbos P.J. A menu of designs for reinforcement learning over time. Neural networks for control 1991, 67-95. MIT Press, Cambridge, MA. W.T. Miller, R.S. Sutton, P.J. Werbos (Eds.).
42. Werbos P.J. Approximate dynamic programming for real-time control and neural modeling. Handbook of intelligent control 1992, 493-525. Van Nostrand Reinhold, New York. D.A. White, D.A. Sofge (Eds.).
43. Werbos, P.J. (2007). Using ADP to understand and replicate brain intelligence: the next level design. In Proceedings of the IEEE symposium on approximate dynamic programming and reinforcement learning (pp. 209-216). Honolulu, HI, April.
44. Werbos P.J. ADP: the key direction for future research in intelligent control and understanding brain intelligence. IEEE Transactions on Systems, Man, Cybernetics, Part B, Cybernetics 2008, 38(4):898-900.
45. Yang Q., Jagannathan S. Reinforcement learning controller design for affine nonlinear discrete-time systems using approximators. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 2012, 42(2):377-390.
46. Yang X., Liu D., Huang Y. Neural-network-based online optimal control for uncertain non-linear continuous-time systems with control constraints. IET Control Theory & Applications 2013, 7(17):2037-2047.
47. Yang L., Si J., Tsakalis K.S., Rodriguez A.A. Direct heuristic dynamic programming for nonlinear tracking control with filtered tracking error. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 2009, 19(6):1617-1622.
48. Yang Q., Vance J.B., Jagannathan S. Control of nonaffine nonlinear discrete-time systems using reiforcement-learning-based linearly parameterized neural networks. IEEE Transactions on Systems, Man, Cybernetics, Part B, Cybernetics 2008, 38(4):994-1001.
49. Yu W. Recent advances in intelligent control systems 2009, Springer-Verlag, London.
50. Zeidler E. Nonlinear functional analysis and its applications: part 1: fixed-point theorems 1985, Springer-Verlag, New York.
51. Zhang, J., Ge, S.S., & Lee, T.H. (2002). Direct RBF neural network control of a class of discrete-time nonaffine nonlinear systems. In Proceeding of the American control conference (pp. 424-429). Anchorage, AK, May.
52. Zhang H., Wei Q., Liu D. An iterative adaptive dynamic programming method for solving a class of nonlinear zero-sum differential games. Automatica 2011, 47(1):207-214.