Reinforcement learning controller design for affine nonlinear discrete-time systems using online approximators

IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics

Volumn 42, Issue 2, 2012, Pages 377-390

  Yang, Qinmin ^a   Jagannathan, Sarangapani ^b  

^a STATE KEY LABORATORY OF INDUSTRIAL CONTROL TECHNOLOGY   (China)

^b Architectural and Environmental Engineering   (United States)

Author keywords

Adaptive critic; dynamic programming (DP); Lyapunov method; neural networks (NNs); online approximators (OLAs); online learning; reinforcement learning

Indexed keywords

ACTION NETWORK; ADAPTIVE CRITIC; APPROXIMATORS; BALANCING SYSTEM; BOUNDED DISTURBANCES; CONTROLLER DESIGNS; CRITIC NETWORK; HEURISTIC DYNAMIC PROGRAMMING; LEARNING CONTROLLERS; LYAPUNOV; LYAPUNOV THEORIES; MULTI-OUTPUT; MULTIINPUT; NONLINEAR DISCRETE-TIME SYSTEMS; ONLINE LEARNING; OPTIMAL SIGNALS; OUTPUT-FEEDBACK; RADIAL BASIS FUNCTIONS; RECURSIVE EQUATIONS; SEPARATION PRINCIPLE; SYSTEM STATE; TWO-LINK; UNIFORM ULTIMATE BOUNDEDNESS;

DIGITAL CONTROL SYSTEMS; DISCRETE TIME CONTROL SYSTEMS; DYNAMIC PROGRAMMING; FUZZY LOGIC; LYAPUNOV METHODS; NEURAL NETWORKS; NONLINEAR FEEDBACK; ONLINE SYSTEMS; RADIAL BASIS FUNCTION NETWORKS; REINFORCEMENT LEARNING; STATE FEEDBACK;

SEARCH ENGINES;

EID: 84859001250     PISSN: 10834419     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSMCB.2011.2166384     Document Type: Article

References (31)

1. R. Bellman and S. Dreyfus, Applied Dynamic Programming. Princeton, NJ: Princeton Univ. Press, 1962.
2. Z. V. Rekasius, "Suboptimal design of intentionally nonlinear controllers," IEEE Trans. Autom. Control, vol. AC-9, no. 4, pp. 380-386, Oct. 1964.
3. R. J. Leake and R. Liu, "Construction of suboptimal control sequences," SIAM J. Control Optim., vol. 5, no. 1, pp. 54-63, 1967.
4. D. Kirk, Optimal Control Theory: An Introduction. Englewood Cliffs, NJ: Prentice-Hall, 1970.
5. F. L. Lewis, S. Jagannathan, and A. Yesildirek, Neural Network Control of Robot Manipulators and Nonlinear Systems. Philadelphia, PA: Taylor & Francis, 1999.
6. J. Si and Y. T. Wang, "On-line learning control by association and reinforcement," IEEE Trans. Neural Netw., vol. 12, no. 2, pp. 264-276, Mar. 2001.
7. J. C. Santamaria, R. S. Sutton, and A. Ram, "Experiments with reinforcement learning in problems with continuous state and action spaces," Univ. Massachusetts, Amherst, MA, COINS Tech. Rep. 96-88, Dec. 1996.
8. R. Luus, Iterative Dynamic Programming. Boca Raton, FL: CRC Press, 2000.
9. A. G. Barto, R. S. Sutton, and C. W. Anderson, "Neuron like adaptive elements that can solve difficult learning control problems," IEEE Trans. Syst., Man, Cybern., vol. SMC-13, no. 5, pp. 834-847, Sep./Oct. 1983.
10. R. S. Sutton, "Learning to predict by the methods of temporal difference," Mach. Learn., vol. 3, no. 1, pp. 9-44, Aug. 1988.
11. C. J. C. H. Watkins and P. Dayan, "Q-learning," Mach. Learn., vol. 8, no. 3/4, pp. 279-292, May 1992.
12. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge: MIT Press, 1998.
13. G. Boone, "Efficient reinforcement learning: Model-based acrobot control," in Proc. IEEE Int. Conf. Robot. Autom., Albuquerque, NM, 1997, pp. 229-234.
14. P. He and S. Jagannathan, "Reinforcement learning-based output feedback control of nonlinear systems with input constraints," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 35, no. 1, pp. 150-154, Feb. 2005.
15. L. Yang, J. Si, K. S. Tsakalis, and A. A. Rodriguez, "Direct heuristic dynamic programming for nonlinear tracking control with filtered tracking error," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 39, no. 6, pp. 1617-1622, Dec. 2009.
16. D. P. Bertsekas, Dynamic Programming and Optimal Control. Belmont, MA: Athena Scientific, 2000.
17. J. Si, A. G. Barto, W. B. Powell, and D. Wunsch, Eds., Handbook of Learning and Approximate Dynamic Programming. New York: Wiley-IEEE Press, 2004.
18. B. Igelnik and Y. H. Pao, "Stochastic choice of basis functions in adaptive function approximation and the functional-link net," IEEE Trans. Neural Networks, vol. 6, no. 6, pp. 1320-1329, Nov. 1995.
19. D. Prokhorov and D. Wunsch, "Adaptive critic designs," IEEE Trans. Neural Netw., vol. 8, no. 5, pp. 997-1007, Sep. 1997.
20. W. T. Miller, R. S. Sutton, and P. J. Werbos, Eds., Neural Networks for Control. Cambridge, MA: MIT Press, 1990.
21. P. J.Werbos, "Building and understanding adaptive systems: A statistical/numerical approach to factory automation and brain research," IEEE Trans. Syst., Man, Cybern., vol. SMC-17, no. 1, pp. 7-20, Jan. 1987.
22. P. J.Werbos, "Advanced forecasting methods for global crisis warning and models of intelligence," Gen. Syst. Yearbook, vol. 22, pp. 25-38, 1977.
23. T. Borgers and R. Sarin, "Learning through reinforcement and replicator dynamics," J. Economic Theory, vol. 77, no. 1, pp. 1-17, Nov. 1997.
24. Q. Yang, J. B. Vance, and S. Jagannathan, "Control of nonaffine nonlinear discrete-time systems using reinforcement learning-based linearly parameterized neural networks," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 994-1001, Aug. 2008.
25. S. Jagannathan, "Discrete-time CMAC control of a feedback linearizable nonlinear systems under a persistence of excitation," IEEE Trans. Neural Netw., vol. 10, no. 1, pp. 128-137, Jan. 1999.
26. S. Jagannathan, Neural Network Control of Nonlinear Discrete-time Systems. Boca Raton, FL: CRC Press, 2006.
27. T. Dierks and S. Jagannathan, "Online optimal control of nonlinear discrete-time systems using approximate dynamic programming," J. Control Theory Appl., vol. 9, no. 3, pp. 361-369, 2011.
28. M. Krstic, I. Kanellakopoulos, and P. Kokotovic, Nonlinear and Adaptive Control Design. New York: Wiley, 1995.
29. J. J. Murray, C. J. Cox, G. G. Lendaris, and R. Saeks, "Adaptive dynamic programming," IEEE Trans. Syst., Man, Cybern. C, Appl. Rev., vol. 32, no. 2, pp. 140-153, May 2002.
30. K. J. Astrom and B. Wittenmark, Adaptive Control., 2nd ed. Reading, MA: Addison-Wesley, 1994.
31. A. Al-Tamimi, F. L. Lewis, and M. Abu-Khalaf, "Discrete-time nonlinear HJB solution using approximate dynamic programming: Convergence proof," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 943-949, Aug. 2008.