Reinforcement learning and feedback control: Using natural decision methods to design optimal adaptive controllers

IEEE Control Systems

Volumn 32, Issue 6, 2012, Pages 76-105

  Lewis, Frank L ^a   Vrabie, Draguna ^b   Vamvoudakis, Kyriakos G ^c  

^a UNIVERSITY OF TEXAS AT ARLINGTON   (United States)

^b UNITED TECHNOLOGIES RESEARCH CENTER   (United States)

^c University of California   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ADAPTIVE CONTROLLERS; CONTINUOUS-TIME DYNAMICAL SYSTEMS; FEEDBACK CONTROLLER; INDIRECT ADAPTIVE CONTROLLERS; OPTIMAL CONTROL POLICY; OPTIMAL CONTROLLER; PERFORMANCE FUNCTIONS; SYSTEM IDENTIFICATION TECHNIQUES;

FEEDBACK CONTROL; IDENTIFICATION (CONTROL SYSTEMS); NONLINEAR CONTROL SYSTEMS; NONLINEAR EQUATIONS; ONLINE SYSTEMS; OPTIMAL CONTROL SYSTEMS; OPTIMIZATION; PHILOSOPHICAL ASPECTS; REINFORCEMENT LEARNING; RICCATI EQUATIONS;

ADAPTIVE CONTROL SYSTEMS;

EID: 84883537695     PISSN: 1066033X     EISSN: None     Source Type: Journal    
DOI: 10.1109/MCS.2012.2214134     Document Type: Review

References (60)

1. K. J. Astrom and B. Wittenmark, Adaptive Control. Reading, MA: Addison-Wesley, 1995.
2. P. Ioannou and B. Fidan, Adaptive Control Tutorial. Philadelphia, PA: SIAM Press, 2006.
3. F. L. Lewis, D. Vrabie, and V. Syrmos, Optimal Control, 3rd ed. New York: Wiley, 2012.
4. Z.-H. Li and M. Krstic, "Optimal design of adaptive tracking controllers for nonlinear systems," Automatica, vol. 33, no. 8, pp. 1459-1473, 1997. (Pubitemid 127392279)
5. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press, 1998.
6. W. B. Powell, Approximate Dynamic Programming. Hoboken, NJ: Wiley, 2007.
7. P. J. Werbos, "A menu of designs for reinforcement learning over time," in Neural Networks for Control, W. T. Miller, R. S. Sutton, and P. J. Werbos, Eds. Cambridge, MA: MIT Press, 1991, pp. 67-95.
8. F. L. Lewis, G. Lendaris, and D. Liu, "Special issue on adaptive dynamic programming and reinforcement learning for feedback control," IEEE Trans. Syst., Man, Cybern. B, vol. 38, no. 4, pp. 896-897, Aug. 2008.
9. X. Cao, Stochastic Learning and Optimization. Berlin, Germany: Springer-Verlag, 2007.
10. J. M. Mendel and R. W. MacLaren, "Reinforcement learning control and pattern recognition systems," in Adaptive, Learning, and Pattern Recognition Systems: Theory and Applications, J. M. Mendel and K. S. Fu, Eds. New York: Academic, 1970, pp. 287-318.
11. L. Busoniu, R. Babuska, B. De Schutter, and D. Ernst, Reinforcement Learning and Dynamic Programming Using Function Approximators. Boca Raton, FL: CRC Press, 2009.
12. W. Schultz, "Neural coding of basic reward terms of animal learning theory, game theory, microeconomics and behavioral ecology," Current Opinion Neurobiol., vol. 14, no. 2, pp. 139-147, 2004. (Pubitemid 38479929)
13. K. Doya, H. Kimura, and M. Kawato, "Neural mechanisms for learning and control," IEEE Control Syst. Mag., vol. 21, no. 4, pp. 42-54, Aug. 2000.
14. P. J. Werbos, "Approximate dynamic programming for real-time control and neural modeling," in Handbook of Intelligent Control, D. A. White and D. A. Sofge, Eds. New York: Van Nostrand Reinhold, 1992.
15. D. Vrabie and F. L. Lewis, "Neural network approach to continuoustime direct adaptive optimal control for partially-unknown nonlinear systems," Neural Netw., vol. 22, no. 3, pp. 237-246, Apr. 2009.
16. A. G. Barto, R. S. Sutton, and C. W. Anderson, "Neuron-like adaptive elements that can solve difficult learning control problems," IEEE Trans. Syst., Man, Cybern., vol. SMC-13, no. 5, pp. 834-846, Sep./Oct. 1983.
17. D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming. Belmont, MA: Athena Scientific, 1996.
18. R. E. Bellman, Dynamic Programming. Princeton, NJ: Princeton Univ. Press, 1957.
19. P. J. Werbos, "Neural networks for control and system identification," in Proc. IEEE Conf. Decision Control, Tampa, FL, 1989, pp. 260-265.
20. D. Prokhorov and D. Wunsch, "Adaptive critic designs," IEEE Trans. Neural Netw., vol. 8, no. 5, pp. 997-1007, Sep. 1997.
21. J. Si, A. Barto, W. Powell, and D. Wunsch, Handbook of Learning and Approximate Dynamic Programming. Piscataway, NJ: IEEE Press, 2004.
22. S. N. Balakrishnan, J. Ding, and F. L. Lewis, "Issues on stability of ADP feedback controllers for dynamical systems," IEEE Trans. Syst., Man, Cybern. B, vol. 38, no. 4, pp. 913-917, Aug. 2008.
23. F. Y. Wang, H. Zhang, and D. Liu, "Adaptive dynamic programming: An introduction," IEEE Comput, Intell, Mag., vol. 4, no. 2, pp. 39-47, May 2009.
24. F. L. Lewis and D. Vrabie, "Reinforcement learning and adaptive dynamic programming for feedback control," IEEE Circuits Syst. Mag., vol. 9, no. 3, pp. 32-50, 2009.
25. D. Han and S. N. Balakrishnan, "State-constrained agile missile control with adaptive-critic-based neural networks," IEEE Trans. Control Syst. Technol., vol. 10, no. 4, pp. 481-489, Jul. 2002. (Pubitemid 34798672)
26. D. Prokhorov, Computational Intelligence in Automotive Applications. New York: Springer-Verlag, 2008.
27. S. Ferrari and R. F. Stengel, "An adaptive critic global controller," in Proc. American Control Conf., May 2002, pp. 2665-2670.
28. D. Prokhorov, R. A. Santiago, and D. C. Wunsch, II, "Adaptive critic designs: A case study for neurocontrol," Neural Netw., vol. 8, no. 9, pp. 1367-1372, 1995. (Pubitemid 26072896)
29. J. J. Murray, C. J. Cox, G. G. Lendaris, and R. Saeks, "Adaptive dynamic programming," IEEE Trans. Syst., Man Cybern. C, vol. 32, no. 2, pp. 140-153, 2002.
30. R. Enns and J. Si, "Helicopter flight control reconfiguration for main rotor actuator failures," AIAA J. Guidance, Control, Dynamics, vol. 26, no. 4, pp. 572-584, 2003.
31. C. Lu, J. Si, and X. Xie, "Direct heuristic dynamic programming method for power system stability enhancement," IEEE Trans. Syst., Man, Cybern. B, vol. 38, no. 4, pp. 1008-1013, 2008.
32. G. G. Lendaris, L. Schultz, and T. Shannon, "Adaptive critic design for intelligent steering and speed control of a 2-axle vehicle," in Proc. Int. Conf. Neural Networks, 2000, pp. 73-78.
33. X. Liu and S. N. Balakrishnan, "Convergence analysis of adaptive critic based optimal control," in Proc. American Control Conf., June 2000, pp. 1929-1933.
34. A. Al-Tamimi, F. L. Lewis, and M. Abu-Khalaf, "Discrete-time nonlinear HJB solution using approximate dynamic programming: Convergence proof," IEEE Trans. Syst., Man, Cybern. B, vol. 38, no. 4, pp. 943-949, Aug. 2008.
35. C. Darwin, On the Origin of Species by Means of Natural Selection. London, U.K.: John Murray, 1859.
36. D. G. Luenberger, Introduction to Dynamic Systems. New York: Wiley, 1979.
37. D. Vrabie, O. Pastravanu, M. Abu-Khalaf, and F. L. Lewis, "Adaptive optimal control for continuous-time linear systems based on policy iteration," Automatica, vol. 45, no. 2, pp. 477-484, 2009.
38. A. Papoulis, Probability Random Variables and Stochastic Processes. New York: McGraw-Hill, 2002.
39. R. M. Wheeler and K. S. Narendra, "Decentralized learning in finite Markov chains," IEEE Trans. Autom. Control, vol. 31, no. 6, pp. 519-526, June 1986.
40. P. Mehta and S. Meyn, "Q-learning and Pontryagin's minimum principle," in Proc. IEEE Conf. Decision Control, Dec. 2009, pp. 3598-3605.
41. H. Zhang, J. Huang, and F. L. Lewis, "Algorithm and stability of ATC receding horizon control," in Proc. IEEE Symp. Adaptive Dynamic Programming Reinforcement, Nashville, TN, Mar. 2009, pp. 28-35.
42. C. Watkins, "Learning from delayed rewards," Ph.D. dissertation, Cambridge University, Cambridge, U.K., 1989.
43. C. J. C. H. Watkins and P. Dayan, "Q-learning," Mach. Learn., vol. 8, no. 3-4, pp. 279-292, 1992.
44. G. A. Hewer, "An iterative technique for the computation of the steady state gains for the discrete optimal regulator," IEEE Trans Autom. Control, vol. 16, no. 4, pp. 382-384, Aug. 1971.
45. P. Lancaster and L. Rodman, Algebraic Riccati Equations. London, U.K.: Oxford Univ. Press, 1995.
46. K. L. Moore, Iterative Learning Control for Deterministic Systems. London, U.K.: Springer-Verlag, 1993.
47. A. J. Van, "L2-gain analysis of nonlinear systems and nonlinear state feedback H? control," IEEE Trans. Autom. Control, vol. 37, no. 6, pp. 770-784, 1992.
48. L. Ljung, System Identification: Theory for the User. Englewood Cliffs, NJ: Prentice-Hall, 1999.
49. S. Bradtke, B. Ydstie, and A. Barto, "Adaptive linear quadratic control using policy iteration," in Proc. Amer. Control Conf., Baltimore, MD, 1994, pp. 3475-3479.
50. F. L. Lewis and K. G. Vamvoudakis, "Reinforcement learning for partially observable dynamic processes: Adaptive dynamic programming using measured output data," IEEE Trans. Syst., Man, Cybern. B, vol. 41, no. 1, pp. 14-25, Feb. 2011.
51. M. Abu-Khalaf, F. L. Lewis, and J. Huang, "Policy iterations on the Hamilton-Jacobi-Isaacs equation for state feedback control with input saturation," IEEE Trans. Autom. Control, vol. 51, no. 12, pp. 1989-1995, Dec. 2006. (Pubitemid 46002295)
52. L. C. Baird, "Reinforcement learning in continuous time: Advantage updating," in Proc. Int. Conf. Neural Networks, Orlando, FL, June1994, pp. 2448-2453.
53. K. Doya, "Reinforcement learning in continuous time and space," Neural Comput., vol. 12, no. 1, pp. 219-245, 2000.
54. T. Hanselmann, L. Noakes, and A. Zaknich, "Continuous-time adaptive critics," IEEE Trans. Neural Netw., vol. 18, no. 3, pp. 631-647, May 2007.
55. D. L. Kleinman, "On an iterative technique for Riccati equation computations," IEEE Trans. Autom. Control, vol. AC-13, no. 1, pp. 114-115, Feb. 1968.
56. V. Nevistic and J. Primbs, "Constrained nonlinear optimal control: A converse HJB approach," Dept. Control Dynamical Systems, California Institute of Technology, Pasadena, CA, Tech. Rep. 96-021, 1996.
57. K. G. Vamvoudakis and F. L. Lewis, "Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem," Automatica, vol. 46, no. 5, pp. 878-888, 2010.
58. D. Vrabie and F. L. Lewis, "Adaptive dynamic programming for online solution of a zero-sum differential game," J. Control Theory: Its Appl., vol. 9, no. 3, pp. 353-360, 2011.
59. K. G. Vamvoudakis and F. Lewis, "Multi-player non-zero sum games: Online adaptive learning solution of coupled Hamilton-Jacobi equations," Automatica, vol. 47, no. 8, pp. 556-569, 2011.
60. J. H. Kim and F. L. Lewis, "Model-free H-infinity control design for unknown linear discrete-time systems via Q-learning with LMI," Automatica, vol. 46, no. 8, pp. 1320-1326, Aug. 2010.