Online learning control using adaptive critic designs with sparse kernel machines

IEEE Transactions on Neural Networks and Learning Systems

Volumn 24, Issue 5, 2013, Pages 762-775

  Xu, Xin ^a   Hou, Zhongsheng ^b   Lian, Chuanqiang ^a   He, Haibo ^c  

^a NATIONAL UNIVERSITY OF DEFENSE TECHNOLOGY   (China)

^b BEIJING JIAOTONG UNIVERSITY   (China)

^c University of Rhode Island   (United States)

Author keywords

Adaptive critic designs; Approximate dynamic programming; Kernel machines; Learning control; Markov decision processes; Reinforcement learning

Indexed keywords

ADAPTIVE CRITIC DESIGNS; APPROXIMATE DYNAMIC PROGRAMMING; KERNEL MACHINE; LEARNING CONTROL; MARKOV DECISION PROCESSES;

DYNAMICAL SYSTEMS; HEURISTIC PROGRAMMING; LEARNING ALGORITHMS; MARKOV PROCESSES; NEURAL NETWORKS; REINFORCEMENT LEARNING;

E-LEARNING;

ALGORITHM; ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; DECISION SUPPORT SYSTEM; FEEDBACK SYSTEM; HUMAN; LEARNING; ONLINE SYSTEM; PROBABILITY;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; DECISION SUPPORT TECHNIQUES; FEEDBACK; HUMANS; LEARNING; MARKOV CHAINS; ONLINE SYSTEMS;

EID: 84884922436     PISSN: 2162237X     EISSN: 21622388     Source Type: Journal    
DOI: 10.1109/TNNLS.2012.2236354     Document Type: Article

References (48)

1. R. Sutton and A. Barto, Reinforcement Learning. Introduction. Cambridge, MA: MIT Press, 1998.
2. F. Y. Wang, H. Zhang, and D. Liu, "Adaptive dynamic programming: An introduction," IEEE Comput. Intell. Mag., vol. 4, no. 2, pp. 39-47, May 2009.
3. C. Szepesvári, Algorithms for Reinforcement Learning. San Mateo, CA: Morgan, 2010.
4. D. A. White and D. A. Sofge, Handbook of Intelligent Control. New York: Van Nostrand, 1992.
5. P. J. Werbos, "Intelligence in the brain: A theory of how it works and how to build it," Neural Netw., vol. 22, no. 3, pp. 200-212, Apr. 2009.
6. P. J. Werbos, "Using ADP to understand and replicate brain intelligence: The next level design," in Proc. IEEE Int. Symp. Approx. Dynamic Program. Reinforcement Learn., Apr. 2007, pp. 209-216.
7. D. P. Bertsekas and J. Tsitsiklis, Neuro-Dynamic Programming. Belmont, MA: Athena Scientific, 1996.
8. D. Liu, Y. Zhang, and H. Zhang, "A self-learning call admission control scheme for CDMA cellular networks," IEEE Trans. Neural Netw., vol. 16, no. 5, pp. 1219-1228, Sep. 2005. (Pubitemid 41444623)
9. R. H. Crites and A. G. Barto, "Elevator group control using multiple reinforcement learning agents," Mach. Learn., vol. 33, nos. 2-3, pp. 235-262, Nov. 1998.
10. G. Tesauro, "TD-Gammon, a self-teaching backgammon program, achieves master-level play," Neural Comput., vol. 6, no. 2, pp. 215-219, Mar. 1994.
11. P. Shih, B. C. Kaul, S. Jagannathan, and J. A. Drallmeier, "Reinforcement-learning-based dual-control methodology for complex nonlinear discrete-time systems with application to spark engine EGR operation," IEEE Trans. Neural Netw., vol. 19, no. 8, pp. 1369-1388, Aug. 2008.
12. W. B. Powell, Approximate Dynamic Programming: Solving the Curses of Dimensionality. New York: Wiley, 2007.
13. L. Busoniu, R. Babuska, B. De Schutter, and D. Ernst, Reinforcement Learning and Dynamic Programming Using Function Approximators. Boca Raton, FL: CRC Press, 2010
14. R. S. Sutton, H. R. Maei, D. Precup, S. Bhatnagar, D. Silver, C. Szepesvari, and E. Wiewiora, "Fast gradient-descent methods for temporaldifference learning with linear function approximation," in Proc. Int. Conf. Mach. Learn., 2009, pp. 993-1000.
15. J. Baxter and P. L. Bartlett, "Infinite-horizon policy-gradient estimation," J. Artif. Intell. Res., vol. 15, no. 1, pp. 319-350, Jul. 2001.
16. V. R. Konda and J. N. Tsitsiklis, "Actor-critic algorithms," in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 2000.
17. D. V. Prokhorov and D. C. Wunsch, "Adaptive critic designs," IEEE Trans. Neural Netw., vol. 8, no. 5, pp. 997-1007, Jul. 1997.
18. A. G. Barto, R. S. Sutton, and C. W. Anderson, "Neuron-like adaptive elements that can solve difficult learning control problems," IEEE Trans. Syst., Man, Cybern., vol. 13, no. 5, pp. 834-846, Sep.-Oct. 1983.
19. G. K. Venayagamoorthy, R. G. Harley, and D. C. Wunsch, "Comparison of heuristic dynamic programming and dual heuristic programming adaptive critics for neurocontrol of a turbogenerator," IEEE Trans. Neural Netw., vol. 13, no. 3, pp. 764-773, May 2002.
20. F. L. Lewis and D. Vrabie, "Reinforcement learning and adaptive dynamic programming for feedback control," IEEE Circuits Syst. Mag., vol. 9, no. 3, pp. 32-50, Aug. 2009.
21. J. Peters and S. Schaal, "Natural actor-critic," Neurocomputing, vol. 71, nos. 7-9, pp. 1180-1190, Mar. 2008.
22. S. Bhatnagar, R. S. Sutton, M. Ghavamzadeh, and M. Lee, Natural Actor-Critic Algorithms. Alberta, Canada: Dept. Comput. Sci., 2009.
23. S. N. Balakrishnan and V. Biega, "Adaptive-critic-based neural networks for aircraft optimal control," J. Guid., Control, Dynamics, vol. 19, no. 4, pp. 893-898, 1996. (Pubitemid 126539437)
24. R. Enns and J. Si, "Helicopter trimming and tracking control using direct neural dynamic programming," IEEE Trans. Neural Netw., vol. 14, no. 4, pp. 929-939, Jul. 2003.
25. C. Lu, J. Si, and X. Xie, "Direct heuristic dynamic programming for damping oscillations in a large power system," IEEE Trans. Syst., Man, Cybern., Part B, Cybern., vol. 38, no. 4, pp. 1008-1013, Aug. 2008.
26. P. Shih, B. C. Kaul, S. Jagannathan, and J. A. Drallmeier, "Reinforcement-learning-based dual-control methodology for complex nonlinear discrete-time systems with application to spark engine EGR operation," IEEE Trans. Neural Netw., vol. 19, no. 8, pp. 1369-1388, Aug. 2008.
27. G. D. Magoulasa, M. N. Vrahatisb, and G. S. Androulakisb, "Effective Backpropagation training with variable stepsize," Neural Netw., vol. 10, no. 1, pp. 69-82, Jan. 1997.
28. S. Bhasin, N. Sharma, P. Patre, and W. E. Dixon, "Asymptotic tracking by a reinforcement learning-based adaptive critic controller," J. Control Theory Appl., vol. 9, No. 3, pp. 400-409, 2011.
29. K. G. Vamvoudakis and F. L. Lewis, "Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem," Automatica, vol. 46, no. 5, pp. 878-888, May 2010.
30. H. Zhang, L. Cui, X. Zhang, and Y. Luo, "Data-driven robust approximate optimal tracking control for unknown general nonlinear systems using adaptive dynamic programming method," IEEE Trans. Neural Netw., vol. 22, no. 12, pp. 2226-2236, Dec. 2011.
31. V. Vapnik, Statistical Learning Theory. New York: Wiley, 1998.
32. B. Schölkopf and A. Smola, Learning With Kernels. Cambridge, MA: MIT Press, 2002.
33. N. Cristianini and J. Shawe-Taylor, An Introduction to Support Vector Machines . Cambridge, U.K.: Cambridge Univ. Press, 2000.
34. F. R. Bach and M. I. Jordan, "Kernel independent component analysis," J. Mach. Learn. Res., vol. 3, pp. 1-48, Jul. 2002.
35. T. Hofmann, B. Schölkopf, and A. J. Smola, "Kernel methods in machine learning," Ann. Statist., vol. 36, no. 3 pp. 1171-1220, 2008.
36. D. Ormoneit and S. Sen, "Kernel-based reinforcement learning," Mach. Learn., vol. 49, nos. 2-3, pp. 161-178, 2002.
37. Y. Engel, S. Mannor, and R. Meir, "Bayes meets bellman: The Gaussian Process approach to temporal difference learning," in Proc. Int. Conf. Mach. Learn., 2003, pp. 154-161.
38. T. G. Dietterich and X. Wang, "Batch value function approximation via support vectors," in Advances in Neural Information Processing Systems 14, Cambridge, MA: MIT Press, 2002, pp. 1491-1498.
39. C. E. Rasmussen and M. Kuss, "Gaussian processes in reinforcement learning," in Advances in Neural Information Processing Systems 16, S. Thrun, L. K. Saul, and B. Schölkopf, Eds., Cambridge, MA: MIT Press, 2004, pp. 751-759.
40. M. G. Lagoudakis and R. Parr, "Least-squares policy iteration," J. Mach. Learn. Res., vol. 4, pp. 1107-1149, Dec. 2003.
41. X. Xu, D. Hu, and X. Lu, "Kernel-based least-squares policy iteration for reinforcement learning," IEEE Trans. Neural Netw., vol. 18, no. 4, pp. 973-992, Jul. 2007. (Pubitemid 47098876)
42. Y. Engel, S. Mannor, and R. Meir, "The kernel recursive least-squares algorithm," IEEE Trans. Signal Process., vol. 52, no. 8, pp. 2275-2285, Aug. 2004.
43. X. Xu, H. G. He, and D. W. Hu, "Efficient reinforcement learning using recursive least-squares methods," J. Artif. Intell. Res., vol. 16, pp. 259-292, Jun. 2002. (Pubitemid 43057174)
44. X. Xu, T. Xie, D. Hu, and X. Lu, "Kernel least-squares temporal difference learning," Int. J. Inf. Technol., vol. 11, no. 9, pp. 54-63, 2005.
45. J. N. Tsitsiklis and B. V. Roy, "An analysis of temporal difference learning with function approximation," IEEE Trans. Autom. Control, vol. 42, no. 5, pp. 674-690, May 1997.
46. A. Nedic and D. P. Bertsekas, "Least squares policy evaluation algorithms with linear function approximation," Discrete Event Dyn. Syst., vol. 13, nos. 1-2, pp. 79-110, Jan.-Apr. 2003.
47. T. Dierks and S. Jagannathan, "Online optimal control of affine nonlinear discrete-time systems with unknown internal dynamics by using time-based policy update," IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 7, pp. 1118-1129, Jul. 2012.
48. S. Zhong, X. Zeng, S. Wu, and L. Han, "Sensitivity-based adaptive learning rules for binary feedforward neural networks," IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 3, pp. 480-491, Mar. 2012.