Kernel-based approximate dynamic programming for real-time online learning control: An experimental study

IEEE Transactions on Control Systems Technology

Volumn 22, Issue 1, 2014, Pages 146-156

  Xu, Xin ^a   Lian, Chuanqiang ^a   Zuo, Lei ^a   He, Haibo ^b  

^a NATIONAL UNIVERSITY OF DEFENSE TECHNOLOGY   (China)

^b The University of Rhode Island   (United States)

Author keywords

Approximate dynamic programming (ADP); learning control; Markov decision processes (MDPs); online learning; reinforcement learning (RL)

Indexed keywords

APPROXIMATE DYNAMIC PROGRAMMING; DUAL HEURISTIC PROGRAMMING; FUNCTION APPROXIMATION TECHNIQUES; LEARNING CONTROL; MARKOV DECISION PROCESSES; MULTI-LAYER PERCEPTRON NEURAL NETWORKS; ONLINE LEARNING; UNCERTAIN NONLINEAR SYSTEMS;

HEURISTIC PROGRAMMING; LEARNING ALGORITHMS; MARKOV PROCESSES; NEURAL NETWORKS; NONLINEAR DYNAMICAL SYSTEMS; OPTIMAL CONTROL SYSTEMS; PENDULUMS; REAL TIME CONTROL; REINFORCEMENT LEARNING; UNCERTAINTY ANALYSIS;

E-LEARNING;

EID: 84891557251     PISSN: 10636536     EISSN: None     Source Type: Journal    
DOI: 10.1109/TCST.2013.2246866     Document Type: Article

References (33)

1. R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction, Cambridge, MA, USA: MIT Press, 1998.
2. L. P. Kaelbling, M. L. Littman, and A. W. Moore, "Reinforcement learning: A survey," J. Artif. Intell. Res., vol. 4, pp. 237-285, May 1996. (Pubitemid 126646155)
3. W. B. Powell, Approximate Dynamic Programming: Solving the Curses of Dimensionality, New York, USA: Wiley, 2007.
4. A. Alessandri, M. Gaggero, and R. Zoppoli "Feedback optimal control of distributed parameter systems by using finite-dimensional approximation schemes," IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 6, pp. 984-996, Jun. 2012.
5. F. Y. Wang, H. Zhang, and D. Liu, "Adaptive dynamic programming: An introduction," IEEE Comput. Intell. Mag., vol. 4, no. 2, pp. 39-47, May 2009.
6. M. A. Wiering and H. van Hasselt, "Ensemble algorithms in reinforcement learning," IEEE Trans. Syst. Man Cybern. B, Cybern., vol. 38, no. 4, pp. 930-936, Aug. 2008.
7. F. L. Lewis and D. Vrabie, "Reinforcement learning and adaptive dynamic programming for feedback control," IEEE Circuits Syst. Mag., vol. 9, no. 3, pp. 32-50, Jul.-Sep. 2009.
8. R. S. Sutton, H. R. Maei, D. Precup, S. Bhatnagar, D. Silver, C. Szepesvári, and E. Wiewiora, "Fast gradient-descent methods for temporal-difference learning with linear function approximation," in Proc. Int. Conf. Mach. Learn., 2009, pp. 993-1000.
9. J. Baxter and P. L. Bartlett, "Infinite-horizon policy-gradient estimation," J. Artif. Intell. Res., vol. 15, pp. 319-350, Nov. 2001.
10. V. R. Konda and J. N. Tsitsiklis, "Actor-critic algorithms," in Advances in Neural Information Processing Systems, Cambridge, MA, USA: MIT Press, 2000.
11. D. V. Prokhorov and D. C. Wunsch, "Adaptive critic designs," IEEE Trans. Neural Netw., vol. 8, no. 5, pp. 997-1007, Sep. 1997. (Pubitemid 127763331)
12. S. Bhasin, N. Sharma, P. Patre, and W. E. Dixon, "Asymptotic tracking by a reinforcement learning-based adaptive critic controller," J. Control Theory Appl., vol. 9, no. 3, pp. 400-409, Aug. 2011.
13. Z. Sun, "Adaptive critic design for energy minimization of portable video communication devices," IEEE Trans. Circuits Syst. Video Technol., vol. 20, no. 1, pp. 27-37, Jun. 2010.
14. J.-W. Sheu and W.-S. Lin, "Adaptive optimal control for designing automatic train regulation for metro line," IEEE Trans. Control Syst. Technol., vol. 20, no. 5, pp. 1319-1327, Sep. 2012.
15. H. Zhang, L. Cui, X. Zhang, and Y. Luo, "Data-driven robust approximate optimal tracking control for unknown general nonlinear systems using adaptive dynamic programming method," IEEE Trans. Neural Netw., vol. 22, no. 12, pp. 2226-2236, Dec. 2011.
16. J. Peters and S. Schaal, "Natural actor-critic," Neurocomputing, vol. 71, nos. 7-9, pp. 1180-1190, Mar. 2008.
17. G. K. Venayagamoorthy, R. G. Harley, and D. C. Wunsch, "Comparison of heuristic dynamic programming and dual heuristic programming adaptive critics for neurocontrol of a turbogenerator," IEEE Trans. Neural Netw., vol. 13, no. 3, pp. 764-773, May 2002. (Pubitemid 34669664)
18. K. G. Vamvoudakis and F. L. Lewis, "Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem," Automatica, vol. 46, no. 5, pp. 878-888, May 2010.
19. R. Enns and J. Si, "Helicopter trimming and tracking control using direct neural dynamic programming," IEEE Trans. Neural Netw., vol. 14, no. 4, pp. 929-939, Jul. 2003.
20. C. Lu, J. Si, and X. Xie, "Direct heuristic dynamic programming for damping oscillations in a large power system," IEEE Trans. Syst. Man Cybern. B, Cybern., vol. 38, no. 4, pp. 1008-1013, Aug. 2008.
21. P. Shih, B. C. Kaul, S. Jagannathan, and J. A. Drallmeier, "Reinforcement-learning-based dual-control methodology for complex nonlinear discrete-time systems with application to spark engine EGR operation," IEEE Trans. Neural Netw., vol. 19, no. 8, pp. 1369-1388, Aug. 2008.
22. D. Ormoneit and S. Sen, "Kernel-based reinforcement learning," Mach. Learn., vol. 49, nos. 2-3, pp. 161-178, Nov. 2002. (Pubitemid 34325684)
23. Y. Engel, S. Mannor, and R. Meir, "Bayes meets bellman: The gaussian process approach to temporal difference learning," in Proc. 20th Int. Conf. Mach. Learn., 2003, pp. 154-161.
24. T. G Dietterich and X. Wang, "Batch value function approximation via support vectors," in Advances in Neural Information Processing Systems, Cambridge, MA, USA: MIT Press, 2002.
25. C. E. Rasmussen and M. Kuss, "Gaussian Processes in reinforcement learning," in Advances in Neural Information Processing Systems, Cambridge, MA, USA: MIT Press, 2004.
26. X. Xu, D. Hu, and X. Lu, "Kernel-based least squares policy iteration for reinforcement learning," IEEE Trans. Neural Netw., vol. 19, no. 4, pp. 973-992, Jul. 2007. (Pubitemid 47098876)
27. V. Vapnik, Statistical Learning Theory, New York, USA: Wiley, 1998.
28. B. Schölkopf and A. Smola, Learning with Kernels, Cambridge, MA, USA: MIT Press, 2002.
29. S. Bhatnagar, R. S. Sutton, M. Ghavamzadeh, and M. Lee. "Natural actor-critic algorithms," Automatica, vol. 45, no. 11, pp. 2471-2482, Nov. 2009.
30. C. H. Chiu, "The design and implementation of a wheeled inverted pendulum using an adaptive output recurrent cerebellar model articulation controller," IEEE Trans. Ind. Electron., vol. 57, no. 5, pp. 1814-1822, May 2010.
31. A. Casavola, E. Mosca, and M. Papini, "Control under constraints: An application of the command governor approach to an inverted pendulum," IEEE Trans. Control Syst. Technol., vol. 12, no. 1, pp. 193-204, Jan. 2004.
32. M. I. El-Hawwary, A. L. Elshafei, H. M. Emara, and H. A. A. Fattah, "Adaptive fuzzy control of the inverted pendulum problem," IEEE Trans. Control Syst. Technol., vol. 14, no. 6, pp. 1135-1144, Nov. 2006. (Pubitemid 44637628)
33. C. Cervellera, M. Gaggero, and D. Maccio, "Efficient kernel models for learning and approximate minimization problems," Neurocomputing, vol. 97, pp. 74-85, Nov. 2012.