Adaptive learning in tracking control based on the dual critic network design

IEEE Transactions on Neural Networks and Learning Systems

Volumn 24, Issue 6, 2013, Pages 913-928

  Ni, Zhen ^a   He, Haibo ^a   Wen, Jinyu ^b  

^a University of Rhode Island   (United States)

^b HUAZHONG UNIVERSITY OF SCIENCE AND TECHNOLOGY   (China)

Author keywords

Adaptive critic design (ACD); adaptive dynamic programming (ADP); internal goal; lyapunov stability analysis; online learning; reinforcement learning; tracking control; virtual reality

Indexed keywords

ADAPTIVE CRITIC DESIGNS; ADAPTIVE DYNAMIC PROGRAMMING; INTERNAL GOAL; LYAPUNOV STABILITY ANALYSIS; ONLINE LEARNING; TRACKING CONTROLS;

BENCHMARKING; DYNAMIC PROGRAMMING; NAVIGATION; REINFORCEMENT LEARNING; VIRTUAL REALITY;

DESIGN;

EID: 84876149222     PISSN: 2162237X     EISSN: 21622388     Source Type: Journal    
DOI: 10.1109/TNNLS.2013.2247627     Document Type: Article

References (54)

1. F. Lewis and D. Liu, eds., Reinforcement Learning and Approximate Dynamic Programming for Feedback Control. Piscataway, NJ, USA: IEEE Press, 2013.
2. J. Si, A. G. Barto, W. B. Powell, and D. C. Wunsch, eds., Handbook of Learning and Approximate Dynamic Programming. Piscataway, NJ, USA: IEEE Press, 2004.
3. D. Bertsekas, Dynamic Programming and Optimal Control. Belmont, MA, USA: Athena Scientific, 1995.
4. W. Powell, Approximate Dynamic Programming: Solving the Curses of Dimensionality. New York, USA: Wiley, 2007.
5. J. J. Murray, C. J. Cox, G. G. Lendaris, and R. Saeks, "Adaptive dynamic programming," IEEE Trans. Syst. Man, Cybern. C, Appl. Rev., vol. 32, no. 2, pp. 140-153, May 2002. (Pubitemid 35289398)
6. J. Lee and J. Lee, "Approximate dynamic programming strategies and their applicability for process control: A review and future directions," Int. J. Control Autom. Syst., vol. 2, no. 3, pp. 263-278, Sep. 2004.
7. H. He, Self-Adaptive Systems for Machine Intelligence. New York,USA: Wiley, 2011.
8. P. Werbos, "Reinforcement learning and approximate dynamic programming (RLADP)-foundations, common misconceptions and challenges ahead," in Reinforcement Learning and Approximate Dynamic Programming for Feedback Control. New York, USA: Wiley, 2013, pp. 3-30.
9. P. J. Werbos, "Intelligence in the brain: A theory of how it works and how to build it," Neural Netw., vol. 22, no. 3, pp. 200-212, Apr. 2009.
10. P. J. Werbos, "ADP: The key direction for future research in intelligent control and understanding brain intelligence," IEEE Trans. Syst. Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 898-900, Aug. 2008.
11. D. Prokhorov and D. Wunsch, "Adaptive critic designs," IEEE Trans. Neural Netw., vol. 8, no. 5, pp. 997-1007, Sep. 1997. (Pubitemid 127763331)
12. F.-Y. Wang, H. Zhang, and D. Liu, "Adaptive dynamic programming: An introduction," IEEE Comput. Intell. Mag., vol. 4, no. 2, pp. 39-47, May 2009.
13. J. Si and Y.-T. Wang, "Online learning control by association and reinforcement," IEEE Trans. Neural Netw., vol. 12, no. 2, pp. 264-276, Mar. 2001. (Pubitemid 32371483)
14. R. Enns and J. Si, "Helicopter trimming and tracking control using direct neural dynamic programming," IEEE Trans. Neural Netw., vol. 14, no. 4, pp. 929-939, Jul. 2003.
15. Z. Ni, H. He, D. Zhao, and D. Prokhorov, "Reinforcement learning control based on multi-goal representation using hierarchical heuristic dynamic programming," in Proc. Int. Joint Conf. Neural Netw., Jun. 2012, pp. 1-8.
16. G. K. Venayagamoorthy, R. G. Harley, and D. C. Wunsch, "Comparison of heuristic dynamic programming and dual heuristic programming adaptive critics for neurocontrol of a turbogenerator," IEEE Trans. Neural Netw., vol. 13, no. 3, pp. 764-773, May 2002. (Pubitemid 34669664)
17. J. W. Park, R. G. Harley, and G. K. Venayagamoorthy, " Adaptivecritic-based optimal neurocontrol for synchronous generators in a power system using MLP/RBF neural networks," IEEE Trans. Ind. Appl., vol. 39, no. 5, pp. 1529-1540, Sep.-Oct. 2003.
18. H. Zhang, Q. Wei, and Y. Luo, "A novel infinte-time optimal tracking control scheme for a class of discrete-time nonlinear systems via the greedy HDP iteration algorithm," IEEE Trans. Syst. Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 937-942, Aug. 2008.
19. H. Zhang, L. Cui, X. Zhang, and Y. Luo, "Data-driven robust approximate optimal tracking control for unknown general nonlinear systems using adaptive dynamic programming method," IEEE Trans. Neural Netw., vol. 22, no. 12, pp. 2226-2236, Dec. 2011.
20. H. Zhang, R. Song, Q. Wei, and T. Zhang, "Optimal tracking control for a class of nonlinear discrete-time systems with time delays based on heuristic dynamic programming," IEEE Trans. Neural Netw., vol. 22, no. 12, pp. 1851-1862, Dec. 2011.
21. L. Yang, J. Si, K. S. Tsakalis, and A. A. Rodriguez, "Direct heuristic dynamic programming for nonlinear tracking conrol with filtered tracking error," IEEE Trans. Syst. Man, Cybern. B, Cybern., vol. 39, no. 6, pp. 1617-1622, Dec. 2009.
22. P. He and S. Jagannathan, "Reinforcement learning neural-networkbased controller for nonlinear discrete-time systems with input contraints," IEEE Trans. Syst. Man, Cybern. B, Cybern., vol. 37, no. 2, pp. 425-436, Apr. 2007. (Pubitemid 46523230)
23. P. He and S. Jagannathan, "Reinforcement-based neuro-output feedback control of nonlinear discrete-time systems with input constraints," IEEE Trans. Systems Man, Cybern. B, Cybern., vol. 35, no. 1, pp. 150-154, Feb. 2005. (Pubitemid 40226209)
24. S. Jagannathan and P. He, "Neural network-based state feedback control of nonlinear discrete-time system in non-strict feedback form," IEEE Trans. Neural Netw., vol. 19, no. 12, pp. 2073-2087, Dec. 2008.
25. Z. Ni, X. Fang, H. He, D. Zhao, and X. Xu, "Real-time tracking control on adaptive critic design with uniformly ultimately bounded condition," in Proc. IEEE Symp. Adapt. Dynamic Program. Reinforcement Learn., IEEE Symp. Ser. Comput. Intell., Apr. 2013, to be published.
26. D. Liu, H. Javaherian, O. Kovalenko, and T. Huang, "Adaptive critic learning techniques for engine torque and air-fuel ratio control," IEEE Trans. Systems Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 988-993, Aug. 2008.
27. J. Fu, H. He, and X. Zhou, "Adaptive learning and control for MIMO system based on adaptive dynamic programming," IEEE Trans. Neural Netw., vol. 22, no. 7, pp. 1133-1148, Jul. 2011.
28. J. Fu, H. He, Q. Liu, and Z. Ni, "An adaptive dynamic programming approach for closely-coupled mimo system control," in Proc. 8th Int. Symp. Neural Netw., vol. 6677. Jun. 2011, pp. 1-10.
29. Z. Ni, H. He, D. V. Prokhorov, and J. Fu, "An online actor-critic learning approach with levenberg-marquardt algorithm," in Proc. Int. Joint Conf. Neural Netw., Aug. 2011, pp. 2333-2340.
30. G. K. Venayagamoorthy, R. G. Harley, and D. C. Wunsch, "Dual heuristic programming excitation neurocontrol for generators in a multimachine power system," IEEE Trans. Ind. Appl., vol. 39, no. 2, pp. 382-394, Apr.-May 2003.
31. W. Qiao, G. Venayagamoorthy, and R. Harley, "DHP-based wide-area coordinating control of a power system with a large wind farm and multiple FACTS devices," in Proc. IEEE Int. Conf. Neural Netw., Aug. 2007, pp. 2093-2098.
32. S. Ray, G. K. Venayagamoorthy, B. Chaudhuri, and R. Majumder, "Comparison of adaptive critics and classical approaches based wide area controllers for a power system," IEEE Trans. Syst. Man, Cybern., B, Cybern., vol. 38, no. 4, pp. 1002-1007, Aug. 2008.
33. Y. Tang, H. He, Z. Ni, J. Wen, and X. Sui, "Reactive power control of grid-connected wind farm based on adaptive dynamic programming," Neurocomputing, to be published.
34. D. Liu, Y. Zhang, and H. G. Zhang, "A self-learning call admission control scheme for CDMA cellular networks," IEEE Trans. Neural Netw., vol. 16, no. 5, pp. 1219-1228, Sep. 2005. (Pubitemid 41444623)
35. H. G. Zhang, Y. H. Luo, and D. Liu, "Neural-network-based nearoptimal control for a class of discrete-time affine nonlinear systems with control constraints," IEEE Trans. Neural Netw., vol. 20, no. 9, pp. 1490-1503, Sep. 2009.
36. F.-Y.Wang, N. Jin, D. Liu, and Q.Wei, "Adaptive dynamic programming for finite-horizon optimal control of discrete-time nonlinear systems with ε-error bound," IEEE Trans. Neural Netw., vol. 22, no. 1, pp. 24-36, Jan. 2011.
37. P. H. Eaton, and D. V. Prokhorov, and D. C. Wunsch, "Neurocontroller alternatives for 'fuzzy' ball-and-beam systems with nonuniform nonlinear friction," IEEE Trans. Neural Netw., vol. 11, no. 2, pp. 423-435, Mar. 2000.
38. M. Fairbank, E. Alonso, and D. V. Prokhorov, "Simple and fast calculation of the second-order gradients for globalized dual heuristic dynamic programming in neural networks," IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 10, pp. 1671-1676, Oct. 2012.
39. H. He, Z. Ni, and D. Zhao, "Data-driven learning and control with multiple critic networks," in Proc. 10th World Congr. Int. Control Autom., Jul. 2012, pp. 523-527.
40. G. G. Lendaris and J. C. Neidhoefer, "Guidance in the use of adaptive critics for control," in Handbook of Learning and Approximate Dynamic Programming. Piscataway, NJ, USA: IEEE Press, 2004, pp. 97-118.
41. H.Wong, M. de Queiroz, and V. Kapila, "Adaptive tracking control using synthesized velocity from attitude measurements," Automatica, vol. 37, no. 6, pp. 947-953, Jun. 2001. (Pubitemid 32426077)
42. G. Tao, S. Joshi, and X. Ma, "Adaptive state feedback and tracking control of systems with actuator failures," IEEE Trans. Autom. Control, vol. 46, no. 1, pp. 78-95, Jan. 2001. (Pubitemid 32191022)
43. H. Zhang and L. Cai, "Nonlinear adaptive control using the fourier integral and its application to cstr systems," IEEE Trans. Systems, Man, Cybern. B, Cybern., vol. 32, no. 3, pp. 367-372, Jun. 2002. (Pubitemid 34692409)
44. H. He, Z. Ni, and J. Fu, "A three-network architecture for on-line learning and optimization based on adaptive dynamic programming," Neurocomputing, vol. 78, no. 1, pp. 3-13, Feb. 2012.
45. H. He, Z. Ni, and D. Zhao, "Learning and optimization in hierarchical adaptive critic design," in Reinforcement Learning and Approximate Dynamic Programming for Feedback Control. Piscataway, NJ, USA: IEEE Press, 2013, pp. 78-95.
46. F. L. Lewis, S. Jagannathan, and A. Yesildirek, "Neural network control of robot manipulators and nonlinear systems," Neural Network Robot Control. New York, USA: Taylor & Francis, 1999, pp. 173-219.
47. F. Yuan, L. Feldkamp, G. Puskorius, and L. Davis, "A simple solution to the bioreactor benchmark problem by application of Q-learning," in Proc. World Congr. Neural Netw., Washington, DC, USA, Jul. 1995, pp. 326-331.
48. S. N. Balakrishnan and V. Biega, "Adaptive critic based neural networks for control (low order system applications)," in Proc. Amer. Control Conf., Seattle, WA, USA, Jun. 1995, pp. 335-339.
49. T. L. Chien, C. C. Chen, Y. C. Huang, and W.-J. Lin, "Stability and almost disturbance decoupling analysis of nonlinear system subject to feedback linearization and feedforward neural network controller," IEEE Trans. Neural Netw., vol. 19, no. 7, pp. 1220-1230, Jul. 2008.
50. X. Fang, H. He, Z. Ni, and Y. Tang, "Learning and control in virtual reality for machine intelligence," in Proc. 3rd Int. Conf. Intell. Control Inf. Process., Jul. 2012, pp. 63-67.
51. [Online]. Available: http://www.youtube.com/watch?v=OeZEDBz6ki0 &feature=youtu.be
52. F. Liu, J. Sun, J. Si, W. Guo, and S. Mei, "A boundedness result for the direct heuristic dynamic programming," Neural Netw., vol. 32, pp. 229-235, Aug. 2012.
53. J. Slotine and W. Li, Applied Nonlinear Control. Englewood Cliffs, NJ, USA: Prentice-Hall, 1991.
54. H. Khalil, Nonlinear Systems, 3rd ed. Englewood Cliffs, NJ, USA: Prentice-Hall, 2001.