Multiple Actor-Critic Structures for Continuous-Time Optimal Control Using Input-Output Data

IEEE Transactions on Neural Networks and Learning Systems

Volumn 26, Issue 4, 2015, Pages 851-865

  Song, Ruizhuo ^a   Lewis, Frank ^b,c   Wei, Qinglai ^d   Zhang, Hua Guang ^e   Jiang, Zhong Ping ^f   Levine, Dan ^b  

^a UNIVERSITY OF SCIENCE AND TECHNOLOGY BEIJING   (China)

^b UNIVERSITY OF TEXAS AT ARLINGTON   (United States)

^c STATE KEY LABORATORY OF SYNTHETICAL AUTOMATION FOR PROCESS INDUSTRIES   (China)

^d INSTITUTE OF AUTOMATION   (China)

^e NORTHEASTERN UNIVERSITY   (China)

^f POLYTECHNIC UNIVERSITY   (United States)

Author keywords

Actor critic; approximate dynamic programming (ADP); category; optimal control; shunting inhibitory artificial neural network (SIANN)

Indexed keywords

CONTINUOUS TIME SYSTEMS; NEURAL NETWORKS; NONLINEAR SYSTEMS; RECURRENT NEURAL NETWORKS;

ACTOR CRITIC; APPROXIMATE DYNAMIC PROGRAMMING; CATEGORY; OPTIMAL CONTROLS; SHUNTING INHIBITORY ARTIFICIAL NEURAL NETWORK (SIANN);

DYNAMIC PROGRAMMING;

ALGORITHM; ARTIFICIAL NEURAL NETWORK; COGNITION; COMPUTER SIMULATION; HUMAN; NONLINEAR SYSTEM; PHYSIOLOGY; SIGNAL PROCESSING; TIME;

ALGORITHMS; COGNITION; COMPUTER SIMULATION; HUMANS; NEURAL NETWORKS (COMPUTER); NONLINEAR DYNAMICS; PHYSIOLOGY; SIGNAL PROCESSING, COMPUTER-ASSISTED; TIME FACTORS;

EID: 85027929469     PISSN: 2162237X     EISSN: 21622388     Source Type: Journal    
DOI: 10.1109/TNNLS.2015.2399020     Document Type: Article

References (68)

1. B. Xu, C. Yang, and Z. Shi, "Reinforcement learning output feedback NN control using deterministic learning technique," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 3, pp. 635-641, Mar. 2014.
2. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, MA, USA: MIT Press, 2005.
3. P. J. Werbos, "A menu of designs for reinforcement learning over time," in Neural Networks for Control, W. T. Miller, R. S. Sutton, and P. J. Werbos, Eds. Cambridge, MA, USA: MIT Press, 1991, pp. 67-95.
4. K. Doya, K. Samejima, K.-I. Katagiri, and M. Kawato, "Multiple model-based reinforcement learning," Neural Comput., vol. 14, no. 6, pp. 1347-1369, 2002.
5. K. Doya, H. Kimura, and M. Kawato, "Neural mechanisms of learning and control," IEEE Control Syst., vol. 21, no. 4, pp. 42-54, Aug. 2001.
6. D. Liu and Q. Wei, "Policy iteration adaptive dynamic programming algorithm for discrete-time nonlinear systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 3, pp. 621-634, Mar. 2014.
7. Q. Wei and D. Liu, "Numerical adaptive learning control scheme for discrete-time non-linear systems," IET Control Theory Appl., vol. 7, no. 11, pp. 1472-1486, Jul. 2013.
8. Q. Wei and D. Liu, "An iterative ε-optimal control scheme for a class of discrete-time nonlinear systems with unfixed initial state," Neural Netw., vol. 32, pp. 236-244, Aug. 2012.
9. D. Liu and Q. Wei, "Finite-approximation-error-based optimal control approach for discrete-time nonlinear systems," IEEE Trans. Cybern., vol. 43, no. 2, pp. 779-789, Apr. 2013.
10. Y. Jiang and Z.-P. Jiang, "Robust adaptive dynamic programming and feedback stabilization of nonlinear systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 5, pp. 882-893, Jan. 2014.
11. H. Zhang, C. Qing, and Y. Luo, "Neural-network-based constrained optimal control scheme for discrete-time switched nonlinear system using dual heuristic programming," IEEE Trans. Autom. Sci. Eng., vol. 11, no. 3, pp. 839-849, Jul. 2014.
12. ∞ state feedback control of unknown affine nonlinear discrete-time systems," IEEE Trans. Cybern., vol. 44, no. 12, pp. 2706-2718, Dec. 2014.
13. K. G. Vamvoudakis and F. L. Lewis, "Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem,"Automatica, vol. 46, no. 5, pp. 878-888, May 2010.
14. T. Bian, Y. Jiang, and Z.-P. Jiang, "Adaptive dynamic programming and optimal control of nonlinear nonaffine systems," Automatica, vol. 50, no. 10, pp. 2624-2632, 2014.
15. H. Zhang, L. Cui, X. Zhang, and Y. Luo, "Data-driven robust approximate optimal tracking control for unknown general nonlinear systems using adaptive dynamic programming method," IEEE Trans. Neural Netw., vol. 22, no. 12, pp. 2226-2236, Dec. 2011.
16. F. L. Lewis, D. Vrabie, and K. G. Vamvoudakis, "Reinforcement learning and feedback control: Using natural decision methods to design optimal adaptive controllers," IEEE Control Syst., vol. 32, no. 6, pp. 76-105, Dec. 2012.
17. Q. Wei and D. Liu, "A novel iterative θ-adaptive dynamic programming for discrete-time nonlinear systems," IEEE Trans. Autom. Sci. Eng., vol. 11, no. 4, pp. 1176-1190, Oct. 2014.
18. Z.-P. Jiang and Y. Jiang, "Robust adaptive dynamic programming for linear and nonlinear systems: An overview," Eur. J. Control, vol. 19, no. 5, pp. 417-425, 2013.
19. H. Zhang, J. Zhang, G.-H. Yang, and Y. Luo, "Leader-based optimal coordination control for the consensus problem of multiagent differential games via fuzzy adaptive dynamic programming," IEEE Trans. Fuzzy Syst., vol. 23, no. 1, pp. 152-163, Feb. 2015.
20. H. Zhang, R. Song, Q. Wei, and T. Zhang, "Optimal tracking control for a class of nonlinear discrete-time systems with time delays based on heuristic dynamic programming," IEEE Trans. Neural Netw., vol. 22, no. 12, pp. 1851-1862, Dec. 2011.
21. D. Wang, D. Liu, and Q. Wei, "Finite-horizon neuro-optimal tracking control for a class of discrete-time nonlinear systems using adaptive dynamic programming approach," Neurocomputing, vol. 78, no. 1, pp. 14-22, Feb. 2012.
22. M. Abu-Khalaf and F. L. Lewis, "Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach," Automatica, vol. 41, no. 5, pp. 779-791, 2005.
23. R. Song, H. Zhang, Y. Luo, and Q. Wei, "Optimal control laws for time-delay systems with saturating actuators based on heuristic dynamic programming," Neurocomputing, vol. 73, nos. 16-18, pp. 3020-3027, Oct. 2010.
24. D. Liu, D. Wang, and X. Yang, "An iterative adaptive dynamic programming algorithm for optimal control of unknown discrete-time nonlinear systems with constrained inputs," Inf. Sci., vol. 220, pp. 331-342, Jan. 2013.
25. R. Song, W. Xiao, and Q. Wei, "Multi-objective optimal control for a class of nonlinear time-delay systems via adaptive dynamic programming,"Soft Comput., vol. 17, no. 11, pp. 2109-2115, 2013.
26. R. Song, W. Xiao, and H. Zhang, "Multi-objective optimal control for a class of unknown nonlinear systems based on finite-approximation-error ADP algorithm," Neurocomputing, vol. 119, no. 7, pp. 212-221, Nov. 2013.
27. R. Song, W. Xiao, H. Zhang, and C. Sun, "Adaptive dynamic programming for a class of complex-valued nonlinear systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 9, pp. 1733-1739, Sep. 2014.
28. Q. Wei, D. Liu, and G. Shi, "A novel dual iterative Q-learning method for optimal battery management in smart residential environments," IEEE Trans. Ind. Electron., doi: 10.1109/TIE. 2014.2361485, 2015.
29. T. Bian, Y. Jiang, and Z. P. Jiang, "Decentralized adaptive optimal control of large-scale systems with application to power systems," IEEE Trans. Ind. Electron., doi: 10.1109/TIE. 2014.2345343, 2014.
30. Y. Jiang and Z.-P. Jiang, "Robust adaptive dynamic programming with an application to power systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 7, pp. 1150-1156, Jul. 2013.
31. D. Liu, D. Wang, F.-Y. Wang, H. Li, and X. Yang, "Neural-network-based online HJB solution for optimal robust guaranteed cost control of continuous-time uncertain nonlinear systems," IEEE Trans. Cybern., vol. 44, no. 12, pp. 2834-2847, Dec. 2014.
32. D. Wang and D. Liu, "Neuro-optimal control for a class of unknown nonlinear dynamic systems using SN-DHP technique," Neurocomputing, vol. 121, pp. 218-225, Dec. 2013.
33. Q. Wei, F.-Y. Wang, D. Liu, and X. Yang, "Finite-approximation-error-based discrete-time iterative adaptive dynamic programming," IEEE Trans. Cybern., vol. 44, no. 12, pp. 2820-2833, Dec. 2014.
34. Q. Wei and D. Liu, "Data-driven neuro-optimal temperature control of water-gas shift reaction using stable iterative adaptive dynamic programming," IEEE Trans. Ind. Electron., vol. 61, no. 11, pp. 6399-6408, Nov. 2014.
35. D. Liu, H. Li, and D. Wang, "Neural-network-based zero-sum game for discrete-time nonlinear systems via iterative adaptive dynamic programming algorithm," Neurocomputing, vol. 110, pp. 92-100, Jun. 2013.
36. H. Li and D. Liu, "Optimal control for discrete-time affine non-linear systems using general value iteration," IET Control Theory Appl., vol. 6, no. 18, pp. 2725-2736, Dec. 2012.
37. D. Wang, D. Liu, Q. Wei, D. Zhao, and N. Jin, "Optimal control of unknown nonaffine nonlinear discrete-time systems based on adaptive dynamic programming," Automatica, vol. 48, no. 8, pp. 1825-1832, Aug. 2012.
38. D. Liu, H. Li, and D. Wang, "Online synchronous approximate optimal learning algorithm for multi-player non-zero-sum games with unknown dynamics," IEEE Trans. Syst., Man, Cybern., Syst., vol. 44, no. 8, pp. 1015-1027, Aug. 2014.
39. D. Liu, D. Wang, and H. Li, "Decentralized stabilization for a class of continuous-time nonlinear interconnected systems using online learning optimal control approach," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 2, pp. 418-428, Feb. 2014.
40. D. Liu, D. Wang, D. Zhao, Q. Wei, and N. Jin, "Neural-network-based optimal control for a class of unknown discrete-time nonlinear systems using globalized dual heuristic programming," IEEE Trans. Autom. Sci. Eng., vol. 9, no. 3, pp. 628-634, Jul. 2012.
41. D. S. Levine and P. A. Ramirez, Jr., "An attentional theory of emotional influences on risky decisions," Progress in Brain Research: Decision Making: Neural and Behavioural Approaches, vol. 202. Oxford, U.K.: Newnes, 2013, pp. 369-388.
42. D. S. Levine, B. Mills, and S. Estrada, "Modeling emotional influences on human decision making under risk," in Proc. IEEE Int. Joint Conf. Neural Netw., Jul./Aug. 2005, pp. 1657-1662.
43. P. J. Werbos, "Intelligence in the brain: A theory of how it works and how to build it," Neural Netw., vol. 22, no. 3, pp. 200-212, 2009.
44. P. J. Werbos, "Stable adaptive control using new critic designs," in Proc. Adaptation, Noise, Self-Organizing Syst., vol. 3728, pp. 510-579, 1999.
45. K. S. Narendra and J. Balakrishnan, "Adaptive control using multiple models," IEEE Trans. Autom. Control, vol. 42, no. 2, pp. 171-187, Feb. 1997.
46. N. Sugimoto, J. Morimoto, S.-H. Hyon, and M. Kawato, "The eMOSAIC model for humanoid robot control," Neural Netw., vols. 29-30, pp. 8-19, May 2012.
47. K. Doya, "What are the computations of the cerebellum, the basal ganglia and the cerebral cortex?" Neural Netw., vol. 12, nos. 7-8, pp. 961-974, Oct./Nov. 1999.
48. O. Hikosaka et al., "Parallel neural networks for learning sequential procedures," Trends Neurosci., vol. 22, no. 10, pp. 464-471, 1999.
49. J. M. Lee and J. H. Lee, "Approximate dynamic programming-based approaches for input-output data-driven control of nonlinear processes,"Automatica, vol. 41, no. 7, pp. 1281-1288, Jul. 2005.
50. H. Li, D. Liu, and D. Wang, "Integral reinforcement learning for linear continuous-time zero-sum games with completely unknown dynamics,"IEEE Trans. Autom. Sci. Eng., vol. 11, no. 3, pp. 706-714, Jul. 2014.
51. X. Yang, D. Liu, and Y. Huang, "Neural-network-based online optimal control for uncertain non-linear continuous-time systems with control constraints," IET Control Theory Appl., vol. 7, no. 17, pp. 2037-2047, Nov. 2013.
52. F. L. Lewis and K. G. Vamvoudakis, "Reinforcement learning for partially observable dynamic processes: Adaptive dynamic programming using measured output data," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 41, no. 1, pp. 14-25, Feb. 2011.
53. Z. Li, Z. Duan, and F. L. Lewis, "Distributed robust consensus control of multi-agent systems with heterogeneous matching uncertainties,"Automatica, vol. 50, no. 3, pp. 883-889, Mar. 2014.
54. H. Modares, F. L. Lewis, and M.-B. Naghibi-Sistani, "Integral reinforcement learning and experience replay for adaptive optimal control of partially-unknown constrained-input continuous-time systems,"Automatica, vol. 50, no. 1, pp. 193-202, Jan. 2014.
55. H. Zhang and F. L. Lewis, "Adaptive cooperative tracking control of higher-order nonlinear systems with unknown dynamics," Automatica, vol. 48, no. 7, pp. 1432-1439, Jul. 2012.
56. Q. Wei and D. Liu, "Adaptive dynamic programming for optimal tracking control of unknown nonlinear systems with application to coal gasification," IEEE Trans. Autom. Sci. Eng., vol. 11, no. 4, pp. 1020-1036, Oct. 2014.
57. D. S. Levine, "Neural dynamics of affect, gist, probability, and choice,"Cognit. Syst. Res., vols. 15-16, pp. 57-72, May/Jun. 2012.
58. P. J. Werbos, "Using ADP to understand and replicate brain intelligence: The next level design," in Proc. IEEE Int. Symp. Approx. Dyn. Program. Reinforcement Learn., Apr. 2007, pp. 209-216.
59. G. Arulampalam and A. Bouzerdoum, "A generalized feedforward neural network architecture for classification and regression," Neural Netw., vol. 16, nos. 5-6, pp. 561-568, 2003.
60. A. Bouzerdoum, "Classification and function approximation using feed-forward shunting inhibitory artificial neural networks," in Proc. IEEE-INNS-ENNS Int. Joint Conf. Neural Netw., vol. 6. 2000, pp. 613-618.
61. F. H. C. Tivive and A. Bouzerdoum, "Efficient training algorithms for a class of shunting inhibitory convolutional neural networks," IEEE Trans. Neural Netw., vol. 16, no. 3, pp. 541-556, May 2005.
62. Y. H. Kim and F. L. Lewis, "Neural network output feedback control of robot manipulators," IEEE Trans. Robot. Autom., vol. 15, no. 2, pp. 301-309, Apr. 1999.
63. H. K. Khalil, Nonlinear Systems. Englewood Cliffs, NJ, USA: Prentice-Hall, 2002.
64. F. L. Lewis, S. Jagannathan, and A. Yesildirek, Neural Network Control of Robot Manipulators and Nonlinear Systems. London, U.K.: Taylor & Francis, 1999.
65. K. Hornik, M. Stinchcombe, H. White, and P. Auer, "Degree of approximation results for feedforward networks approximating unknown mappings and their derivatives," Neural Comput., vol. 6, no. 6, pp. 1262-1275, Nov. 1994.
66. C. Yang, Z. Li, and J. Li, "Trajectory planning and optimized adaptive control for a class of wheeled inverted pendulum vehicle models," IEEE Trans. Cybern., vol. 43, no. 1, pp. 24-36, Feb. 2013.
67. C. Yang, Z. Li, R. Cui, and B. Xu, "Neural network-based motion control of an underactuated wheeled inverted pendulum model," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 1, pp. 2004-2016, Nov. 2014.
68. R. W. Beard, "Improving the closed-loop performance of nonlinear systems," Ph.D. dissertation, Dept. Elect. Eng., Rensselaer Polytech. Inst., Troy, NY, USA, 1995.