Goal representation heuristic dynamic programming on maze navigation

IEEE Transactions on Neural Networks and Learning Systems

Volumn 24, Issue 12, 2013, Pages 2038-2050

  Ni, Zhen ^a   He, Haibo ^a   Wen, Jinyu ^b   Xu, Xin ^c  

^a University of Rhode Island   (United States)

^b HUAZHONG UNIVERSITY OF SCIENCE AND TECHNOLOGY   (China)

^c NATIONAL UNIVERSITY OF DEFENSE TECHNOLOGY   (China)

Author keywords

Adaptive dynamic programming; goal representation heuristic dynamic programming; Markov decision process; maze navigation path planning; reinforcement learning

Indexed keywords

ADAPTIVE DYNAMIC PROGRAMMING; CHARACTERISTICS ANALYSIS; HEURISTIC DYNAMIC PROGRAMMING; LEARNING PERFORMANCE; MARKOV DECISION PROCESSES; REINFORCEMENT SIGNAL; THEORETICAL GUARANTEES; VALUE FUNCTION APPROXIMATION;

BENCHMARKING; LEARNING ALGORITHMS; MARKOV PROCESSES; NAVIGATION; REINFORCEMENT LEARNING;

DYNAMIC PROGRAMMING;

EID: 84887990637     PISSN: 2162237X     EISSN: 21622388     Source Type: Journal    
DOI: 10.1109/TNNLS.2013.2271454     Document Type: Article

References (49)

1. A. G. Barto, R. S. Sutton, and C. J. C. H. Watkins, "Learning and sequential decision making," in Learning Computational Neuroscience. Cambridge, MA, USA: MIT Press, 1989, pp. 539-602.
2. R. Sutton and A. Barto, Reinforcement Learning: An Introduction. Cambridge, U.K.: Cambridge Univ. Press, 1998.
3. R. Bellman, Dynamic Programming. Princeton, NJ, USA: Princeton Univ. Press, 1957.
4. F. L. Lewis, D. Vrabie, and V. L. Syrmos, Optimal Control. New York, NY, USA: Wiley, 2012.
5. D. P. Bertsekas, Dynamic Programming and Optimal Control, vol. 1, Belmont, MA, USA: Athena Scientific, 1995.
6. S. Singh, A. Barto, R. Grupen, and C. Connolly, "Robust reinforcement learning in motion planning," in Proc. Adv. Neural Inf. Process. Syst., 1994, pp. 655-662.
7. R. S. Sutton, "Integrated architectures for learning, planning, and reacting based on approximating dynamic programming," in Proc. 7th Int. Conf. Mach. Learn., 1990, pp. 216-224.
8. R. Sutton, "Learning to predict by the methods of temporal differences," Mach. Learn., vol. 3, no. 1, pp. 9-44, 1988.
9. R. Sutton, Temporal credit assignment in reinforcement learning, Ph.D. dissertation, Dept. Comput. Sci., Univ. Massachusetts Amherst, Amherst, MA, USA, Jan. 1984.
10. P. Werbos, "Reinforcement learning and approximate dynamic programming (RLADP) - foundations, common misconceptions and challenges ahead," in Reinforcement Learning and Approximate Dynamic Programming for Feedback Control. New York, USA: Wiley, 2013, pp. 3-30.
11. P. J. Werbos, "ADP: The key direction for future research in intelligent control and understanding brain intelligence," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 898-900, Aug. 2008.
12. P. J. Werbos, "Intelligence in the brain: A theory of how it works and how to build it," Neural Netw., vol. 22, no. 3, pp. 200-212, 2009.
13. P. J. Werbos, Handbook of Itelligent Control. New York, NY, USA: Van Nostrand, 1992.
14. P. J. Werbos, "Consistency of HDP applied to a simple reinforcement learning problem," Neural Netw., vol. 3, no. 2, pp. 179-189, 1990.
15. J. Si and Y.-T. Wang, "Online learning control by association and reinforcement," IEEE Trans. Neural Netw., vol. 12, no. 2, pp. 264-276, Mar. 2001. (Pubitemid 32371483)
16. R. Enns and J. Si, "Helicopter trimming and tracking control using direct neural dynamic programming," IEEE Trans. Neural Netw., vol. 14, no. 4, pp. 929-939, Jul. 2003.
17. L. Yang, J. Si, K. S. Tsakalis, and A. A. Rodriguez, "Direct heuristic dynamic programming for nonlinear tracking conrol with filtered tracking error," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 39, no. 6, pp. 1617-1622, Dec. 2009.
18. D. Liu and Q. Wei, "Finite-approximation-error-based optimal control approach for discrete-time nonlinear systems," IEEE Trans. Cybern., vol. 43, no. 2, pp. 779-789, Apr. 2013.
19. D. Prokhorov and D. Wunsch, "Adaptive critic designs," IEEE Trans. Neural Netw., vol. 8, no. 5, pp. 997-1007, Sep. 1997. (Pubitemid 127763331)
20. D. V. Prokhorov, "Adaptive critic designs and their applications," Ph.D. Dissertation, Dept. Electr. Eng., Texas Tech. Univ., Lubbock, TX, USA, Oct. 1997.
21. D. Liu, D. Wang, D. Zhao, Q. Wei, and N. Jin, "Neural-network-based optimal control for a class of unknown discrete-time nonlinear systems using globalized dual heuristic programming," IEEE Trans. Autom. Sci. Eng., vol. 9, no. 3, pp. 628-634, Jul. 2012.
22. D. Wang, D. Liu, Q. Wei, D. Zhao, and N. Jin, "Optimal control of unknown nonaffine nonlinear discrete-time systems based on adaptive dynamic programming," Automatica, vol. 48, pp. 1825-1832, Jul. 2012.
23. D. Liu and D. Wang, "Optimal control of unkonwn nonlinear discretetime systems using the iterative globalized dual heuristic programming algorithm," in Reinforcement Learning and Approximate Dynamic Programming for Feedback Control. New York, NY, USA: Wiley, 2013, pp. 52-74.
24. J. Fu, H. He, and X. Zhou, "Adaptive learning and control for MIMO system based on adaptive dynamic programming," IEEE Trans. Neural Netw., vol. 22, no. 7, pp. 1133-1148, Jun. 2011.
25. Z. Ni, H. He, D. V. Prokhorov, and J. Fu, "An online actor-critic learning approach with Levenberg-Marquardt algorithm," in Proc. IJCNN, 2011, pp. 2333-2340.
26. J. Fu, H. He, and Z. Ni, "Adaptive dynamic programming with balanced weights seeking strategy," in Proc. IEEE Symp. ADPRL, Apr. 2011, pp. 210-217.
27. H. He, Z. Ni, and J. Fu, "A three-network architecture for on-line learning and optimization based on adaptive dynamic programming," Neurocomputing, vol. 78, no. 1, pp. 3-13, 2012.
28. Z. Ni, H. He, and J. Wen, "Adaptive learning in tracking control based on the dual critic network design," IEEE Trans. Neural Netw. Learn. Syst., vol. 6, no. 24, pp. 913-928, Jun. 2013.
29. H. He, Self-Adaptive Systems for Machine Intelligence. New York, NY, USA: Wiley, 2011.
30. Z. Ni, H. He, D. Zhao, and D. Prokhorov, "Reinforcement learning control based on multi-goal representation using hierarchical heuristic dynamic programming," in Proc. IJCNN, Jun. 2012, pp. 1-8.
31. H. He, Z. Ni, and D. Zhao, "Learning and optimization in hierarchical adaptive critic design," in Reinforcement Learning and Approximate Dynamic Programming for Feedback Control. Piscataway, NJ, USA: IEEE Press, 2013, pp. 78-95.
32. H. He, Z. Ni, and D. Zhao, "Data-driven learning and control with multiple critic networks," in Proc. 10th WCICA, Jul. 2012, pp. 523-527.
33. Z. Ni, X. Fang, H. He, D. Zhao, and X. Xu, Real-time tracking control on adaptive critic design with uniformly ultimately bounded condition, in Proc. IEEE Symp. ADPRL, Apr. 2013.
34. X. Fang, H. He, Z. Ni, and Y. Tang, Learning and control in virtual reality for machine intelligence, in Proc. 3rd ICICIP, pp. 63-67, Jul. 2012.
35. Virtual Reality (VR) Platform for Adaptive Learning and Control Based on Adaptive Dynamic Programming [Online]. Available: http://www.youtube.com/watch- v=OeZEDBz6ki0&feature=youtu.be
36. X. Pang and P. J. Werbos, "Neural network design for J function approximation in dynamic programming," Math. Model. Sci. Comput., vol. 5, nos. 2-3, pp. 1-3, 1996.
37. D. Wunsch, "The cellular simultaneous recurrent network adaptive critic design for the generalized maze problem has a simple closed-form solution," in Proc. IEEE IJCNN, vol. 3. Jul. 2000, pp. 79-82.
38. R. Ilin, R. Kozma, and P. Werbos, "Cellular SRN trained by extended Kalman filter shows promise for ADP," in Proc. IEEE IJCNN, Jul. 2006, pp. 506-510. (Pubitemid 351369350)
39. R. Ilin, R. Kozma, and P. Werbos, "Efficient learning in cellular simultaneous recurrent neural networks - The case of maze navigation problem," in Proc. IEEE Int. Symp. ADPRL, Apr. 2007, pp. 324-329. (Pubitemid 47431403)
40. R. Ilin, R. Kozma, and P. Werbos, "Beyond feedforward models trained by backpropagation: A practical training tool for a more efficient universal approximator," IEEE Trans. Neural Netw., vol. 19, no. 6, pp. 929-937, Jun. 2008.
41. M. Wiering and H. Van Hasselt, "Two novel on-policy reinforcement learning algorithms based on TD (λ)-methods," in Proc. IEEE Int. Symp. ADPRL, Apr. 2007, pp. 280-287. (Pubitemid 47431397)
42. P. Werbos and X. Pang, "Generalized maze navigation: SRN critics solve what feedforward or Hebbian nets cannot," in Proc. IEEE Int. Conf. Syst., Man, Cybern., vol. 3. Oct. 1996, pp. 1764-1769.
43. T. M. Mitchell, Machine Learning. New York, NY, USA: McGraw-Hill, 1997.
44. F. Liu, J. Sun, J. Si, W. Guo, and S. Mei, "A boundedness result for the direct heuristic dynamic programming," Neural Netw., vol. 32, pp. 229-235, Aug. 2012.
45. P. He and S. Jagannathan, "Reinforcement learning neural-networkbased controller for nonlinear discrete-time systems with input contraints," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 37, no. 2, pp. 425-436, Apr. 2007. (Pubitemid 46523230)
46. P. Dayan and T. J. Sejnowski, "TD (λ) converges with probability 1," Mach. Learn., vol. 14, no. 3, pp. 295-301, 1994.
47. P. Dayan, "The convergence of TD (λ) for general λ," Mach. Learn., vol. 8, no. 34, pp. 341-362, 1992.
48. H. Robbins and S. Monro, "A stochastic approximation method," Anna. Math. Stat., vol. 10, no. 1, pp. 400-407, 1951.
49. H. J. Kushner and G. G. Yin, Stochastic Approximation Algorithms and Applications. New York, NY, USA: Springer-Verlag, 1997.