Hierarchical approximate policy iteration with binary-tree state space decomposition

IEEE Transactions on Neural Networks

Volumn 22, Issue 12 PART 1, 2011, Pages 1863-1877

  Xu, Xin ^a   Liu, Chunming ^a   Yang, Simon X ^b   Hu, Dewen ^a  

^a NATIONAL UNIVERSITY OF DEFENSE TECHNOLOGY   (China)

^b UNIVERSITY OF GUELPH   (Canada)

Author keywords

Adaptive dynamic programming; approximate policy iteration; binary tree; hierarchical reinforcement learning; Markov decision processes; time optimal control

Indexed keywords

ADAPTIVE DYNAMIC PROGRAMMING; APPROXIMATE POLICY ITERATION; HIERARCHICAL REINFORCEMENT LEARNING; MARKOV DECISION PROCESSES; TIME OPTIMAL CONTROL;

APPLICATION PROGRAMMING INTERFACES (API); DISCRETE TIME CONTROL SYSTEMS; DYNAMIC PROGRAMMING; FORESTRY; MARKOV PROCESSES; OPTIMIZATION; REINFORCEMENT; REINFORCEMENT LEARNING; TREES (MATHEMATICS);

LEARNING ALGORITHMS;

ALGORITHMS; AUTOMATIC CONTROL; COMPUTER PROGRAMING; DECISION MAKING; FORESTRY; OPTIMIZATION;

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; AUTOMATED PATTERN RECOGNITION; COMPUTER SIMULATION; METHODOLOGY; THEORETICAL MODEL;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; MODELS, THEORETICAL; PATTERN RECOGNITION, AUTOMATED;

EID: 83855164075     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2011.2168422     Document Type: Article

References (46)

1. R. Sutton and A. Barto, Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press, 1998.
2. F. Y. Wang, H. Zhang, and D. Liu, "Adaptive dynamic programming: An introduction," IEEE Comput. Intell. Mag., vol. 4, no. 2, pp. 39-47, May 2009.
3. G. Tesauro, "TD-gammon a self-teaching backgammon program achieves master-level play," Neural Comput., vol. 6, no. 2, pp. 215-219, Mar. 1994.
4. W. Zhang and T. Dietterich, "A reinforcement learning approach to job-shop scheduling," in Proc. 14th Int. Joint Conf. Art. Intell., San Francisco, C.A., 1995, pp. 1114-1120.
5. R. H. Crites and A. G. Barto, "Elevator group control using multiple reinforcement learning agents," Mach. Learn., vol. 33, nos. 2-3, pp. 235-262, 1998. (Pubitemid 128522644)
6. A. Y. Ng, H. J. Kim, M. Jordan, and S. Sastry, "Autonomous helicopter flight via reinforcement learning," in Advances in Neural Information Processing Systems 16. Cambridge, MA: MIT Press, 2004.
7. R. E. Bellman, Dynamic Programming. Princeton, N.J: Princeton Univ. Press, 1957.
8. R. Sutton, "Generalization in reinforcement learning: Successful examples using sparse coarse coding," in Advances in Neural Information Processing Systems 8. Cambridge, MA: MIT Press, 1996, pp. 1038-1044.
9. X. Xu and H. G. He, "Residual-gradient-based neural reinforcement learning for the optimal control of an acrobat," in Proc. IEEE Int. Symp. Intell. Control, Vancouver, Canada, 2002, pp. 758-763.
10. J. Baxter and P. L. Bartlett, "Infinite-horizon policy-gradient estimation," J. Art. Intell. Res., vol. 15, no. 1, pp. 319-350, Jul. 2001.
11. V. R. Konda and J. N. Tsitsiklis, "Actor-critic algorithm," in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 2000.
12. X. Xu, H. G. He, and D. W. Hu, "Efficient reinforcement learning using recursive least-squares methods," J. Art. Intell. Res., vol. 16, no. 1, pp. 259-292, Jan. 2002. (Pubitemid 43057174)
13. J. Fu, H. He, and X. Zhou, "Adaptive learning and control for MIMO system based on adaptive dynamic programming," IEEE Trans. Neural Netw., vol. 22, no. 7, pp. 1133-1148, Jul. 2011.
14. ε-error bound," IEEE Trans. Neural Netw., vol. 22, no. 1, pp. 24-36, Jan. 2011.
15. H. G. Zhang, Y. H. Luo, and D. R. Liu, "Neural-network-based near-optimal control for a class of discrete-time affine nonlinear systems with control constraints," IEEE Trans. Neural Netw., vol. 20, no. 9, pp. 1490-1503, Sep. 2009.
16. M. G. Lagoudakis and R. Parr, "Least-squares policy iteration," J. Mach. Learn. Res., vol. 4, pp. 1107-1149, Dec. 2003.
17. X. Xu, D. W. Hu, and X. C. Lu, "Kernel based least-squares policy iteration for reinforcement learning," IEEE Trans. Neural Netw., vol. 18, no. 4, pp. 973-992, Jul. 2007. (Pubitemid 47098876)
18. A. G. Barto and S. Mahadevan, "Recent advances in hierarchical reinforcement learning," Discrete Event Dynamic Syst.-Theory Applicat., vol. 13, nos. 1-2, pp. 41-77, Jan.-Apr. 2003.
19. M. Ghavamzadeh and S. Mahadevan, "Hierarchical average reward reinforcement learning," J. Mach. Learn. Res., vol. 8, pp. 2629-2669, Nov. 2007. (Pubitemid 350241862)
20. T. G. Dietterich, "Hierarchical reinforcement learning with the Max-Q value function decomposition," J. Art. Intell. Res., vol. 13, no. 1, pp. 227-303, Aug. 2000. (Pubitemid 33682087)
21. R. Sutton, D. Precup, and S. Singh, "Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning," Art. Intell., vol. 112, nos. 1-2, pp. 181-211, Aug. 1999. (Pubitemid 32079890)
22. T. G. Dietterich, "State abstraction in MAXQ hierarchical reinforcement learning," in Proc. Adv. Neural Inf. Process. Syst., 2000, pp. 994-1000.
23. D. Andre and S. J. Russell, "State abstraction for programmable reinforcement learning agents," in Proc. 18th Nat. Conf. Art. Intell., 2002, pp. 119-125.
24. B. Hengst, "Safe state abstraction and reusable continuing subtasks in hierarchical reinforcement learning," in Proc. AI: Adv. Art. Intell. Lecture Notes Comput. Sci., 2007, pp. 58-67.
25. R. Parr and S. Russell, "Reinforcement learning with hierarchies of machines," in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 1998.
26. Q. Y. Hu and W. Y. Yue, Markov Decision Processes with Their Applications. New York: Springer-Verlag, 2008.
27. J. Boyan, "Technical update: Least-squares temporal difference learning," Mach. Learn., vol. 49, nos. 2-3, pp. 233-246, 2002. (Pubitemid 34325688)
28. V. Vapnik, Statistical Learning Theory. New York: Wiley, 1998.
29. Y. Engel, S. Mannor, and R. Meir, "The kernel recursive least-squares algorithm," IEEE Trans. Signal Process., vol. 52, no. 8, pp. 2275-2285, Aug. 2004.
30. R. Munos, "Error bounds for approximate policy iteration," in Proc. 20th Annu. Int. Conf. Mach. Learn., 2003, p. 560.
31. C. Dimitrakakis and M. G. Lagoudakis, "Algorithms and bounds for rollout sampling approximate policy iteration," in Proc. 8th Eur. Workshop, Recent Adv. Reinforce. Learn., Villeneuve d'Ascq, France, LNAI 5323. Jun.-Jul. 2008, pp. 27-40.
32. R. Munos and C. Szepesvári, "Finite-time bounds for fitted value iteration," J. Mach. Learn. Res., vol. 9, pp. 815-857, May 2008.
33. B. H. Li and J. Si, "Approximate robust policy iteration using multilayer perceptron neural networks for discounted infinite-horizon Markov decision processes with uncertain correlated transition matrices," IEEE Trans. Neural Netw., vol. 21, no. 8, pp. 1270-1280, Aug. 2010.
34. J. Seiffertt and D. C. Wunsch, "Backpropagation and ordered derivatives in the time scales calculus," IEEE Trans. Neural Netw., vol. 21, no. 8, pp. 1262-1269, Aug. 2010.
35. A. D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming, Belmont, MA: Athena Scientific, 1996.
36. J. N. Tsitsiklis and B. V. Roy, "An analysis of temporal-difference learning with function approximation," IEEE Trans. Automat. Control, vol. 42, no. 5, pp. 674-690, May 1997. (Pubitemid 127760263)
37. A. Antos, C. Szepesvari, and R. Munos, "Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path," Mach. Learn., vol. 71, no. 1, pp. 89-129, 2008.
38. A. M. Farahmand, R. Munos, and C. Szepesvári. "Error propagation for approximate policy and value iteration," in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 2010.
39. A. McCallum, "Reinforcement Learning with Selective Perception and Hidden State," Ph.D. thesis, Comput. Sci. Dept., Univ. Rochester., Rochester, NY, 1995.
40. T. B. U. William and M. M. Veloso. "Tree based discretization for continuous state space reinforcement learning," in Proc. AAAI-98 15th Nat./10th Conf. Art. Intell./Innovative Applicat. Art. Intell., Madison, WI, Jul. 1998, pp. 769-774.
41. D. Ernst, P. Geurts, and L. Wehenkel, "Tree-based batch mode reinforcement learning," J. Mach. Learn. Res., vol. 6, pp. 503-556, Apr. 2005.
42. A. Jonsson and A. G. Barto, "Automated state abstraction for options using the U-tree algorithm," in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 2000.
43. M. A. Wiering and H. V. Hasselt, "Ensemble algorithms in reinforcement learning," IEEE Trans. Syst. Man Cybern. Part B: Cybern., vol. 38, no. 4, pp. 930-936, Aug. 2008.
44. H. Alexander and U. Steffen, "Ensembles of neural networks for robust reinforcement learning," in Proc. 19th Int. Conf. Mach. Learn. Applicat., Washington D.C., 2010, pp. 401-406.
45. K. J. Nicholas and S. Peter, Model-Based Exploration in Continuous State Spaces, I. Miguel and W. Ruml, Eds. New York: Springer-Verlag, 2007, pp. 258-272.
46. L. H. Li, M. L. Littman, and C. R. Mansley, "Online exploration in least-squares policy iteration," in Proc. AAMAS-09 8th Int. Conf. Autonomous Agents Multiagent Syst., Budapest, Hungary, May 2009, pp. 733-739.