Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition

Journal of Artificial Intelligence Research

Volumn 13, Issue , 2000, Pages 227-303

  Dietterich, Thomas G ^a  

^a OREGON STATE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0002278788     PISSN: 10769757     EISSN: None     Source Type: Journal    
DOI: 10.1613/jair.639     Document Type: Article

References (33)

1. Bellman, R. E. (1957). Dynamic Programming. Princeton University Press.
2. Bertsekas, D. P., & Tsitsiklis, J. N. (1996). Neuro-Dynamic Programming. Athena Scientific, Belmont, MA.
3. Boutilier, C., Dearden, R., & Goldszmidt, M. (1995). Exploiting structure in policy construction. In Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pp. 1104-1111.
4. Currie, K., & Tate, A. (1991). O-plan: The open planning architecture. Artificial Intelligence, 52(1), 49-86.
5. Dayan, P., & Hinton, G. (1993). Feudal reinforcement learning. In Advances in Neural Information Processing Systems, 5, pp. 271-278. Morgan Kaufmann, San Francisco, CA.
6. Dean, T., & Lin, S.-H. (1995). Decomposition techniques for planning in stochastic domains. Tech. rep. CS-95-10, Department of Computer Science, Brown University, Providence, Rhode Island.
7. Dietterich, T. G. (1998). The MAXQ method for hierarchical reinforcement learning. In Fifteenth International Conference on Machine Learning, pp. 118-126. Morgan Kaufmann.
8. Fikes, R. E., Hart, P. E., & Nilsson, N. J. (1972). Learning and executing generalized robot plans. Artificial Intelligence, 3, 251-288.
9. Forgy, C. L. (1982). Rete: A fast algorithm for the many pattern/many object pattern match problem. Artificial Intelligence, 19(1), 17-37.
10. Hauskrecht, M., Meuleau, N., Kaelbling, L. P., Dean, T., & Boutilier, C. (1998). Hierarchical solution of Markov decision processes using macro-actions. In Proceedings of the Fourteenth Annual Conference on Uncertainty in Artificial Intelligence (UAI-98), pp. 220-229 San Francisco, CA. Morgan Kaufmann Publishers.
11. Howard, R. A. (1960). Dynamic Programming and Markov Processes. MIT Press, Cambridge, MA.
12. Jaakkola, T., Jordan, M. I., & Singh, S. P. (1994). On the convergence of stochastic iterative dynamic programming algorithms. Neural Computation, 6(6), 1185-1201.
13. Kaelbling, L. P. (1993). Hierarchical reinforcement learning: Preliminary results. In Proceedings of the Tenth International Conference on Machine Learning, pp. 167-173 San Francisco, CA. Morgan Kaufmann.
14. Kalmár, Z., Szepesvári, C., & Lörincz, A. (1998). Module based reinforcement learning for a real robot. Machine Learning, 31, 55-85.
15. Knoblock, C. A. (1990). Learning abstraction hierarchies for problem solving. In Proceedings of the Eighth National Conference on Artificial Intelligence, pp. 923-928 Boston, MA. AAAI Press.
16. Korf, R. E. (1985). Macro-operators: A weak method for learning. Artificial Intelligence, 26(1), 35-77.
17. Lin, L.-J. (1993). Reinforcement learning for robots using neural networks. Ph.D. thesis, Carnegie Mellon University, Department of Computer Science, Pittsburgh, PA.
18. Moore, A. W., Baird, L., & Kaelbling, L. P. (1999). Multi-value-functions: Efficient automatic action hierarchies for multiple goal MDPs. In Proceedings of the International Joint Conference on Artificial Intelligence, pp. 1316-1323 San Francisco. Morgan Kaufmann.
19. Parr, R. (1998a). Flexible decomposition algorithms for weakly coupled Markov decision problems. In Proceedings of the Fourteenth Annual Conference on Uncertainty in Artificial Intelligence (UAI-98), pp. 422-430 San Francisco, CA. Morgan Kaufmann Publishers.
20. Parr, R. (1998b). Hierarchical control and learning for Markov decision processes. Ph.D. thesis, University of California, Berkeley, California.
21. Parr, R., & Russell, S. (1998). Reinforcement learning with hierarchies of machines. In Advances in Neural Information Processing Systems, Vol. 10, pp. 1043-1049 Cambridge, MA. MIT Press.
22. Pearl, J. (1988). Probabilistic Inference in Intelligent Systems. Networks of Plausible Inference. Morgan Kaufmann, San Mateo, CA.
23. Rummery, G. A., & Niranjan, M. (1994). Online Q-learning using connectionist systems. Tech. rep. CUED/FINFENG/TR 166, Cambridge University Engineering Department, Cambridge, England.
24. Sacerdoti, E. D. (1974). Planning in a hierarchy of abstraction spaces. Artificial Intelligence, 5(2), 115-135.
25. Singh, S., Jaakkola, T., Littman, M. L., & Szepesvári, C. (1998). Convergence results for single-step on-policy reinforcement-learning algorithms. Tech. rep., University of Colorado, Department of Computer Science, Boulder, CO. To appear in Machine Learning.
26. Singh, S. P. (1992). Transfer of learning by composing solutions of elemental sequential tasks. Machine Learning, 8, 323-339.
27. Sutton, R. S., Singh, S., Precup, D., & Ravindran, B. (1999). Improved switching among temporally abstract actions. In Advances in Neural Information Processing Systems, Vol. 11, pp. 1066-1072. MIT Press.
28. Sutton, R., & Barto, A. G. (1998). Introduction to Reinforcement Learning. MIT Press, Cambridge, MA.
29. Sutton, R. S., Precup, D., & Singh, S. (1998). Between MDPs and Semi-MDPs: Learning, planning, and representing knowledge at multiple temporal scales. Tech. rep., University of Massachusetts, Department of Computer and Information Sciences, Amherst, MA. To appear in Artificial Intelligence.
30. Tadepalli, P., & Dietterich, T. G. (1997). Hierarchical explanation-based reinforcement learning. In Proceedings of the Fourteenth International Conference on Machine Learning, pp. 358-366 San Francisco, CA. Morgan Kaufmann.
31. Tambe, M., & Rosenbloom, P. S. (1994). Investigating production system representations for non-combinatorial match. Artificial Intelligence, 68(1), 155-199.
32. Watkins, C. J. C. H. (1989). Learning from Delayed Rewards. Ph.D. thesis, King's College, Oxford. (To be reprinted by MIT Press.).
33. Watkins, C. J., & Dayan, P. (1992). Technical note Q-Learning. Machine Learning, 8, 279.