Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning

Artificial Intelligence

Volumn 112, Issue 1, 1999, Pages 181-211

  Sutton, Richard S ^a   Precup, Doina ^b   Singh, Satinder ^a  

^a AT AND T LABS RESEARCH   (United States)

^b Biologically Inspired Neural and Dynamical Systems Laboratory   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

APPROXIMATION THEORY; COMPUTER SYSTEMS PROGRAMMING; DECISION THEORY; KNOWLEDGE REPRESENTATION; MARKOV PROCESSES; OPTIMIZATION; THEOREM PROVING;

REINFORCEMENT LEARNING; SEMI-MARKOV DECISION PROCESS (SMDP);

LEARNING SYSTEMS;

EID: 0033170372     PISSN: 00043702     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0004-3702(99)00052-1     Document Type: Article

References (82)

1. E.G. Araujo, R.A. Grupen, Learning control composition in a complex environment, in: Proc. 4th International Conference on Simulation of Adaptive Behavior, 1996, pp. 333-342.
2. M. Asada, S. Noda, S. Tawaratsumida, K. Hosada, Purposive behavior acquisition for a real robot by vision-based reinforcement learning, Machine Learning 23 (1996) 279-303.
3. A.G. Barto, S.J. Bradtke, S.P. Singh, Learning to act using real-time dynamic programming, Artificial Intelligence 72 (1995) 81-138.
4. C. Boutilier, R.I. Brafman, C. Geib, Prioritized goal decomposition of markov decision processes: Toward a synthesis of classical and decision theoretic planning, in: Proc. IJCAI-97, Nagoya, Japan, 1997, pp. 1162-1165.
5. S.J. Bradtke, M.O. Duff, Reinforcement learning methods for continuous-time markov decision problems, in: Advances in Neural Information Processing Systems 7, MIT Press, Cambridge, MA, 1995, pp. 393-400.
6. R.I. Brafman, M. Tennenholtz, Modeling agents as qualitative decision makers, Artificial Intelligence 94 (1) (1997) 217-268.
7. R.W. Brockett, Hybrid models for motion control systems, in: Essays in Control: Perspectives in the Theory and its Applications, Birkhäuser, Boston, MA, 1993, pp. 29-53.
8. L. Chrisman, Reasoning about probabilistic actions at multiple levels of granularity, in: Proc. AAAI Spring Symposium: Decision-Theoretic Planning, Stanford University, 1994.
9. M. Colombetti, M. Dorigo, G. Borghi, Behavior analysis and training: A methodology for behavior engineering, IEEE Trans. Systems Man Cybernet. Part B 26 (3) (1996) 365-380.
10. R.H. Crites, A.G. Barto, Improving elevator performance using reinforcement learning, in: Advances in Neural Information Processing Systems 8, MIT Press, Cambridge, MA, 1996, pp. 1017-1023.
11. P. Dayan, Improving generalization for temporal difference learning: The successor representation, Neural Computation 5 (1993) 613-624.
12. P. Dayan, G.E. Hinton, Feudal reinforcement learning, in: Advances in Neural Information Processing Systems 5, Morgan Kaufmann, San Mateo, CA, 1993, pp. 271-278.
13. T. Dean, L.P. Kaelbling, J. Kirman, A. Nicholson, Planning under time constraints in stochastic domains, Artificial Intelligence 76 (1-2) (1995) 35-74.
14. T. Dean, S.-H. Lin, Decomposition techniques for planning in stochastic domains, in: Proc. IJCAI-95, Montreal, Quebec, Morgan Kaufmann, San Mateo, CA, 1995, pp. 1121-1127. See also Technical Report CS-95-10, Brown University, Department of Computer Science, 1995.
15. G.F. DeJong, Learning to plan in continuous domains, Artificial Intelligence 65 (1994) 71-141.
16. T.G. Dietterich, The MAXQ method for hierarchical reinforcement learning, in: Machine Learning: Proc. 15th International Conference, Morgan Kaufmann, San Mateo, CA, 1998, pp. 118-126.
17. M. Dorigo, M. Colombetti, Robot shaping: Developing autonomous agents through learning, Artificial Intelligence 71 (1994) 321-370.
18. G.L. Drescher, Made Up Minds: A Constructivist Approach to Artificial Intelligence, MIT Press, Cambridge, MA, 1991.
19. C. Drummond, Composing functions to speed up reinforcement learning in a changing world, in: Proc. 10th European Conference on Machine Learning, Springer, Berlin, 1998.
20. O. Etzioni, Why PRODIGY/EBL works, in: Proc. AAAI-90, Boston, MA, MIT Press, Cambridge, MA, 1990, pp. 916-922.
21. R.E. Fikes, P.E. Hart, N.J. Nilsson, Learning and executing generalized robot plans, Artificial Intelligence 3 (1972) 251-288.
22. H. Geffner, B. Bonet, High-level planning and control with incomplete information using POMDPs, in: Proc. AIPS-98 Workshop on Integrating Planning, Scheduling and Execution in Dynamic and Uncertain Environments, 1998.
23. J. Gratch, G. DeJong, A statistical approach to adaptive problem solving, Artificial Intelligence 88 (1-2) (1996) 101-161.
24. R. Greiner, I. Jurisica, A statistical approach to solving the EBL utility problem, in: Proc. AAAI-92, San Jose, CA, 1992, pp. 241-248.
25. R.L. Grossman, A. Nerode, A.P. Ravn, H. Rischel, Hybrid Systems, Springer, New York, 1993.
26. M. Hauskrecht, N. Meuleau, C. Boutilier, L.P. Kaelbling, T. Dean, Hierarchical solution of Markov decision processes using macro-actions, in: Uncertainty in Artificial Intelligence: Proc. 14th Conference, 1998, pp. 220-229.
27. R. Howard, Dynamic Programming and Markov Processes, MIT Press, Cambridge, MA, 1960.
28. M. Huber, R.A. Grupen, A feedback control structure for on-line learning tasks, Robotics and Autonomous Systems 22 (3-4) (1997) 303-315.
29. G.A. Iba, A heuristic approach to the discovery of macro-operators, Machine Learning 3 (1989) 285-317.
30. T. Jaakkola, M.I. Jordan, S. Singh, On the convergence of stochastic iterative dynamic programming algorithms, Neural Computation 6 (6) (1994) 1185-1201.
31. L.P. Kaelbling, Hierarchical learning in stochastic domains: Preliminary results, in: Proc. 10th International Conference on Machine Learning, Morgan Kaufmann, San Mateo, CA, 1993, pp. 167-173.
32. Zs. Kalmár, Cs. Szepesvári, A. Lörincz, Module based reinforcement learning: Experiments with a real robot, Machine Learning 31 (1998) 55-85 and Autonomous Robots 5 (1998) 273-295 (special joint issue).
33. J. de Kleer, J.S. Brown, A qualitative physics based on confluences, Artificial Intelligence 24 (1-3) (1984) 7-83.
34. R.E. Korf, Learning to Solve Problems by Searching for Macro-Operators, Pitman Publishers, Boston, MA, 1985.
35. J.R. Koza, J.P. Rice, Automatic programming of robots using genetic programming, in: Proc. AAAI-92, San Jose, CA, 1992, pp. 194-201.
36. B.J. Kuipers, Commonsense knowledge of space: Learning from experience, in: Proc. IJCAI-79, Tokyo, Japan, 1979, pp. 499-501.
37. J.E. Laird, P.S. Rosenbloom, A. Newell, Chunking in SOAR: The anatomy of a general learning mechanism, Machine Learning 1 (1986) 11-46.
38. L.-J. Lin, Reinforcement learning for robots using neural networks, Ph.D. Thesis, Carnegie Mellon University, Technical Report CMU-CS-93-103, 1993.
39. P. Maes, R. Brooks, Learning to coordinate behaviors, in: Proc. AAAI-90, Boston, MA, 1990, pp. 796-802.
40. S. Mahadevan, J. Connell, Automatic programming of behavior-based robots using reinforcement learning, Artificial Intelligence 55 (2-3) (1992) 311-365.
41. S. Mahadevan, N. Marchalleck, T. Das, A. Gosavi, Self-improving factory simulation using continuous-time average-reward reinforcement learning, in: Proc. 14th International Conference on Machine Learning, 1997, pp. 202-210.
42. P. Marbach, O. Mihatsch, M. Schulte, J.N. Tsitsiklis, Reinforcement learning for call admission control in routing in integrated service networks, in: Advances in Neural Information Processing Systems 10, Morgan Kaufmann, San Mateo, CA, 1998, pp. 922-928.
43. M.J. Mataric, Behavior-based control: Examples from navigation, learning, and group behavior, J. Experiment. Theoret. Artificial Intelligence 9 (2-3) (1997) 323-336.
44. A. McGovern, R.S. Sutton, Macro-actions in reinforcement learning: An empirical analysis, Technical Report 98-70, University of Massachusetts, Department of Computer Science, 1998.
45. N. Meuleau, M. Hauskrecht, K.-E. Kim, L. Peshkin, L.P. Kaelbling, T. Dean, C. Boutilier, Solving very large weakly coupled Markov decision processes, in: Proc. AAAI-98, Madison, WI, 1998, pp. 165-172.
46. S. Minton, Learning Search Control Knowledge: An Explanation-Based Approach, Kluwer Academic, Dordrecht, 1988.
47. S. Minton, Quantitative results concerning the utilty of explanation-based learning, Artificial Intelligence 42 (2-3) (1990) 363-391.
48. A.W. Moore, The parti-game algorithm for variable resolution reinforcement learning in multidimensional spaces, in: Advances in Neural Information Processing Systems 6, MIT Press, Cambridge, MA, 1994, pp. 711-718.
49. A. Newell, H.A. Simon, Human Problem Solving, Prentice-Hall, Englewood Cliffs, NJ, 1972.
50. J. Nie, S. Haykin, A Q-learning based dynamic channel assignment technique for mobile communication systems, IEEE Transactions on Vehicular Technology, to appear.
51. N. Nilsson, Teleo-reactive programs for agent control, J. Artificial Intelligence Res. 1 (1994) 139-158.
52. R. Parr, Hierarchical Control and Learning for Markov Decision Processes, Ph.D. Thesis, University of California at Berkeley, 1998.
53. R. Parr, S. Russell, Reinforcement learning with hierarchies of machines, in: Advances in Neural Information Procesing Systems 10, MIT Press, Cambridge, MA, 1998, pp. 1043-1049.
54. D. Precup, R.S. Sutton, Multi-time models for reinforcement learning, in: Proc. ICML'97 Workshop on Modeling in Reinforcement Learning, 1997.
55. D. Precup, R.S. Sutton, Multi-time models for temporally abstract planning, in: Advances in Neural Information Processing Systems 10, MIT Press, Cambridge, MA, 1998, pp. 1050-1056.
56. D. Precup, R.S. Sutton, S.P. Singh, Planning with closed-loop macro actions, in: Working Notes 1997 AAAI Fall Symposium on Model-directed Autonomous Systems, 1997, pp. 70-76.
57. D. Precup, R.S. Sutton, S.P. Singh, Theoretical results on reinforcement learning with temporally abstract options, in: Proc. 10th European Conference on Machine Learning, Springer, Berlin, 1998.
58. M.L. Puterman, Markov Decision Problems, Wiley, New York, 1994
59. M. Ring, Incremental development of complex behaviors through automatic construction of sensory-motor hierarchies, in: Proc. 8th International Conference on Machine Learning, Morgan Kaufmann, San Mateo, CA, 1991, pp. 343-347.
60. E.D. Sacerdoti, Planning in ahierarchy of abstraction spaces, Artificial Intelligence 5 (1974) 115-135.
61. S. Sastry, Algorithms for design of hybrid systems, in: Proc. International Conference of Information Sciences, 1997.
62. A.C.C. Say, S. Kuru, Qualitative system identification: Deriving structure from behavior, Artificial Intelligence 83 (1) (1996) 75-141.
63. J. Schmidhuber, Neural Sequence Chunkers, Technische Universität München, TR FKI-148-91, 1991.
64. R. Simmons, S. Koenig, Probabilistic robot navigation in partially observable environments, in: Proc. IJCAI-95, Montreal, Quebec, Morgan Kaufmann, San Mateo, CA, 1995, pp. 1080-1087.
65. S.P. Singh, Reinforcement learning with ahierarchy of abstract models, in: Proc. AAAI-92, San Jose, CA, MIT/AAAI Press, Cambridge, MA, 1992, pp. 202-207.
66. S.P. Singh, Scaling reinforcement learning by learning variable temporal resolution models, in: Proc. 9th International Conference on Machine Learning, Morgan Kaufmann, San Mateo, CA, 1992, pp. 406-415.
67. S.P. Singh, The efficient learning of multiple task sequences, in: Advances in Neural Information Processing Systems 4, Morgan Kaufmann, San Mateo, CA, 1992, pp. 251-258.
68. S.P. Singh, Transfer of learning by composing solutions of elemental sequential tasks, Machine Learning 8 (3/4) (1992) 323-340.
69. S.P. Singh, A.G. Barto, R.A. Grupen, C.I. Connolly, Robust reinforcement learning in motion planning, in: Advances in Neural Information Processing Systems 6, Morgan Kaufmann, San Mateo, CA, 1994, pp. 655-662.
70. S.P. Singh, D. Bertsekas, Reinforcement learning for dynamic channel allocation in cellular telephone systems, in: Advances in Neural Information Processing Systems 9, MIT Press, Cambridge, MA, 1997, pp. 974-980.
71. R.S. Sutton, TD models: Modeling the world at a mixture of time scales, in: Proc. 12th International Conference on Machine Learning, Morgan Kaufmann, San Mateo, CA, 1995, pp. 531-539.
72. R.S. Sutton, A.G. Barto, Reinforcement Learning: An Introduction, MIT Press, Cambridge, MA, 1998.
73. R.S. Sutton, B. Pinette, The learning of world models by connectionist networks, in: Proc. 7th Annual Conference of the Cognitive Science Society, 1985, pp. 54-64.
74. R.S. Sutton, D. Precup, S. Singh, Intra-option learning about temporally abstract actions, in: Proc. 15th International Conference on Machine Learning, Morgan Kaufmann, San Mateo, CA, 1998, pp. 556-564.
75. R.S. Sutton, S. Singh, D. Precup, B. Ravindran, Improved switching among temporally abstract actions, in: Advances in Neural Information Processing Systems 11, MIT Press, Cambridge, MA, 1999, pp. 1066-1072.
76. M. Tambe, A. Newell, P. Rosenbloom, The problem of expensive chunks and its solution by restricting expressiveness, Machine Learning 5 (3) (1990) 299-348.
77. G.J. Tesauro, Temporal difference learning and TD-Gammon, Comm. ACM 38 (1995) 58-68.
78. T. Thrun, A. Schwartz, Finding structure in reinforcement learning, in: Advances in Neural Information Processing Systems 7, Morgan Kaufmann, San Mateo, CA, 1995, pp. 385-392.
79. M. Uchibe, M. Asada, K. Hosada, Behavior coordination for a mobile robot using modular reinforcement learning, in: Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems, 1996, pp. 1329-1336.
80. C.J.C.H. Watkins, Learning with Delayed Rewards, Ph.D. Thesis, Cambridge University, 1989.
81. M. Wiering, J. Schmidhuber, HQ-learning, Adaptive Behavior 6 (2) (1997) 219-246.
82. L.E. Wixson, Scaling reinforcement learning techniques via modularity, in: Proc. 8th International Conference on Machine Learning, Morgan Kaufmann, San Mateo, CA, 1991, pp. 368-372.