Module-Based Reinforcement Learning: Experiments with a Real Robot

Machine Learning

Volumn 31, Issue 1-3, 1998, Pages 55-85

  Kalmár, Zsolt ^a,b   Szepesvári, Csaba ^a,b   Lörincz, András ^a,b  

^a UNIVERSITY OF SZEGED   (Hungary)

^b Associative Computing Ltd   (Hungary)

Author keywords

Feature space; Local control; Markovian Decision Problems; Module based RL; Problem decomposition; Reinforcement learning; Robot learning; Subgoals; Switching control

Indexed keywords

FEATURE SPACE; LOCAL CONTROL; MARKOVIAN DECISION PROBLEMS; MODULE-BASED RL; PROBLEM DECOMPOSITION; REINFORCEMENT LEARNING (RL); SUBGOALS; SWITCHING CONTROL;

CONTROL EQUIPMENT; DISCRETE TIME CONTROL SYSTEMS; LEARNING ALGORITHMS; MARKOV PROCESSES; MATHEMATICAL MODELS; PROBLEM SOLVING; DECISION THEORY;

ROBOTS; ROBOT LEARNING;

FEATURE SPACE; LOCAL CONTROL; MODULE-BASED REINFORCEMENT LEARNING; PROBLEM DECOMPOSITION; SWITCHING CONTROL;

EID: 0032045145     PISSN: 08856125     EISSN: None     Source Type: Journal    
DOI: 10.1023/a:1007440607681     Document Type: Article

References (72)

1. Asada, M., Noda, S., Tawaratsumida, S. & Hosoda, K. (1996). Purposive behavior acquisition for a real robot by vision-based reinforcement learning. Machine Learning, 23:279-303.
2. Barto, A.G., Bradtke, S.J. & Singh, S.P. (1995). Learning to act using real-time dynamic programming. Artificial Intelligence, 1(72):81-138.
3. Bellman, R. (1957). Dynamic Programming. Princeton University Press, Princeton, New Jersey.
4. Birk, A. & Demiris, J. (1998). Sixth European Workshop on Learning Robots. Lecture Notes in Artificial Intelligence. Springer, Berlin, 1998.
5. Brafman, R.I. & Moshe, T. (1997). Modeling agents as qualitative decision makers. Artificial Intelligence, 94 (1):217-268.
6. Branicky, M.S. (1995). Studies in Hybrid Systems: Modeling, Analysis, and Control. PhD thesis, Laboratory of Information and Decision, MIT, 77 Massachusetts Avenue, Cambridge, MA 02139-4307 USA.
7. Branicky, M.S., Borkar, V.S. & Mitter, S.K. (1994). A unified framework for hybrid control: Background, model, and theory. Technical report lids-p-2239, Laboratory for Information and Decision Systems, MIT, 77 Massachusetts Avenue, Cambridge, MA 02139-4307 USA.
8. Brockett, R.W. (1993). Hybrid models for motion control systems. In Essays in Control: Perspectives in the Theory and its Applications, pages 29-53. Birkhäuser, Boston.
9. Brooks, R. (1991a). Artificial life and real robots. In Proceedings of the First European Conference on Artificial Life (ECAL), pages 3-10. MIT Press.
10. Brooks, R.A. (1991b). Elephants don't play chess. In Designing Autonomous Agents. Bradford-MIT Press, 1991.
11. Chrisman, L. (1992). Reinforcement learning with perceptual aliasing: The perceptual distinctions approach. In Proceedings of the Tenth National Conference on Artificial Intelligence, pages 183-188, San Jose, CA. AAAI Press.
12. Colombetti, M., Dorigo, M. & Borghi, G. (1996). Behavior analysis and training: A methodology for behavior engineering. IEEE Transactions on Systems, Man, and Cybernetics-Part B, 26(3):365-380.
13. de Kleer, J. & Seely, B.J. (1984). A qualitative physics based on confluences. Artificial Intelligence, 24(1-3): 7-83.
14. Dorigo, M. (1995). Alecsys and the autonomouse: Learning to control a real robot by distributed classifier systems. Machine Learning, 19(3):209-240.
15. Dorigo, M. & Colombetti, M. (1994). Robot shaping: Developing autonomous agents through learning. Artificial Intelligence, 71:321-370.
16. Gábor, Z., Kalmár, Zs. & Szepesvári, Cs.. (1998). Multi-criteria reinforcement learning. Technical report 98-115, Research Group on Artificial Intelligence, JATE-MTA.
17. Grossman, R.L., Nerode, A., Ravn, A. P. & Rischel, H. (1993). Hybrid Systems, volume 736 of Lecture Notes in Computer Science. Springer-Verlag, New York.
18. Heger, M. (1996). The loss from imperfect value functions in expectation-based and minimax-based tasks. Machine Learning, 22:197-225.
19. Henig, M.I. (1983). Vector-valued dynamic programming. SIAM J. Control and Optimization, 21(3):490-499.
20. Jaakkola, T., Jordan, M.I. & Singh, S.P. (1994). On the convergence of stochastic iterative dynamic programming algorithms. Neural Computation, 6(6):1185-1201.
21. Kaebling, L.P., Littman, M.L. & Moore, A.W. (1996). Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4:237-285.
22. Kalmár, Zs., Szepesvári, Cs. & Lorincz, A. (1994). Generalization in an autonomous agent. In Proc. of IEEE WCCI ICNN'94, volume 3, pages 1815-1817, Orlando, Florida. IEEE Inc.
23. Kalmár, Zs., Szepesvári, Cs. & Lorincz, A. (1995). Generalized dynamic concept model as a route to construct adaptive autonomous agents. Neural Network World, 5:353-360.
24. Kalmár, Zs., Szepesvári, Cs. & Lorincz, A. (1997). Module based reinforcement learning for a real robot. In Proc. of the 6th European Workshop on Learning Robots, pages 22-32.
25. Koenig, S. & Simmons, R.G. (1997). Complexity analysis of real-time reinforcement learning applied to finding shortest paths in deterministic domains. Machine Learning: A Special Issue on Reinforcement Learning, 12:234-345.
26. Korf, R.E. (1985a). Learning to solve problems by searching for macro-operators. Pitman Publisher, Massachusetts.
27. Korf, R.E. (1985b). Macro-operators: A weak method for learning. Artificial Intelligence, 26:35-77.
28. Korf, R.E. (1987). Planning as search: A quantitative approach. Artificial Intelligence, 33:65-88.
29. Koza, J.R. & Rice, J.P. (1992). Automatic programming of robots using genetic programming. In Proceedings of Tenth National Conference on Artificial Intelligence, pages 194-201, Menlo Park, CA. AAAI Press/The MIT Press.
30. Kumar, P.R. (1985). A survey of some results in stochastic adaptive controls. SIAM Journal of Control and Optimization, 23:329-380.
31. Littman, M.L. (1996). Algorithms for Sequential Decision Making. PhD thesis, Department of Computer Science, Brown University. Also Technical Report CS-96-09.
32. Littman, M.L. & Szepesvári, Cs. (1996). A Generalized Reinforcement Learning Model: Convergence and applications. In Int. Conf. on Machine Learning, pages 310-318.
33. Lygeros, J., Godbole, D.N. & Sastry, S.S. (1997). A design framework for hierarchical, hybrid control. IEEE Transactions on Automatic Control, special issue on Hybrid Systems, (submitted).
34. Maes, P. (1991a). Adaptive action selection. In Proc. of the Thirteenth Annual Conf. of the Cognitive Science Society. Lawrence Erlbaum Associates.
35. Maes, P. (1991b), A bottom-up mechanism for behavior selection in an artificial creature. In J.A. Meyer and S. Wilson, editors, Proc. of the First International Conference on Simulation of Adaptive Behavior. MIT Press.
36. Maes, P. (1992). Learning behavior networks from experience. In Toward a Practice of Autonomous Systems (Proc. First European Conference on Artificial Life), pages 48-57. MIT Press, Cambridge, Massachusetts.
37. Maes, P. & Brooks, R.A. (1990). Learning to coordinate behaviors. In Proc. of AAAI-90, pages 796-802, Boston, MA.
38. Mahadevan, S. & Connell, J. (1992). Automatic programming of behavior-based robots using reinforcement learning. Artificial Intelligence, 55:311-365.
39. Matarić, M. (1997). Reinforcement learning in the multi-robot domain. Autonomous Robots, 4.
40. McCallum, R.A. (1993). Overcoming incomplete perception with utile distinction memory. In Proceedings of the Tenth International Conference on Machine Learning, pages 190-196, Amherst, Massachusetts. Morgan Kaufmann.
41. Munos, R. (1997). Finite-element methods with local triangulation refinement for continuous reinforcement learning problems. In M.van Someren and G. Widmer, editors, Machine Learning: ECML'97 (9th European Conf. on Machine Learning, Proceedings), volume 1224 of Lecture Notes in Artificial Intelligence, pages 170-183. Springer, Berlin.
42. Newell, A. & Simon, H.A. (1972). Human Problem Solving. Prentice-Hall, Englewood Cliffs, NJ.
43. Parr, R. & Russell, S. (1997). Reinforcement learning with hierarchies of machines. In Advances in Neural Information Processing Systems 11, Cambridge, MA. MIT Press. in press.
44. Pólya, Gy.. (1945). How to solve it? Princeton University Press, Princeton, NJ.
45. Precup, D., Sutton, R.S. & Singh, S.P. (1997). Planning with closed-loop macro actions. In Working notes of the 1997 AAAI Fall Symposium on Model-directed Autonomous Systems. AAAI Press/The MIT Press. in press.
46. Ross, S.M. (1970). Applied Probability Models with Optimization Applications. Holden Day, San Francisco, California.
47. Sacerdoti, E.D. (1974). Planning in a hierarchy of abstraction spaces. Artificial Intelligence, 5:115-135.
48. Sastry, S. (1997). Algorithms for design of hybrid systems. In Proc. of Int. Conf. of Information Sciences.
49. Say, A.C.C. & Selahattin, K. (1996). Qualitative system identification: deriving structure from behavior. Artificial Intelligence, 83(1):75-141.
50. Singh, S. & Cohn, D. (1997). How to dynamically merge markov decision processes. In Advances in Neural Information Processing Systems 11, Cambridge, MA. MIT Press. in press.
51. Singh, S., Jaakkola, T., Littman, M.L. & Szepesvári, Cs.. (1997). On the convergence of single-step on-policy reinforcement-learning algorithms. Machine Learning. accepted.
52. Singh, S.P. (1992). Reinforcement learning with a hierarchy of abstract models. In Proceedings of the Tenth National Conference on Artificial Intelligence, pages 202-207, San Jose, CA. AAAI Press.
53. Singh, S.P., Jaakkola, T. & Jordan, M.I. (1995). Learning without state-estimation in partially observable markovian decision processes. In Proc. of the Eleventh Machine Learning Conference, pages pp. 284-292.
54. Sutton, R.S. (1996). Generalization in reinforcement learning: Successful examples using sparse coarse coding. Advances in Neural Information Processing Systems, 8.
55. Sutton, R.S. (1984). Temporal Credit Assignment in Reinforcement Learning. PhD thesis, University of Massachusetts, Amherst, MA.
56. Szepesvári, Cs. (1994). Dynamic Concept Model learns optimal policies. In Proc. of IEEE WCCI ICNN'94, volume 3, pages 1738-1742, Orlando, Florida. IEEE Inc.
57. Szepesvári, Cs. (1997a). Learning and exploitation do not conflict under minimax optimality. In M.van Someren and G. Widmer, editors, Machine Learning: ECML'97 (9th European Conf. on Machine Learning, Proceedings), volume 1224 of Lecture Notes in Artificial Intelligence, pages 242-249. Springer, Berlin.
58. Szepesvári, Cs. (1997b). Static and Dynamic Aspects of Optimal Sequential Decision Making. PhD thesis, Bolyai Institute of Mathematics, University of Szeged, Szeged, Aradi vrt. tere 1, HUNGARY, 6720.
59. Szepesvári, Cs. & Littman, M.L. (1997). Generalized Markov Decision Processes: Dynamic programming and reinforcement learning algorithms. Neural Computation. in preparation.
60. Szepesvári, Cs. & Lorincz, A. (1994). Behavior of an adaptive self-organizing autonomous agent working with cues and competing concepts. Adaptive Behavior, 2(2):131-160.
61. Thrun, S.B. (1992). The role of exploration in learning control. Van Nostrand Rheinhold, Florence KY.
62. Thrun, S. & Schwartz, A. (1995). Finding structure in reinforcement learning. In Gerald Tesauro, David S. Touretzky, and Todd K. Leen, editors, Advances in Neural Information Processing Systems, volume 7, pages 385-392. The MIT Press, Cambridge.
63. Tóth, G.J., Kovács, Sz. & Lorincz, A. (1995). Genetic algorithm with alphabet optimization. Biological Cybernetics, 73:61-68.
64. Tsitsiklis, J.N. (1994). Asynchronous stochastic approximation and q-learning. Machine Learning, 8(3-4): 257-277.
65. Tsitsiklis, J.N. & Van Roy, B. (1996). Feature-based methods for large scale dynamic programming. Machine Learning, 22:59-94.
66. Tsitsiklis, J.N. & Van Roy, B. (1995). An analysis of temporal difference learning with function approximation. Technical Report LIDS-P-2322, Laboratory for Information and Decision Systems, Massachusetts Institute of Technology.
67. Tyrrell, T. (1993). Computational Mechanisms for Action Selection. PhD thesis, University of Edinburgh.
68. Uchibe, E., Asada, M. & Hosoda, K. (1996). Behavior coordination for a mobile robot using modular reinforcement learning. In Proc. of IEEE/RSJ Int. Conf. on Intelligent Robot and Sytems, pages 1329-1336.
69. Watkins, C.J.C.H. & Dayan, P. (1992). Q-learning. Machine Learning, 3(8):279-292.
70. Werbös, P.J. (1977). Advanced forecasting methods for global crisis warning and models of intelligence. General Systems Yearbook, 22:25-38.
71. Wiering, M. & Schmidhuber, J. (1997). HQ-learning. Adaptive Behavior, 6(2).
72. Zabczyk, J. (1973). Optimal control by means of switching. Studia Mathematica, 65:161-171.