Finding and transferring policies using stored behaviors

Autonomous Robots

Volumn 29, Issue 2, 2010, Pages 169-200

  Stolle, Martin ^a,b   Atkeson, Christopher ^a  

^a NONE   (United States)

^b GOOGLE   (Switzerland)

Author keywords

Behaviour based systems; Learning and adaptive systems; Learning from demonstration; Legged robots; Planning

Indexed keywords

BEHAVIOUR-BASED SYSTEMS; CONTROL APPROACH; CONTROL LAWS; FAST CONTROL; GLOBAL POLICIES; LEARNING AND ADAPTIVE SYSTEM; LEARNING FROM DEMONSTRATION; LEGGED ROBOTS; LOCAL FEATURE; PLANNING ALGORITHMS; POLICY ITERATION; QUADRUPED ROBOTS; ROUGH TERRAINS; SPEED-UPS; VALUE FUNCTIONS;

ADAPTIVE SYSTEMS; ALGORITHMS; DYNAMIC PROGRAMMING; EMBEDDED SYSTEMS; LANDFORMS; MARBLE; MULTIPURPOSE ROBOTS; TRAJECTORIES;

ROBOT PROGRAMMING;

EID: 77955774980     PISSN: 09295593     EISSN: None     Source Type: Journal    
DOI: 10.1007/s10514-010-9191-2     Document Type: Article

References (58)

1. Argall, B., Browning, B., & Veloso, M. M. (2008 Learning robot motion control with demonstration and advice-operators. In Proceedings IEEE/RSJ international conference on intelligent robots and systems.
2. Atkeson, C. G. (1994 Using local trajectory optimizers to speed up global optimization in dynamic programming. In J. D. Cowan, G. Tesauro, & J. Alspector (Eds.), Advances in Neural Information Processing Systems (Vol. 6, pp. 663-670 San Mateo: Morgan Kaufmann. URL ftp://ftp.cc.gatech.edu/pub/people/cga/ local.html.
3. Atkeson, C. G., & Morimoto, J. (2003 Nonparametric representation of policies and value functions: a trajectory-based approach. In S. Becker, S. Thrun, & K. Obermayer (Eds.), Advances in Neural Information Processing Systems (Vol. 15, pp. 1611-1618 Cambridge: MIT Press. URL http://www-2.cs. cmu.edu/~cga/publications.html.
4. Bailey, S., Grossman, R. L., Gu, L., & Hanley, D. (1996 A data intensive computing approach to path planning and mode management for hybrid systems. In R. Alur, T. A. Henzigner, & E. Sontag (Eds.), Lecture notes in computer science: Vol.1066. Hybrid systems III, Proceedings of the DIMACS workshop on verification and control of hybrid systems (pp. 485-495 Berlin: Springer. URL http://www.rgrossman.com/pubs.htm.
5. Bentivegna, D. C. (2004 Learning from observation using primitives. PhD thesis, Georgia Institute of Technology. URL http://etd. gatech.edu/theses/ available/etd-06202004-213721/.
6. Bentivegna, D. C., Atkeson, C. G., & Cheng, G. (2006 Learning similar tasks from observation and practice. In Proceedings of the 2006 IEEE/RSJ international conference on intelligent robots and systems (pp. 2677-2683 Beijing, China.
7. Boddy, M. S., & Dean, T. (1994 Deliberation scheduling for problem solving in time-constrained environments. Artificial Intelligence, 67(2), 245-285. doi:10.1016/0004-3702(94)90054-X.
8. Caruana, R. (1993 Multitask learning: a knowledge-based source of inductive bias. In International conference on machine learning.
9. Chernova, S., & Veloso, M. (2004a An evolutionary approach to gait learning for four-legged robots. In Proceedings of the international conference on intelligent robots and systems (IROS 2004 URL http://www-2.cs.cmu.edu/~coral/ publications/b2hd-iros04-chernova.html.
10. Chernova, S., & Veloso, M. (2004b Learning and using models of kicking motions for legged robots. In Proceedings of the international conference on robotics and automation (ICRA 2004 URL http://www-2.cs.cmu.edu/ ~coral/publications/b2hd-icra04- chernova.html.
11. Chernova, S., & Veloso, M. (2008 Learning equivalent action choices from demonstration. In International conference on intelligent robots and systems.
12. Chestnutt, J., Lau, M., Cheung, K. M., Kuffner, J., Hodgins, J. K., & Kanade, T. (2005 Footstep planning for the Honda ASIMO humanoid. In Proceedings of the IEEE international conference on robotics and automation. URL http://www.ri.cmu.edu/pubs/ pub-4970.html.
13. Conner, D. C., Rizzi, A., & Choset, H. (2003 Composition of local potential functions for global robot control and navigation. In Proceedings of the international conference on intelligent robots and systems (IROS 2003) (Vol. 4, pp. 3546-3551 New York: IEEE. URL http://www.ri.cmu.edu/pubs/pub-4556.html.
14. Connolly, C., & Grupen, R. (1993 The application of harmonic potential functions to robotics. Journal of Robotic Systems, 10(7), 931-946.
15. Davies, S. (1997 Multidimensional interpolation and triangulation for reinforcement learning. In Advances in neural information processing systems (Vol. 9 San Mateo: Morgan Kaufmann. URL http://www.autonlab.org/autonweb/ showPaper.jsp?ID=daviesmultidimensional.
16. Fern, A., Yoon, S. W., & Givan, R. (2004 Learning domain-specific control knowledge from random walks. In Proceedings of the international conference on automated planning and scheduling (ICAPS) (pp. 191-199
17. Fern, A., Yoon, S., & Givan, R. (2006 Approximate policy iteration with a policy language bias: solving relational Markov decision processes. Journal of Artificial Intelligence Research, 25, 85-118.
18. Fikes, R. E., Hart, P. E., & Nilsson, N. J. (1972 Learning and executing generalized robot plans. Artificial Intelligence, 3, 251-288.
19. Frazzoli, E. (2001 Robust hybrid control for autonomous vehicle motion planning. Department of aeronautics and astronautics, Massachusetts Institute of Technology, Cambridge, MA. URL http://rigoletto.seas.ucla.edu/papers/Year/2001. html.
20. Friedman, J. H., Bentley, J. L., & Finkel, R. A. (1977 An algorithm for finding best matches in logarithmic expected time. ACM Transactions on Mathematical Software, 3(3), 209-226. doi:10.1145/355744.355745, URL http://portal.acm.org/citation. cfm?doid=355744.355745.
21. Grollman, D. H., & Jenkins, O. C. (2008 Sparse incremental learning for interactive robot control policy estimation. In International conference on robotics and automation (pp. 3315-3320
22. Grossman, R., Mehta, S., & Qin, X. (1992 Path planning by querying persistent stores of trajectory segments (Tech. Rep. LAC 93- R3 Laboratory for Advanced Computing University of Illinois at Chicago. URL http://www.rgrossman. com/pubs.htm.
23. Guestrin, C., Koller, D., Gearhart, C., & Kanodia, N. (2003 Generalizing plans to new environments in relational MDPs. In Proceedings of the eighteenth international joint conference on artificial intelligence.
24. Howard, T., & Kelly, A. (2007 Optimal rough terrain trajectory generation for wheeled mobile robots. International Journal of Robotics Research, 26(2), 141-166. URL http://www.ri.cmu.edu/ pubs/pub-5739.html. Iba, G. A. (1989 A heuristic approach to the discovery of macrooperators. Machine Learning, 3, 285-317.
25. Iba, G. A. (1989). A heuristic approach to the discovery of macrooperators. Machine Learning, 3, 285-317.
26. Jacobson, D. H., & Mayne, D. Q. (1970 Differential dynamic programming. Amsterdam: Elsevier.
27. Kavraki, L., Svestka, P., Latombe, J., & Overmars, M. (1996 Probabilistic roadmaps for path planning in high-dimensional configuration spaces. IEEE Transactions on Robotics and Automation, 12(4), 566-580. doi:10.1109/70.508439, URL http://ai.stanford. edu/~latombe/pub.htm.
28. Kohl, N., & Stone, P. (2004 Machine learning for fast quadrupedal locomotion. In Proceedings of the nineteenth national conference on artificial intelligence (pp. 611-616 URL http://www. cs.utexas.edu/~nate/pubs/b2hd- kohlaaai04.html.
29. Kolter, J. Z., Abbeel, P., & Ng, A. Y. (2008 Hierarchical apprenticeship learning with application to quadruped locomotion. In Neural information processing systems 20.
30. Laird, J., Rosenbloom, P., & Newell, A. (1986 Chunking in Soar: The anatomy of a general learning mechanism. Machine Learning, 1, 11-46.
31. Lau, M., & Kuffner, J. J. (2005 Behavior planning for character animation. In SCA '05: proceedings of the 2005 ACM SIGGRAPH/Eurographics symposium on computer animation (pp. 271-280 New York: ACM Press. doi:10.1145/1073368. 1073408.
32. LaValle, S. M. (2006 Planning algorithms. Cambridge: Cambridge University Press. URL http://msl.cs.uiuc.edu/planning/, to appear.
33. LaValle, S. M., & Kuffner, J. J. Jr. (2001 Randomized kinodynamic planning. The International Journal of Robotics Research, 20(5), 378-400. doi:10.1177/02783640122067453, URL http://ijr.sagepub.com/cgi/content/abstract/ 20/5/378. (Pubitemid 32813681)
34. Lee, J., Chai, J., Reitsma, P. S. A., Hodgins, J. K., & Pollard, N. S. (2002 Interactive control of avatars animated with human motion data. In SIGGRAPH '02: Proceedings of the 29th annual conference on computer graphics and interactive techniques (pp. 491-500 New York: ACM Press. doi:10.1145/566570. 566607.
35. Littman, M. L., Sutton, R. S., & Singh, S. (2002 Predictive representations of state. In Advances in neural information processing systems (Vol. 14, pp. 1555-1561 San Mateo: Morgan Kaufmann. URL http://www.eecs.umich. edu/~baveja/PSRmainpage.html.
36. Mahadevan, S. (1992 Enhancing transfer in reinforcement learning by building stochastic models of robot actions. In Proceedings of the ninth international conference on machine learning (pp. 290-299 URL http://www.cs.umass. edu/~mahadeva/organized-pubs-by-year.html.
37. Mannor, S., Menache, I., Hoze, A., & Klein, U. (2004 Dynamic abstraction in reinforcement learning via clustering. In Proceedings of the twenty-first international conference on machine learning. McGovern, A. (2002 Autonomous discovery of temporal abstractions from interaction with an environment. PhD thesis, University of Massachusetts Amherst. URL http://www. cs.ou.edu/~amy/pubs.html.
38. Munos, R., & Moore, A. (2002 Variable resolution discretization in optimal control. Machine Learning, 49(2/3), 291-323. URL http://www.autonlab. org/autonweb/showPaper.jsp?ID=munosvariable.
39. Pearl, J. (1985 Heuristics: intelligent search strategies for computer problem solving. Reading: Addison-Wesley.
40. Ravindran, B., & Barto, A. G. (2003 SMDP homomorphisms: an algebraic approach to abstraction in semi Markov decision processes. In Proceedings of the eighteenth international joint conference on artificial intelligence. Menlo Park: AAAI Press. URL http://www.cs.iitm.ernet.in/~ravi/.
41. Reitsma, P. S. A., & Pollard, N. S. (2004 Evaluating motion graphs for character navigation. In Proceedings of ACM SIGGRAPH/ Eurographics 2004 symposium on computer animation.
42. Rimon, E., & Koditschek, D. E. (1992 Exact robot navigation using artificial potential fields. IEEE Transactions on Robotics and Automation, 8(5), 501-518.
43. Röfer, T. (2005 Evolutionary gait-optimization using a fitness function based on proprioception. In Eighth international workshop on robocup 2004. URL http://www.informatik. uni-bremen.de/~roefer/public-e.htm.
44. Sermanet, P., Hadsell, R., Scoffier, M., Grimes, M., Ben, J., Erkan, A., Crudele, C., Muller, U., & LeCun, Y. (2009 A multi-range architecture for collision-free off-road robot navigation. Journal of Field Robotics, 26(1), 58-87. URL http://yann.lecun.com/ exdb/publis/index.html.
45. Sim̧sek,Ö, & Barto, A. G. (2004). Using relative novelty to identify useful temporal abstractions in reinforcement learning. In Proceedings of the twenty-first international conference on machine learning. URL http://www.cs.umass.edu/~ozgur/.
46. Stolle, M. (2007 Images of mazes used. URL http://www.cs.cmu. edu/~mstoll/files/adprl2007-mazes.tar.gz, http://www.cs.cmu. edu/~mstoll/files/adprl2007-mazes.tar.gz.
47. Stolle, M. (2008 Finding and transferring policies using stored behaviors. PhD thesis, Carnegie Mellon University, 5000 Forbes Ave Pittsburgh, PA 15213. URL http://www.cs.cmu. edu/~mstoll/publications.shtml.
48. Stolle, M., & Atkeson, C. G. (2006 Policies based on trajectory libraries. In Proceedings of the international conference on robotics and automation (ICRA 2006 URL http://www.cs.cmu.edu/~mstoll/publications.shtml.
49. Stolle, M., & Atkeson, C. (2007a Transfer of policies based on trajectory libraries. In Proceedings of the international conference on intelligent robots and systems (IROS 2007 URL http://www.cs.cmu.edu/~mstoll/ publications.shtml.
50. Stolle, M., & Atkeson, C. G. (2007b Knowledge transfer using local features. In Proceedings of the IEEE symposium on approximate dynamic programming and reinforcement learning (ADPRL 2007 URL http://www.cs.cmu.edu/ ~mstoll/publications.shtml.
51. Stolle, M., & Precup, D. (2002 Learning options in reinforcement learning. In Lecture notes in computer science (Vol. 2371, pp. 212-223 Berlin: Springer. URL http://www.cs.cmu.edu/~ mstoll/publications.shtml. von Stryk, O.
52. 2001 DIRCOL. http://www.sim.informatik.tudarmstadt. de/sw/dircol.html.en, URL http://www.sim.informatik. tu-darmstadt.de/sw/dircol.html.en.
53. Sutton, R. S., & Barto, A. G. (1998 Reinforcement learning: an introduction. Cambridge: MIT Press. URL http://www.cs. ualberta.ca/~sutton/book/ the-book.html.
54. Veloso, M. M. (1992 Learning by analogical reasoning in general problem solving. PhD thesis, Carnegie Mellon University.
55. Weingarten, J. D., Lopes, G. A. D., Buehler, M., Groff, R. E., & Koditschek, D. E. (2004 Automated gait adaptation for legged robots. In International conference in robotics and automation. New York: IEEE Press.
56. Winner, E., & Veloso, M. (2002 Automatically acquiring planning templates from example plans. In Proceedings of AIPS'02 workshop on exploring real-world planning. URL http://www-2. cs.cmu.edu/~coral/publications/b2hd- 02aipsw-elly.html.
57. Yang, L., & LaValle, S. M. (2004 The sampling-based neighborhood graph: a framework for planning and executing feedback motion strategies. IEEE Transactions on Robotics and Automation, 20(3), 419-432.
58. Yoon, S., Fern, A., & Givan, R. (2008 Learning control knowledge for forward search planning. Journal of Machine Learning Research, 9, 683-718.