Autonomous learning of high-level states and actions in continuous environments

IEEE Transactions on Autonomous Mental Development

Volumn 4, Issue 1, 2012, Pages 70-86

  Mugan, Jonathan ^a   Kuipers, Benjamin ^b  

^a 21St Century Technologies Inc   (United States)

^b UNIVERSITY OF MICHIGAN   (United States)

Author keywords

Active learning; intrinsic motivation; qualitative reasoning; reinforcement learning; unsupervised learning

Indexed keywords

ACTIVE LEARNING; AUTONOMOUS LEARNING; CONTINUOUS VARIABLES; DISCRETE VARIABLE REPRESENTATION; DISCRETIZATIONS; INTRINSIC MOTIVATION; LEARNING AGENTS; LEARNING MODELS; PREDICTIVE MODELS; QUALITATIVE REASONING; QUALITATIVE REPRESENTATION; REALISTIC PHYSICS; SIMULATED ROBOT;

ARTIFICIAL INTELLIGENCE; REINFORCEMENT LEARNING; UNSUPERVISED LEARNING;

AUTONOMOUS AGENTS;

EID: 84858634841     PISSN: 19430604     EISSN: None     Source Type: Journal    
DOI: 10.1109/TAMD.2011.2160943     Document Type: Article

References (56)

1. A. Saxena, J. Driemeyer, J. Kearns, and A. Ng, "Robotic grasping of novel objects," Adv. Neural Inform. Process. Syst., vol. 19, p. 1209, 2007.
2. P. Fitzpatrick, G. Metta, L. Natale, S. Rao, and G. Sandini, "Learning about objects through action-initial steps towards artificial cognition," in Proc. IEEE Int. Conf. Robot. Autom. (ICRA'03), Taipei, Taiwan, 2003, vol. 3.
3. S. Vijayakumar, A. D'souza, and S. Schaal, "Incremental online learning in high dimensions," Neural Comput., vol. 17, no. 12, pp. 2602-2634, 2005.
4. C. M. Vigorito and A. G. Barto, "Intrinsically motivated hierarchical skill learning in structured environments," IEEE Trans. Autonom. Mental Develop., vol. 2, no. 2, Jun. 2010.
5. G. L. Drescher, Made-Up Minds: A Constructivist Approach to Artificial Intelligence. Cambridge, MA: MIT Press, 1991.
6. P. R. Cohen, M. S. Atkin, T. Oates, and C. R. Beal, "Neo: Learning conceptual knowledge by sensorimotor interaction with an environment," in Proc. Agents '97, Marina del Rey, CA, 1997, ACM.
7. L. S. Zettlemoyer, H. Pasula, and L. P. Kaelbling, "Learning planning rules in noisy stochasticworlds," in Proc. 2nd Conf. Artif. Intell. (AAAI- 2005), Pittburgh, PA, 2005, pp. 911-918.
8. H. Pasula, L. Zettlemoyer, and L. Kaelbling, "Learning symbolic models of stochastic domains," J. Artif. Intell. Res., vol. 29, pp. 309-352, 2007.
9. C. Xu and B. Kuipers, "Towards the object semantic hierarchy," in Proc. Int. Conf. Develop. Learn. (ICDL 2010), Ann Arbor, MI, 2010.
10. J. Mugan and B. Kuipers, "Learning to predict the effects of actions: Synergy between rules and landmarks," in Proc. Int. Conf. Develop. Learn., London, U.K., 2007.
11. J. Mugan and B. Kuipers, "Learning distinctions and rules in a continuous world through active exploration," in Proc. Int. Conf. Epigenet. Robot., Piscataway, NJ, 2007.
12. J. Mugan and B. Kuipers, "Towards the application of reinforcement learning to undirected developmental learning," in Proc. Int. Conf. Epigenet. Robot., Brighton, U.K., 2008.
13. J. Mugan and B. Kuipers, "A comparison of strategies for developmental action acquisition in QLAP," in Proc. Int. Conf. Epigenet. Robot., Venice, Italy, 2009.
14. J. Mugan and B. Kuipers, "Autonomously learning an action hierarchy using a learned qualitative state representation," in Proc. Int. Joint Conf. Artif. Intell., Pasadena, CA, 2009.
15. J. Mugan, "Autonomous Qualitative Learning of Distinctions and Actions in a Developing Agent," Ph.D. dissertation, Univ. Texas Austin, Austin, TX, 2010.
16. U. Fayyad and K. Irani, "On the handling of continuous-valued attributes in decision tree generation," Mach. Learn., vol. 8, no. 1, pp. 87-102, 1992.
17. J. Mugan and B. Kuipers, "The qualitative learner of action and perception, QLAP," in Proc. AAAI Video Competition (AIVC 2010), Atlanta, GA, 2010 [Online]. Available: Http://videolectures. net/aaai2010-mugan-qlap
18. R. S. Sutton and A. G. Barto, Reinforcement Learning. Cambridge, MA: MIT Press, 1998.
19. C. G. Atkeson, A. W. Moore, and S. Schaal, "Locally weighted learning," Artif. Intell. Rev., vol. 11, no. 1/5, pp. 11-73, 1997.
20. C. G. Atkeson, A. W. Moore, and S. Schaal, "Locally weighted learning for control," Artif. Intell. Rev., vol. 11, no. 1/5, pp. 75-113, 1997.
21. S. Vijayakumar and S. Schaal, "Locally weighted projection regression: An O(n) algorithm for incremental real time learning in high dimensional space," in Proc. 17th Int. Conf. Mach. Learn. (ICML 2000), Palo Alto, CA, 2000, vol. 1, pp. 288-293.
22. M. Jordan and D. Rumelhart, "Forward models: Supervised learning with a distal teacher," Cogn. Sci., vol. 16, pp. 307-354, 1992.
23. C. Rasmussen, "Gaussian processes in machine learning," Adv. Lectures Mach. Learn., pp. 63-71, 2006.
24. B. Kuipers, Qualitative Reasoning. Cambridge, MA: MIT Press, 1994.
25. T. Degris, O. Sigaud, and P.Wuillemin, "Learning the structure of factored Markov decision processes in reinforcement learning problems," in Proc. Int. Conf. Mach. Learn. (ICML), Pittsburgh, PA, 2006, pp. 257-264.
26. A. Jonsson and A. Barto, "Active learning of dynamic bayesian networks in markov decision processes," Lecture Notes Artif. Intell.: Abstraction, Reformulation, and Approximation-SARA, pp. 273-284, 2007.
27. A. Strehl, C. Diuk, and M. Littman, "Efficient structure learning in factored-state MDPs," in Proc. AAAI, Vancouver, BC, Canada, 2007, vol. 22, no. 1, p. 645.
28. G. A. Miller, E. Galanter, and K. H. Pribram, Plans and the Structure of Behavior. New York: Holt, Rinehart and Winston, 1960.
29. N. J. Nilsson, Principles of Artificial Intelligence. Wellsboro, PA: Tioga Publishing Company, 1980.
30. M. Puterman, Markov Decision Problems. New York: Wiley, 1994.
31. C. Boutilier, T. Dean, and S. Hanks, "Decision theoretic planning: Structural assumptions and computational leverage," J. Artif. Intell. Res., vol. 11, no. 1, p. 94, 1999.
32. R. S. Sutton, D. Precup, and S. Singh, "Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning," Artif. Intell., vol. 112, no. 1-2, pp. 181-211, 1999.
33. R. S. Sutton and A. G. Barto, Reinforcement Learning. Cambridge, MA: MIT Press, 1998.
34. X. Huang and J. Weng, "Novelty and reinforcement learning in the value system of developmental robots," in Proc. 2nd Inter. Workshop Epigenet. Robot., Edinburgh, Scotland, 2002.
35. J. Marshall, D. Blank, and L. Meeden, "An emergent framework for self-motivation in developmental robotics," in Proc. 3rd Int. Conf. Develop. Learn. (ICDL 2004), La Jolla, CA, 2004.
36. R. Brafman and M. Tennenholtz, "R-max-a general polynomial time algorithm for near-optimal reinforcement learning," J. Mach. Learn. Res., vol. 3, pp. 213-231, 2003.
37. J. Schmidhuber, "Curious model-building control systems," in Proc. Int. Joint Conf. Neural Netw., Seattle,WA, 1991, vol. 2, pp. 1458-1463.
38. P. Oudeyer, F. Kaplan, and V. Hafner, "Intrinsic motivation systems for autonomous mental development," IEEE Trans. Evol. Comput., vol. 11, no. 2, pp. 265-286, 2007.
39. A. Bonarini, A. Lazaric, and M. Restelli, "Incremental skill acquisition for self-motivated learning animats," in Proc. 9th Int. Conf. Simul. Adapt. Behav. (SAB), Rome, Italy, 2006, pp. 357-368, Springer.
40. M. Marjanovic, B. Scassellati, and M. Williamson, "Self-taught visually guided pointing for a humanoid robot," in Proc. 4th Int. Conf. Simul. Adapt. Behav. (SAB), Cape Cod, MA, 1996, pp. 35-44.
41. G. Metta and P. Fitzpatrick, "Early integration of vision and manipulation," Adapt. Behav., vol. 11, no. 2, pp. 109-128, 2003.
42. K. Gold and B. Scassellati, "Learning acceptable windows of contingency," Connect. Sci., vol. 18, no. 2, pp. 217-228, 2006.
43. J. Klein, "Breve: A 3d environment for the simulation of decentralized systems and artificial life," in Proc. Int. Conf. Artif. Life, Sydney, Australia, 2003.
44. R. Smith, Open Dynamics Engine v 0.5 User Guide [Online]. Available: Http://ode.org/ode-latest-userguide.pdf 2004
45. V. G. Payne and L. D. Isaacs, Human Motor Development: A Lifespan Approach. New York: McGraw-Hill Humanities/Social Sciences/Languages, 2007.
46. A. Needham, T. Barrett, and K. Peterman, "A pick-me-up for infants' exploratory skills: Early simulated experiences reaching for objects using 'sticky mittens' enhances young infants' object exploration skills," Inf. Behav. Develop., vol. 25, no. 3, pp. 279-295, 2002.
47. J. Provost [Online]. Available: Sourceforge.net, 2008
48. D. M. Pierce and B. J. Kuipers, "Map learning with uninterpreted sensors and effectors," Artif. Intell., vol. 92, pp. 169-227, 1997.
49. R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification. New York: Wiley-Interscience Publication, 2000.
50. A. Jonsson and A. Barto, "Causal graph based decomposition of factored MDPs," The J. Mach. Learn. Res., vol. 7, pp. 2259-2301, 2006.
51. C. M. Vigorito and A. G. Barto, "Autonomous hierarchical skill acquisition in factored mdps," in Proc. Yale Workshop Adapt. Learn. Syst., New Haven, CT, 2008.
52. W.-M. Shen, Autonomous Learning from the Environment. New York: W. H. Freeman, 1994.
53. T. Dietterich, "The MAXQ method for hierarchical reinforcement learning," in Proc. Int. Conf. Mach. Learn. (ICML), Madison, WI, 1998.
54. B. Digney, "Emergent hierarchical control structures: Learning reactive/ hierarchical relationships in reinforcement environments," in Proc. 4th Int. Conf. Simul. Adapt. Behav., Cape Cod, MA, 1996, p. 363, MIT Press.
55. B. Hengst, "Discovering hierarchy in reinforcement learning with HEXQ," in Proc. 19th Int. Conf. Mach. Learn., Sydney, Australia, 2002, pp. 243-250.
56. T. Hester and P. Stone, International Foundation for Autonomous Agents and Multiagent Systems, "Generalized model learning for reinforcement learning in factored domains," in Proc. 8th Int. Conf. Autonom. Agents Multiagent Syst.-Volume 2, Budapest, Hungary, 2009, pp. 717-724.