Learning control in robotics

IEEE Robotics and Automation Magazine

Volumn 17, Issue 2, 2010, Pages 20-29

  Schaal, Stefan ^a   Atkeson, Christopher G ^b  

^a UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^b CARNEGIE MELLON UNIVERSITY   (United States)

Author keywords

Learning control; Optimal control; Reinforcement learning; Robot learning

Indexed keywords

BAYESIAN; CONTROL POLICY; CONTROLLED SYSTEM; CONTROLLER PARAMETER; INTERNAL MODELS; INVERSE MODELS; LEARNING CONTROL; LEARNING CONTROL TECHNIQUES; LOCALLY WEIGHTED REGRESSION; OPTIMAL CONTROL; OPTIMAL CONTROLS; ROBOT ARMS; ROBUST CONTROLLERS; STATE VECTOR; TRAJECTORY-BASED;

CONTROL; INVERSE KINEMATICS; LEARNING ALGORITHMS; MATHEMATICAL MODELS; OPTIMIZATION; REINFORCEMENT; REINFORCEMENT LEARNING; ROBOT LEARNING; ROBOTICS; ROBOTS;

CONTROLLERS;

EID: 77953330028     PISSN: 10709932     EISSN: None     Source Type: Journal    
DOI: 10.1109/MRA.2010.936957     Document Type: Article

References (83)

1. S. Schaal, "The new robotics-Towards human-centeredmachines," HFSP J. Frontiers Interdisciplinary Res. Life Sci., vol.1, no.2, pp. 115-126, 2007.
2. K. J. Åstrom and B. Wittenmark, Adaptive Control. Reading, MA: Addison- Wesley, 1989.
3. S. Thrun, W. Burgard, and D. Fox, Probabilistic Robotics. Cambridge, MA: MIT Press, 2005.
4. M. Buehler, The DARPA Urban Challenge: Autonomous Vehicles in City Traffic, 1st ed. New York: Springer-Verlag, 2009.
5. M. Buehler, K. Iagnemma, and S. Singh, The 2005 DARPA Grand Challenge: The Great Robot Race. New York: Springer-Verlag, 2007.
6. M. Roy, G. Gordon, and S. Thrun, "Finding approximate POMDP solutions through belief compression," J. Artif. Intell. Res., vol.23, pp. 1-40, 2005.
7. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press, 1998.
8. J. Si, Handbook of Learning and Approximate Dynamic Programming. Hoboken, NJ: IEEE Press/Wiley-Interscience, 2004.
9. R. Bellman, Dynamic Programming. Princeton, NJ: Princeton Univ. Press, 1957.
10. P. Dyer and S. R. McReynolds, The Computation and Theory of Optimal Control. New York: Academic, 1970.
11. L. Sciavicco and B. Siciliano, Modelling and Control of Robot Manipulators. New York: Springer-Verlag, 2000.
12. I. M. Jordan, D. E. Rumelhart, "Supervised learning with a distal teacher," Cogn. Sci., vol.16, pp. 307-354, 1992.
13. A. D'Souza, S. Vijayakumar, and S. Schaal, "Learning inverse kinematics," in Proc. IEEE Int. Conf. Intelligent Robots and Systems (IROS 2001), Maui, HI, Oct. 29-Nov. 3, 2001, pp. 298-301.
14. D. Bullock, S. Grossberg, and F. H. Guenther, "A self-organizing neural model of motor equivalent reaching and tool use by a multijoint arm," J. Cogn. Neurosci., vol.5, no.4, pp. 408-435, 1993.
15. J. Peters and S. Schaal, "Learning to control in operational space," Int. J. Robot. Res., vol.27, pp. 197-212, 2008.
16. Z. Ghahramani and M. I. Jordan, "Supervised learning from incomplete data via an EM approach," in Advances in Neural Information Processing Systems 6, J. D. Cowan, G. Tesauro, and J. Alspector, Eds. San Mateo, CA: Morgan Kaufmann, 1994, pp. 120-127.
17. S. Schaal and C. G. Atkeson, "Constructive incremental learning from only local information," Neural Comput., vol.10, no.8, pp. 2047-2084, 1998.
18. W. S. Cleveland, "Robust locally weighted regression and smoothing scatterplots," J. Amer. Statist. Assoc., vol.74, pp. 829-836, 1979.
19. C. G. Atkeson, "Using local models to control movement," in Advances in Neural Information Processing Systems 1, D. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 1989, pp. 157-183.
20. C. G. Atkeson, A. W. Moore, and S. Schaal, "Locally weighted learning," Artif. Intell. Rev., vol.11, no.1-5, pp. 11-73, 1997.
21. C. G. Atkeson, A. W. Moore, and S. Schaal, "Locally weighted learning for control," Artif. Intell. Rev., vol.11, no.1-5, pp. 75-113, 1997.
22. S. Vijayakumar, A. D'Souza, and S. Schaal, "Incremental online learning in high dimensions," Neural Comput., vol.17, no.12, pp. 2602-2634, 2005.
23. J.-A. Ting, A. D'Souza, S. Vijayakumar, and S. Schaal, "A Bayesian approach to empirical local linearizations for robotics," in Proc. Int. Conf. Robotics and Automation (ICRA2008), Pasadena, CA, May 19-23, 2008, pp. 2860-2865.
24. C. E. Rasmussen and C. K. I. Williams, Gaussian Processes for Machine Learning. Cambridge, MA: MIT Press, 2006.
25. D. Nguyen-Tuong, M. Seeger, and J. Peters, "Local gaussian process regression for real time online model learning and control," in Proc. Advances in Neural Information Processing Systems 21 (NIPS 2008), D. Schuurmans, J. Benigio, and D. Koller, Eds. Vancouver, BC, Dec. 8-11, 2009, pp. 1193-1200.
26. M. P. Deisenroth, C. E. Rasmussen, and J. Peters, "Gaussian process dynamic programming," Neurocomputing, vol.72, no.7-9, pp. 1508- 1524, 2009.
27. L. Csat'o and M. Opper, "Sparse representation for gaussian process models," in Proc. Advances in Neural Information Processing Systems 13 (NIPS 2000), Denver, CO, 2001, pp. 444-450.
28. D. M. Wolpert and M. Kawato, "Multiple paired forward and inverse models for motor control," Neural Netw., vol.11, no.7-8, pp. 1317- 1329, 1998.
29. D. H. Jacobson and D. Q. Mayne, Differential Dynamic Programming. New York: American Elsevier, 1970.
30. K. Doya, "Reinforcement learning in continuous time and space," Neural Comput., vol.12, no.1, pp. 219-245, Jan. 2000.
31. E. Theodorou, J. Buchli, and S. Schaal, "Reinforcement learning in high dimensional state spaces: A path integral approach," submitted for publication.
32. R. Munos and A. Moore, "Variable resolution discretization in optimal control," Mach. Learn., vol. 49, no. 2/3, p. 33, 2002.
33. C. G. Atkeson and B. J. Stephens, "Random sampling of states in dynamic programming," IEEE Trans. Syst., Man, Cybern. B, vol.38, no.4, pp. 924-929, 2008.
34. C. G. Atkeson, "Randomly sampling actions in dynamic programming," in Proc. IEEE Int. Symp. Approximate Dynamic Programming and Reinforcement Learning, 2007, ADPRL'07, pp. 185-192.
35. E. Whitman and C. G. Atkeson, "Control of a walking biped using a combination of simple policies," in Proc. IEEE/RAS Int. Conf. Humanoid Robotics, Paris, France, Dec. 7-10, 2009, pp. 520-527.
36. Tomlab Optimization Inc. (2010). PROPT-Matlab optimal control software [Online]. Available: http://tomdyn.com/
37. Technische Universitat Darmstadt. (2010). DIRCOL: A direct collocation method for the numerical solution of optimal control problems [Online]. Available: http://www.sim.informatik.tu-darmstadt.de/sw/dircol
38. Stanford Business Software Corporation. (2010). SNOPT; Software for large-scale nonlinear programming [Online]. Available: http://www.sbsisol- optimize.com/asp/sol-product-snopt.htm
39. A. Safonova, J. K. Hodgins, and N. S. Pollard, "Synthesizing physically realistic human motion in low-dimensional, behavior-specific spaces," ACM Trans. Graph. J.(SIGGRAPH 2004 Proc.), vol.23, no.3, pp. 514- 521, 2004.
40. L. Chenggang and C. G. Atkeson, "Standing balance control using a trajectory library," presented at the IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS 2009), 2009.
41. A. Ng, "Pegasus: A policy search method for large MDPs and POMDPs," presented at the Uncertainty in Artificial Intelligence (UAI), 2000.
42. E. Tse, Y. Bar-Shalom, and L. Meier, III, "Wide-sense adaptive dual control for nonlinear stochastic systems," IEEE Trans. Automat. Contr., vol.18, no.2, pp. 98-108, 1973.
43. Y. Bar-Shalom and E. Tse, "Caution, probing and the value of information in the control of uncertain systems," Ann. Econ. Social Meas., vol.4, no.3, pp. 323-338, 1976.
44. C. G. Atkeson and S. Schaal, "Robot learning from demonstration," in Proc. 14th Int. Conf. Machine Learning (ICML'97), D. H. Fisher, Jr., Ed. Nashville, TN, July 8-12, 1997, pp. 12-20.
45. R. S. Sutton, "Learning to predict by the methods of temporal differences," Mach. Learn., vol.3, no.1, pp. 9-44, 1988.
46. C. J. C. H. Watkins, "Learning with delayed rewards," Ph.D. thesis, Cambridge Univ., U.K., 1989.
47. J. Morimoto and K. Doya, "Acquisition of stand-up behavior by a real robot using hierarchical reinforcement learning," Robot. Auton. Syst., vol.36, no.1, pp. 37-51, 2001.
48. S. Schaal, "Is imitation learning the route to humanoid robots?" Trends Cogn. Sci., vol.3, no.6, pp. 233-242, 1999.
49. S. Schaal, A. Ijspeert, and A. Billard, "Computational approaches to motor learning by imitation," Philos. Trans. R. Soc. London B, Biol. Sci., vol.358, no.1431, pp. 537-547, 2003.
50. C. G. Atkeson and S. Schaal, "Learning tasks from a single demonstration," in Proc. IEEE Int. Conf. Robotics and Automation (ICRA'97), Albuquerque, NM, Apr. 20-25, 1997, pp. 1706-1712.
51. S. Schaal, "Learning from demonstration," in Proc. Advances in Neural Information Processing Systems 9, M. C. Mozer, M. Jordan, and T. Petsche, Eds. Cambridge, MA, 1997, pp. 1040-1046.
52. D. Ernst, P. Geurts, and L. Wehenkel, "Tree-based batch mode reinforcement learning," J. Mach. Learn. Res., vol.6, pp. 503-556, 2005.
53. G. Neumann and J. Peters, "Fitted Q-iteration by advantage weighted regression," in Proc. Advances in Neural Information Processing Systems 21 (NIPS 2008), D. Schuurmans, J. Benigio, and D. Koller, Eds. Vancouver, BC, Dec. 8-11, 2009, pp. 1177-1184.
54. R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour, "Policy gradient methods for reinforcement learning with function approximation," in Proc. Advances in Neural Processing Systems 12, S. A. Solla, T. K. Leen, and K.-R.Muller, Eds. Denver, CO, 2000.
55. J. Peters and S. Schaal, "Reinforcement learning of motor skills with policy gradients," Neural Netw., vol.21, no.4, pp. 682-697, May 2008.
56. P. Sadegh and J. Spall, "Optimal random perturbations for stochastic approximation using a simultaneous perturbation gradient approximation," presented at the Proc. American Control Conf., 1997.
57. R. J. Williams, "Simple statistical gradient-following algorithms for connectionist reinforcement learning," Mach. Learn., vol.8, no.3-4, pp. 229- 256, 1992.
58. V. Gullapalli, "A stochastic reinforcement learning algorithm for learning real-valued functions," Neural Netw., vol.3, no.6, pp. 671-692, 1990.
59. D. Aberdeen and J. Baxter, "Scaling internal-state policy-gradient methods for POMDPs," in Proc. 19th Int. Conf. Machine Learning (ICML-2002), Sydney, Australia, 2002, pp. 3-10.
60. J. Peters and S. Schaal, "Natural actor critic," Neurocomputing, vol.71, no.7-9, pp. 1180-1190, 2008.
61. S. Amari, "Natural gradient learning for over- and under-complete bases In ICA," Neural Comput., vol.11, no.8, pp. 1875-1883, Nov. 1999.
62. S. Kakade, "Natural policy gradient," presented at the Advances in Neural Information Processing Systems, Vancouver, CA, 2002.
63. T. Ruckstieß, M. Felder, and J. Schmidhuber, "State-dependent exploration for policy gradient methods," presented at the European Conf. Machine Learning and Principles and Practice of Knowledge Discovery in Databases 2008, Part II, LNAI 5212, 2008.
64. G. Endo, J. Morimoto, T. Matsubara, J. Nakanish, and G. Cheng, "Learning CPG-based biped locomotion with a policy gradient method: Application to a humanoid robot," Int. J. Robot. Res., vol.27, no.2, pp. 213-228, 2008.
65. R. Tedrake, T.W. Zhang, and S. Seung, "Stochastic policy gradient reinforcement learning on a simple 3D biped," in Proc. Int. Conf. Intelligent Robots and Systems (IROS 2004), Sendai, Japan, Oct. 2004, pp. 2849-2854.
66. J. Peters and S. Schaal, "Policy gradient methods for robotics," in Proc. IEEE Int. Conf. Intelligent Robotics Systems (IROS 2006), Beijing, Oct. 9- 15, 2006, pp. 2219-2225.
67. P. Dayan and G. Hinton, "Using EM for reinforcement learning," Neural Comput., vol.9, no.2, pp. 271-278, 1997.
68. A. P. Dempster, N. M. Laird, and D. B. Rubin, "Maximum likelihood from incomplete data via the EM algorithm," J. R. Statist. Soc. B, vol.39, no.1, pp. 1-38, 1977.
69. J. Kober and J. Peters, "Learning motor primitives in robotics," in Proc. Advances in Neural Information Processing Systems 21 (NIPS 2008), D. Schuurmans, J. Benigio, and D. Koller, Eds. Vancouver, BC, Dec. 8-11, 2009, pp. 297-304.
70. M. Toussaint and A. Storkey, "Probabilistic inference for solving discrete and continuous state Markov decision processes," presented at the 23nd Int. Conf. Machine Learning (ICML 2006), 2006.
71. N. Vlassis, M. Toussaint, G. Kontes, and S. Piperidis, "Learning modelfree control by a Monte-Carlo EM algorithm," Auton. Robots, vol.27, no.2, pp. 123-130, 2009.
72. H. J. Kappen, "Linear theory for control of nonlinear stochastic systems," Phys. Rev. Lett., vol.95, no.20, pp. 200201-200204, Nov. 2005.
73. H. J. Kappen, "An introduction to stochastic control theory, path integrals and reinforcement learning," in Cooperative Behavior in Neural Systems, vol.887, J. Marro, P. L. Garrido, and J. J. Torres, Eds. 2007, pp. 149-181.
74. E. Theodorou, J. Buchli, and S. Schaal, "Path integral stochastic optimal control for rigid body dynamics," presented at the IEEE Int. Symp. Approximate Dynamic Programming and Reinforcement Learning (ADPRL2009), Nashville, TN, Mar. 30-Apr. 2, 2009.
75. E. Todorov, "Efficient computation of optimal actions," Proc. Nat. Acad. Sci. USA, vol.106, no.28, pp. 11478-11483, July 2009.
76. A. Ijspeert, J. Nakanishi, and S. Schaal, "Learning attractor landscapes for learning motor primitives," in Advances in Neural Information Processing Systems 15, S. Becker, S. Thrun, and K.Obermayer, Eds. 2003, pp. 1547-1554.
77. C. M. Bishop, Pattern Recognition and Machine Learning. New York: Springer-Verlag, 2006.
78. A. Billard, S. Calinon, R. Dillmann, and S. Schaal, "Robot programming by demonstration," in Handbook of Robotics, vol.1, B. Siciliano andO.Khatib, Eds. Cambridge,MA:MIT Press, 2008, ch. 59.
79. Y. Wada and M. Kawato, "Trajectory formation of arm movement by a neural network with forward and inverse dynamics models," Syst. Comput. Jpn., vol.24, pp. 37-50, 1994.
80. T. Inamura, I. Toshima, H. Tanie, and Y. Nakamura, "Embodied symbol emergence based on mimesis theory," Int. J. Robot. Res., vol.23, no.4-5, p. 363, Apr.-May 2004.
81. A. Y. Ng and S. Russell, "Algorithms for inverse reinforcement learning," in Proc. 17th Int. Conf. Machine Learning (ICML 2000), Stanford, CA, 2000, pp. 663-670.
82. P. Abbeel and A. Ng, "Apprenticeship learning via inverse reinforcement learning," in Proc. 21st Int. Conf. Machine Learning, 2004.
83. N. Ratliff, D. Silver, and J. A. Bagnell, "Learning to search: Functional gradient techniques for imitation learning," Auton. Robots, vol.27, no.1, pp. 25-53, 2009.