Evaluation of linearly solvable Markov decision process with dynamic model learning in a mobile robot navigation task

Frontiers in Neurorobotics

Volumn 7, Issue APR, 2013, Pages

  Kinjo, Ken ^a,b   Uchibe, Eiji ^b   Doya, Kenji ^a,b  

^a NARA INSTITUTE OF SCIENCE AND TECHNOLOGY   (Japan)

^b OKINAWA INSTITUTE OF SCIENCE AND TECHNOLOGY GRADUATE UNIVERSITY   (Japan)

Author keywords

Linearly solvable Markov decision process; Model learning; Model based reinforcement learning; Optimal control; Robot navigation

Indexed keywords

ARTICLE; CAMERA; COMPUTER SIMULATION; DECISION MAKING; LEARNING; LINEARLY SOLVABLE MARKOV DECISION PROCESS; MATHEMATICAL COMPUTING; ROBOTICS; STATISTICAL MODEL; TASK PERFORMANCE;

EID: 84904902685     PISSN: None     EISSN: 16625218     Source Type: Journal    
DOI: 10.3389/fnbot.2013.00007     Document Type: Article

References (28)

1. Barto, A. G., and Sutton, R. S. (1998). Reinforcement Learning. Cambridge, MA: MIT Press/Bradford Books.
2. Boyan, J. A. (2002). Technical update: least-squares temporal difference learning. Mach. Learn. 49, 233-246.
3. Burdelis, M. A. P., and Ikeda, K. (2011). "Estimating passive dynamics distributions in linearly solvable markov decision processes from measured immediate costs in reinforcement learning problems," in Proceedings of the 21st Annual Conference of the Japanese Neural Network Society (Okinawa).
4. da Silva, M., Durand, F., and Popovic, J. (2009). Linear Bellman combination for control of character animation. ACM Trans. Grap. 28. doi: 10.1145/1531326.1531388.
5. Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., and Dolan, R. J. (2011). Model-based influences on humans' choices and striatal prediction errors. Neuron 69, 1204-1215.
6. Deisenroth, M. P., and Rasmussen, C. E. (2011). "PILCO: a model-based and data-efficient approach to policy search," in Proceedings of the 28th International Conference on Machine Learning, eds L. Getoor and T. Scheffer (Bellevue, WA, USA).
7. Deisenroth, M. P., Rasmussen, C. E., and Peters, J. (2009). Gaussian process dynamic programming. Neurocomputing 72, 1508-1524.
8. Doll, B. B., Simon, D. A., and Daw, N. D. (2012). The ubiquity of model-based reinforcement learning. Curr. Opin. Neurobiol. 22, 1075-1081.
9. Doya, K. (2009). How can we learn efficiently to act optimally and flexibly? Proc. Natl. Acad. Sci. U.S.A. 106, 11429-11430.
10. Fleming, W., and Soner, H. (eds.). (2006). "Logalithmic transformations and risk sensitivity," in Controlled Markov Processes and Viscosity Solutions, Chapter 6 (New York, NY: Springer Science + Business Media, Inc.), 227-260.
11. Hester, T., Quinlan, M., and Stone, P. (2010). "Generalized model learning for Reinforcement Learning on a humanoid robot," in Proceedings of IEEE International Conference on Robotics and Automation (Anchorage, AK: IEEE), 2369-2374.
12. Kappen, H. (2005a). Linear theory for control of nonlinear stochastic systems. Phys. Rev. Lett. 95, 200201-200204.
13. Kappen, H. (2005b). Path integrals and symmetry breaking for optimal control theory. J. Stat. Mech. Theor. Exp. 11, P11011.
14. Lagoudakis, M. G., and Parr, R. (2003). Least-squares policy iteration. J. Mach. Learn. Res. 4, 1107-1149.
15. Nguyen-Tuong, D., and Peters, J. (2011). Model learning for robot control: a survey. Cogn. Proc. 12, 319-340.
16. Niv, Y. (2009). Reinforcement learning in the brain. J. Math. Psychol. 53, 139-154.
17. Sigaud, O., Salaün, C., and Padois, V. (2011). On-line regression algorithms for learning mechanical models of robots: a survey. Robot. Auton. Syst. 59, 1115-1129.
18. Stulp, F., and Sigaud, O. (2012). "Path integral policy improvement with covariance matrix adaptation," in Proceedings of the 10th European Workshop on Reinforcement Learning (EWRL 2012) (Edinburgh).
19. Sugimoto, N., and Morimoto, J. (2011). "Phase-dependent trajectory optimization for periodic movement using path integral reinforcement learning," in Proceedings of the 21st Annual Conference of the Japanese Neural Network Society (Okinawa).
20. Sugiyama, M., Takeuchi, I., Suzuki, T., Kanamori, T., and Hachiya, H. (2010). Least-squares conditional density estimation. IEICE Trans. Inform. Syst. E93-D, 583-594.
21. Theodorou, E., Buchli, J., and Schaal, S. (2010). A generalized path integral control approach to reinforcement learning. J. Mach. Learn. Res. 11, 3137-3181.
22. Theodorou, E., and Todorov, E. (2012). "Relative entropy and free energy dualities: connections to path integral and kl control," in the 51th IEEE Conference on Decision and Control (Maui), 1466-1473.
23. Todorov, E. (2006). "Optimal control theory," in Bayesian Brain: Probabilistic Approaches to Neural Coding, Chapter 12, eds D. Kenji, S. Ishii, A. Pouget, and R. P. Rao (Cambridge, MA: MIT Press), 269-298.
24. Todorov, E. (2007). Linearly-solvable Markov decision problems. Adv. Neural Inform. Proc. Syst. 19, 1369-1379.
25. Todorov, E. (2009a). Compositionality of optimal control laws. Adv. Neural Inform. Proc. Syst. 22, 1856-1864.
26. Todorov, E. (2009b). Efficient computation of optimal actions. Proc. Natl. Acad. Sci. U.S.A. 106, 11478-11483.
27. Todorov, E. (2009c). "Eigenfunction approximation methods for linearly-solvable optimal control problems," in Proceedings of the 2nd IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (Nashville, TN), 161-168.
28. Zhong, M., and Todorov, E. (2011). "Aggregation methods for lineary-solvable Markov decision process," in Proceedings of the World Congress of the International Federation of Automatic Control (Milano).