Approximate value iteration with temporally extended actions

Journal of Artificial Intelligence Research

Volumn 53, Issue , 2015, Pages 375-438

  Mann, Timothy A ^a   Mannor, Shie ^a   Precup, Doina ^b  

^a TECHNION ISRAEL INSTITUTE OF TECHNOLOGY   (Israel)

^b MCGILL UNIVERSITY   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

NAVIGATION; REINFORCEMENT LEARNING;

APPROXIMATION ERRORS; CONVERGENCE RATES; DIFFERENT DOMAINS; EFFICIENT PLANNING; FAST CONVERGENCE; FASTER CONVERGENCE; FITTED VALUE ITERATION; PRIMITIVE ACTIONS;

ITERATIVE METHODS;

EID: 84938498958     PISSN: 10769757     EISSN: None     Source Type: Journal    
DOI: 10.1613/jair.4676     Document Type: Article

References (51)

1. Barry, J. L., Kaelbling, L. P., & Lozano-Prez, T. (2011). DetH∗: Approximate Hierarchical Solution of Large Markov Decision Processes. In International Joint Conference on Artificial Intelligence.
2. Bertsekas, D. P., & Tsitsiklis, J. (1996). Neuro-dynamic programming. Athena Scientific.
3. th Conference on Uncertainty in Artificial Intelligence (UAI-08).
4. Chassin, D. P., Fuller, J. C., & Djilali, N. (2014). GridLAB-D: An agent-based simulation framework for smart grids. Journal of Applied Mathematics, 2014.
5. th International Conference on Autonomous Agents and Multiagent Systems, pp. 709-714.
6. Dietterich, T. G., Taleghan, M. A., & Crowley, M. (2013). PAC optimal planning for invasive species management: Improved exploration for reinforcement learning from simulatordefined MDPs. In Proceedings of the National Conference on Artificial Intelligence.
7. Dijkstra, E. (1959). A note on two problems in connexion with graphs. Numerische Mathematik, 1 (1), 269-271.
8. Farahmand, A., Ghavamzadeh, M., Szepesvári, C., & Mannor, S. (2008). Regularized fitted Q-iteration: Application to planning. In Recent Advances in Reinforcement Learning, pp. 55-68. Springer.
9. Farahmand, A., Munos, R., & Szepesvári, C. (2010). Error propagation for approximate policy and value iteration. In Advances in Neural Information Processing Systems.
10. th International Joint Conference on Autonomous Agents and Multiagent Systems, pp. 720-727.
11. Hart, P., Nilsson, N., & Raphael, B. (1968). A formal basis for the heuristic determination of minimum cost paths. Systems Science and Cybernetics, IEEE Transactions on, 4 (2), 100-107.
12. He, R., Brunskill, E., & Roy, N. (2011). Efficient planning under uncertainty with macroactions. Journal of Artificial Intelligence Research, 40, 523-570.
13. Hoey, J., St-Aubin, R., Hu, A. J., & Boutilier, C. (1999). SPUDD: Stochastic Planning Using Decision Diagrams. In Proceedings of Uncertainty in Artificial Intelligence, Stockholm, Sweden.
14. Iba, G. A. (1989). A heuristic approach to the discovery of macro-operators. Machine Learning, 3, 285-317.
15. th International Conference on Machine Learning.
16. Kearns, M., Mansour, Y., & Ng, A. Y. (2002). A sparse sampling algorithm for near-optimal planning in large Markov decision processes. Machine Learning, 49 (2-3), 193-208.
17. Kocsis, L., & Szepesvári, C. (2006). Bandit based Monte-Carlo Planning. In Machine Learning: ECML-2006, pp. 282-293. Springer.
18. Konidaris, G., & Barto, A. (2009). Skill discovery in continuous reinforcement learning domains using skill chaining. In Advances in Neural Information Processing Systems 22, pp. 1015-1023.
19. Konidaris, G., & Barto, A. G. (2007). Building portable options: Skill transfer in reinforcement learning. In Proceedings of the International Joint Conference on Artificial Intelligence, Vol. 7, pp. 895-900.
20. Konidaris, G., Kuindersma, S., Barto, A., & Grupen, R. (2010). Constructing skill trees for reinforcement learning agents from demonstration trajectories. In Advances in Neural Information Processing Systems, pp. 1162-1170.
21. Lazanas, A., & Latombe, J.-C. (1992). Landmark-based robot navigation. Tech. rep., Stanford University.
22. th International Conference on Machine Learning.
23. th conference on Uncertainty in artificial intelligence, pp. 394-402.
24. st International Conference on Machine Learning.
25. Mann, T. A. (2014). Cyclic Inventory Management (CIM). https://code.google.com/p/rddlsim/source/browse/trunk/files/rddl2/examples/cim.rddl2. Accessed: 2015-06-29.
26. st International Conference on Machine Learning.
27. st International Conference on Machine learning, ICML '04, pp. 71-, New York, NY, USA. ACM.
28. th International Conference on Machine Learning, pp. 361-368, San Fransisco, USA.
29. th International Symposium on Inventories.
30. Munos, R. (2005). Error bounds for approximate value iteration. In Proceedings of the National Conference on Artificial Intelligence.
31. Munos, R., & Szepesvári, C. (2008). Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, 9, 815-857.
32. Peleg, D., & Schäffer, A. A. (1989). Graph spanners. Journal of Graph Theory, 13 (1), 99-116.
33. Peters, J., & Schaal, S. (2008). Reinforcement learning of motor skills with policy gradients. Neural Networks, 21, 682-691.
34. Precup, D., & Sutton, R. S. (1997). Multi-time models for temporally abstract planning. In Advances in Neural Information Processing Systems 10.
35. Precup, D., Sutton, R. S., & Singh, S. (1998). Theoretical results on reinforcement learning with temporally abstract options. In Machine Learning: ECML-1998, pp. 382-393. Springer.
36. Puterman, M. L. (1994). Markov Decision Processes - Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc.
37. Riedmiller, M. (2005). Neural fitted Q iteration-first experiences with a data efficient neural reinforcement learning method. In Machine Learning: ECML-2005, pp. 317-328. Springer.
38. Sanders, P., & Schultes, D. (2005). Highway hierarchies hasten exact shortest path queries. In Brodal, G., & Leonardi, S. (Eds.), Algorithms: ESA-2005, Vol. 3669 of Lecture Notes in Computer Science, pp. 568-579. Springer Berlin Heidelberg.
39. Scarf, H. (1959). The optimality of (s,S) policies in the dynamic inventory problem. Tech. rep. NR-047-019, Office of Naval Research.
40. th International Conference on Machine Learning, Edinburgh, United Kingdom.
41. Sethi, S. P., & Cheng, F. (1997). Optimality of (s,S) policies in inventory models with markovian demand. Operations Research, 45 (6), 931-939.
42. Shantia, A., Begue, E., & Wiering, M. (2011). Connectionist reinforcement learning for intelligent unit micro management in starcraft. In Proceedings of the International Joint Conference on Neural Networks, pp. 1794-1801. IEEE.
43. th International Conference on Machine Learning, Edinburgh.
44. st International Conference on Machine Learning, pp. 95-102, New York, NY, USA. ACM.
45. th International Conference on Autonomous Agents and Multiagent Systems, pp. 31-38.
46. Stolle, M., & Precup, D. (2002). Learning options in reinforcement learning. In Abstraction, Reformulation, and Approximation, pp. 212-223. Springer.
47. Stone, P., Sutton, R. S., & Kuhlmann, G. (2005). Reinforcement learning for robocup soccer keepaway. Adaptive Behavior, 13 (3), 165-188.
48. Sutton, R. S., Precup, D., & Singh, S. (1999). Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112 (1), 181-211.
49. th International Conference on Machine Learning.
50. nd International Conference on Machine Learning, pp. 816-823.
51. Yoon, S. W., Fern, A., & Givan, R. (2007). FF-Replan: A Baseline for Probabilistic Planning. In Proceedings of the International Conference on Automated Planning and Scheduling, Vol. 7, pp. 352-359.