Efficient planning under uncertainty with macro-actions

Journal of Artificial Intelligence Research

Volumn 40, Issue , 2011, Pages 523-570

  He, Ruijie ^a   Brunskill, Emma ^b   Roy, Nicholas ^a  

^a MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ACTIVE AREA; CONTROL PERFORMANCE; EFFICIENT PLANNING; FORMAL ANALYSIS; LOOK-AHEAD; MULTI-STEP; NOVEL METHODS; PARTIALLY OBSERVABLE ENVIRONMENTS; POSTERIOR DISTRIBUTIONS; PRIMITIVE ACTIONS; SCIENTIFIC EXPLORATION; SEQUENCE OF ACTIONS; SIMULATION EXPERIMENTS;

ALGORITHMS; BUDGET CONTROL; EXPERIMENTS; ROBOTICS;

ROBOT PROGRAMMING;

EID: 79956361567     PISSN: None     EISSN: 10769757     Source Type: Journal    
DOI: 10.1613/jair.3171     Document Type: Article

References (52)

1. Bachrach, A., He, R., & Roy, N. (2009). Autonomous flight in unstructured and unknown indoor environments. In Proceedings of the European Micro Aerial Vehicle (EMAV) Conference.
2. Barndorff-Nielsen, O. (1979). Information and exponential families in statistical theory. Bulletin of the American Mathematics Society, 273, 667-668.
3. Bellingham, J., Richards, A., & How, J. (2002). Receding horizon control of autonomous aerial vehicles. In Proceedings of the American Control Conference (ACC), Vol. 5, pp. 3741-3746.
4. Bertsekas, D. (2007). Dynamic Programming and Optimal Control, vol. 1 & 2, 2nd. Athena Scientific.
5. Bonet, B., & Geffner, H. (2009). Solving POMDPs: RTDP-Bel vs. point-based algorithms. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pp. 1641- 1646.
6. Brooks, A., Makarenko, A., Williams, S., & Durrant-Whyte, H. (2006). Parametric POMDPs for planning in continuous state spaces. Robotics and Autonomous Systems, 54(11), 887-897. (Pubitemid 44572669)
7. Brunskill, E., Kaelbling, L., Lozano-Perez, T., & Roy, N. (2008). Continuous-state POMDPs with hybrid dynamics. In Proceedings of the International Symposium on Artificial Intelligence and Mathematics (ISAIM).
8. Charlin, L., Poupart, P., & Shioda, R. (2007). Automated hierarchy discovery for planning in partially observable environments. In Advances in Neural Information Processing Systems (NIPS).
9. Durbin, J., & Koopman, S. (2000). Time series analysis of non-Gaussian observations based on state space models from both classical and Bayesian perspectives. Journal of the Royal Statistical Society: Series B (Methodological), 62(1), 3-56.
10. Erez, T., & Smart, W. (2010). A Scalable Method for Solving High-Dimensional Continuous POMDPs Using Local Approximation. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI).
11. Foka, A., & Trahanias, P. (2007). Real-time hierarchical POMDPs for autonomous robot navigation. Robotics and Autonomous Systems, 55(7), 561-571. (Pubitemid 46777274)
12. Hansen, E., & Zhou, R. (2003). Synthesis of hierarchical finite-state controllers for POMDPs. In Proceedings of the Thirteenth International Conference on Automated Planning and Scheduling (ICAPS).
13. He, R., Bachrach, A., Achtelik, M., Geramifard, A., Gurdan, D., Prentice, S., Stumpf, J., & Roy, N. (2010a). On the design and use of a micro air vehicle to track and avoid adversaries. International Journal of Robotics Research, 29(5), 529-546.
14. He, R., Brunskill, E., & Roy, N. (2010b). PUMA: Planning under uncertainty with macro-actions. In Proceedings of the Association for the Advancement of Artificial Intelligence (AAAI).
15. He, R., Prentice, S., & Roy, N. (2008). Planning in information space for a quadrotor helicopter in GPS-denied environments. In Proceedings of the International Conference of Robotics and Automation (ICRA), pp. 1814-1820.
16. Hoeffding, W. (1963). Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58(301), 13-30.
17. Hoey, J., & Poupart, P. (2005). Solving POMDPs with continuous or large discrete observation spaces. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI).
18. Hsiao, K., Kaelbling, L., & Lozano-Pérez, T. (2010). Task-driven tactile exploration. In Proceedings of Robotics: Science and Systems (RSS).
19. Hsiao, K., Lozano-Pérez, T., & Kaelbling, L. (2008). Robust belief-based execution of manipulation programs. In Proceedings of the Workshop on the Algorithmic Foundations of Robotics (WAFR).
20. Kalman, R. E. (1960). A new approach to linear filtering and prediction problems. Transactions of the ASME-Journal of Basic Engineering, 82(Series D), 35-45.
21. Kearns, M., Mansour, Y., & Ng, A. (2002). A sparse sampling algorithm for near-optimal planning in large Markov decision processes. Machine Learning, 49(2-3), 193-209.
22. Krause, A., & Guestrin, C. (2007). Near-optimal observation selection using submodular functions. In Proceedings of the National Conference on Artificial Intelligence (AAAI), Vol. 22, pp. 1650-1654. (Pubitemid 350149805)
23. Kreucher, C., Hero III, A., Kastella, K., & Chang, D. (2004). Efficient methods of non-myopic sensor management for multitarget tracking. In Proceedings of the IEEE Conference on Decision and Control (CDC), Vol. 1, pp. 722-727. (Pubitemid 40291418)
24. Kurniawati, H., Du, Y., Hsu, D., & Lee, W. (2009). Motion planning under uncertainty for robotic tasks with long time horizons. In Proceedings of the International Symposium of Robotics Research (ISRR).
25. Kurniawati, H., Hsu, D., & Lee, W. (2008). SARSOP: Efficient point-based POMDP planning by approximating optimally reachable belief spaces. In Proceedings of the Robotics: Science and Systems (RSS).
26. Kuwata, Y., & How, J. (2004). Three dimensional receding horizon control for UAVs. In Proceedings of the AIAA Guidance, Navigation, and Control Conference and Exhibit (GNC), pp. 16-19.
27. Littman, M., Cassandra, A., & Kaelbling, L. (1995). Learning policies for partially observable environments: Scaling up. In Proceedings of the Twlfth International Conference on Machine Learning (ICML), pp. 362-370.
28. Mahadevan, S., Marchalleck, N., Das, T., & Gosavi, A. (1997). Self-improving factory simulation using continuous-time average-reward reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML), pp. 202-210.
29. Mayne, D. Q., Rawlings, J. B., Rao, C. V., & Scokaert, P. O. M. (2000). Constrained model predictive control: Stability and optimality. Automatica, 36, 789-814. (Pubitemid 30587683)
30. McAllester, D., & Singh, S. (1999). Approximate planning for factored POMDPs using belief state simplification. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI), pp. 409-416.
31. McGovern, A. (1998). acQuire-macros: An algorithm for automatically learning macro-actions. In NIPS 98 Workshop on Abstraction and Hierarchy in Reinforcement Learning.
32. Paquet, S., Chaib-draa, B., & Ross, S. (2006). Hybrid POMDP algorithms. In Workshop on Multi- Agent Sequential Decision Making in Uncertain Domains (MSDM), pp. 133-147.
33. Paquet, S., Tobin, L., & Chaib-draa, B. (2005). An online POMDP algorithm for complex multiagent environments. In Proceedings of the Conference on Autonomous agents and Multiagent systems (AAMAS), pp. 970-977.
34. Patek, S., Breton, M., Chen, Y., Solomon, C., & Kovatchev, B. (2007). Linear quadratic gaussianbased closed-loop control of type 1 diabetes. Journal of Diabetes Science and Technology, 1.
35. Pineau, J., Gordon, G., & Thrun, S. (2003a). Point-based value iteration: An anytime algorithm for POMDPs. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), Vol. 18, pp. 1025-1032.
36. Pineau, J., Gordon, G., & Thrun, S. (2003b). Policy-contingent abstraction for robust robot control. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI).
37. Platt, R., Tedrake, R., Lozano-Perez, T., & Kaelbling, L. (2010). Belief space planning assuming maximum likelihood observations. In Proceedings of Robotics: Science and Systems (RSS).
38. Porta, J., Vlassis, N., Spaan, M., & Poupart, P. (2006). Point-based value iteration for continuous POMDPs. Journal of Machine Learning Research, 7, 2329-2367. (Pubitemid 44708007)
39. Poupart, P. (2005). Exploiting Structure to Efficiently Solve Large Scale Partially Observable Markov Decision Processes. Ph.D. thesis, University of Toronto.
40. Richards, A., Kuwata, Y., & How, J. (2003). Experimental demonstrations of real-time MILP control. In Proceeding of the AIAA Guidance, Navigation, and Control Conference (GNC).
41. Ross, S., & Chaib-draa, B. (2007). AEMS: An anytime online search algorithm for approximate policy refinement in large POMDPs. In Proceedings of the International Joint Conference in Artificial Intelligence (IJCAI), pp. 2592-2598.
42. Ross, S., Pineau, J., Paquet, S., & Chaib-draa, B. (2008a). Online planning algorithms for POMDPs. Journal of Artificial Intelligence Research, 32(1), 663-704.
43. Ross, S., Chaib-draa, B., & Pineau, J. (2008b). Bayesian reinforcement learning in continuous POMDPs with application to robot navigation. In Proceedings of the International Conference on Robotics and Automation (ICRA). IEEE.
44. Scott, A., Harris, Z., & Chong, E. (2009). A POMDP framework for coordinated guidance of autonomous UAVs for multitarget tracking. EURASIP Journal on Advances in Signal Processing, 2009, 1-17.
45. Smallwood, R., & Sondik, E. (1973). The optimal control of partially observable Markov processes over a finite horizon. Operations Research, 21(5), 1071-1088.
46. Smith, T., & Simmons, R. (2005). Point-based POMDP algorithms: Improved analysis and implementation. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI).
47. Stolle, M., & Precup, D. (2002). Learning options in reinforcement learning. Lecture Notes in Computer Science, 212-223.
48. Sutton, R., Precup, D., & Singh, S. (1999). Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112, 181-211. (Pubitemid 32079890)
49. Theocharous, G., & Kaelbling, L. (2003). Approximate planning in POMDPs with macro-actions. In Advances in Neural Processing Information Systems (NIPS).
50. Triantafyllopoulos, K. (2003). On the central moments of the multidimensional Gaussian distribution. The Mathematical Scientist, 28, 125-128. (Pubitemid 38126665)
51. West, M., Harrison, P., & Migon, H. (1985). Dynamic generalized linear models and Bayesian forecasting. Journal of the American Statistical Association, 80(389), 73-83.
52. Yu, C., Chuang, J., Gerkey, B., Gordon, G., & Ng, A. (2005). Open-loop plans in multi-robot POMDPs. Tech. rep., Stanford University.