Robust adaptive Markov decision processes: Planning with model uncertainty

IEEE Control Systems

Volumn 32, Issue 5, 2012, Pages 96-109

  Bertuccelli, Luca F ^a,b   Wu, Albert ^c   How, Jonathan P ^d,e,f  

^a UNITED TECHNOLOGIES RESEARCH CENTER   (United States)

^b not available   (United States)

^c CARNEGIE MELLON UNIVERSITY   (United States)

^d MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^e AIAA   (United States)

^f IEEE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ANTENNAS; HUMAN RESOURCE MANAGEMENT; MARKOV PROCESSES; UNCERTAINTY ANALYSIS;

AUTONOMOUS SYSTEMS; CO-OPERATIVE CONTROL; COMPLEX DECISION; MARKOV DECISION PROCESSES; MODEL UNCERTAINTIES; ROBUST ADAPTIVE; UNCERTAIN ENVIRONMENTS; VEHICLE SENSORS;

UNMANNED AERIAL VEHICLES (UAV);

EID: 84877947571     PISSN: 1066033X     EISSN: None     Source Type: Journal    
DOI: 10.1109/MCS.2012.2205478     Document Type: Article

References (31)

1. M. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming. Hoboken, NJ: Wiley, 2005.
2. H. Song, C. C. Liu, J. Lawarree, and R. Dahlgren, "Optimal electricity supply bidding by Markov decision process," IEEE Trans. Power Syst., vol. 15, no. 2, pp. 618-624, 2000.
3. M. Schal, Handbook of Markov Decision Processes: Methods and Applications, Chapter Markov Decision Processes in Finance and Dynamic Options. Norwell, MA: Kluwer, 2002.
4. B. Bethke, J. How, and J. Vian, "Group health management of UAV teams with applications to persistent surveillance," in Proc. American Controls Conf., 2008, pp. 3145-3150.
5. D. Akselrod, A. Sinha, and T. Kirubarajan, "Collaborative distributed sensor management for multitarget tracking using hierarchical Markov decision processes," Proc. SPIE, vol. 6699, pp. 1-14, 2007.
6. E. Rachelson, P. Fabiani, and F. Garcia, "Adapting an MDP planner to time-dependency: Case study on a UAV coordination problem," in Proc. 4th Workshop Planning and Plan Execution for Real-World Systems: Principles and Practices for Planning in Execution, 2009, pp. 1-8.
7. Z. Lian and A. Deshmukh, "Performance prediction of an unmanned airborne vehicle multiagent system," Eur. J. Oper. Res., vol. 172, no. 2, pp. 680-695, 2006. (Pubitemid 43332997)
8. S. Mannor, D. Simester, P. Sun, and J. Tsitsiklis, "Bias and variance approximation in value function estimates," Manage. Sci., vol. 52, no. 2, pp. 308-322, 2007. (Pubitemid 46326182)
9. C. C. White and H. K. Eldeib, "Markov decision processes with imprecise transition probabilities," Oper. Res., vol. 42, no. 4, pp. 739-749, 1994.
10. L. F. Bertuccelli, "Robust decision-making with model uncertainty in aerospace systems," Ph.D. dissertation, MIT, 2008.
11. E. Delage and S. Mannor, "Percentile optimization for Markov decision processes with parameter uncertainty," Oper. Res., vol. 58, no. 1, pp. 203-213, 2010.
12. S. Marcus, E. Fernandez-Gaucherand, D. Hernandez-Hernandez, S. Coraluppi, and P. Fard, "Risk-sensitive Markov decision processes," in Systems and Control in the Twenty-First Century, 1997.
13. A. Nilim and L. El Ghaoui, "Robust solutions to Markov decision problems with uncertain transition matrices," Oper. Res., vol. 53, no. 5, pp. 780-798, 2005. (Pubitemid 41525849)
14. G. Iyengar, "Robust dynamic programming," Math. Oper. Res., vol. 30, no. 2, pp. 257-280, 2005.
15. J. Bagnell, A. Y. Ng, and J. Schneider, "Solving uncertain Markov decision problems," Robotics Inst., Carnegie Mellon Univ., Pittsburgh, PA, Tech. Rep. CMU-RI-TR-01-25, 2001.
16. L. F. Bertuccelli and J. P. How, "Robust decision-making for uncertain Markov decision processes using sigma point sampling," in Proc. American Control Conf., 2008, pp. 5003-5008.
17. V. Gullapalli and A. G. Barto, "Convergence of indirect adaptive asynchronous value iteration algorithms," in Advances in Neural Information Processing Systems, 1994, pp. 695-695.
18. A. Moore and C. Atkeson, "Prioritized sweeping: Reinforcement learning with less data and less time," Mach. Learn., vol. 13, no. 1, pp. 103-130, 1991.
19. R. B. Israel, J. S. Rosenthal, and J. Z. Wei, "Finding generators for Markov chains via empirical transition matrices with applications to credit ratings," Math. Finance, vol. 11, no. 2, pp. 245-265, 2001.
20. Y. Liu, J. B. Cruz, and C. J. Schumacher, "Pop-up threat models for persistent area denial," IEEE Trans. Aerosp. Electron. Syst., vol. 43, no. 2, pp. 509-521, 2007. (Pubitemid 47308045)
21. B. Bethke, L. Bertuccelli, and J. P. How, "Experimental demonstration of MDP-based planning with model uncertainty," in Proc. AIAA Guidance Navigation and Control Conf., Aug. 2008, AIAA-2008-6322.
22. A. Barto, S. Bradtke, and S. Singh, "Learning to act using real-time dynamic programming," Artif. Intell., vol. 72, pp. 81-138, 1993.
23. A. Gelman, J. Carlin, H. Stern, and D. Rubin, Bayesian Data Analysis. London, U.K.: Chapman and Hall, 1995.
24. D. Fox, "Adapting the sample size in particle filters through KLD-sampling," Int. J. Robot. Res., vol. 22, no. 12, pp. 985, 2003.
25. A. Doucet, N. De Freitas, and N. Gordon, Sequential Monte Carlo Methods in Practice. New York: Springer-Verlag, 2001.
26. S. Julier and J. Uhlmann, "Unscented filtering and nonlinear estimation," Proc. IEEE, vol. 92, no. 3, pp. 401-422, 2004.
27. R. Jaulmes, J. Pineau, and D. Precup, "Learning in non-stationary partially observable Markov decision processes," in Proc. ECML Workshop Reinforcement Learning in Non-Stationary Environments, 2005, vol. 2, pp. 26.
28. V. Jilkov and X. Li, "Online Bayesian estimation of transition probabilities for Markovian jump systems," IEEE Trans. Signal Processing, vol. 52, no. 6, pp. 307-315, 2004.
29. Y. Bar Shalom, X. Rong Li, and T. Kirubarajan, Estimation with Applications to Tracking and Navigation. New York: Wiley Interscience, 2001.
30. R. W. Miller, "Asymptotic behavior of the Kalman filter with exponential aging," AIAA J., vol. 9, pp. 537-539, 1971.
31. D. Bertsekas, Dynamic Programming and Optimal Control. Athena Scientific, 2005.