Shannon meets bellman: Feature based markovian models for detection and optimization

Proceedings of the IEEE Conference on Decision and Control

Volumn , Issue , 2008, Pages 5558-5564

  Meyn, Sean ^a,b   Mathew, George ^c  

^a Department of Electrical and Computer Engineering ^*

^b UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

^c University of California at Santa Barbara   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

GAUSSIAN DISTRIBUTION; HIDDEN MARKOV MODELS; LEARNING ALGORITHMS; TRELLIS CODES;

DYNAMIC PROGRAMMING EQUATIONS; FINITE DIMENSIONAL; GAUSSIAN MIXTURE MODEL; GENERAL APPLICATIONS; MARGINAL DISTRIBUTION; MATRIX CALCULATIONS; PROJECTIVE METHODS; SIMPLE LEARNING ALGORITHMS;

DYNAMIC PROGRAMMING;

EID: 62949191986     PISSN: 07431546     EISSN: 25762370     Source Type: Conference Proceeding    
DOI: 10.1109/CDC.2008.4739405     Document Type: Conference Paper

References (29)

1. E. Altman. Constrained Markov decision processes. Stochastic Modeling. Chapman & Hall/CRC, Boca Raton, FL, 1999.
2. D.P. Bertsekas and J. N. Tsitsiklis. Neuro-Dynamic Programming. Atena Scientific, Cambridge, Mass, 1996.
3. V. S. Borkar. Average cost dynamic programming equations for controlled Markov chains with partial observations. SIAM J. Control Optim., 39(3):673-681 (electronic), 2000.
4. V. S. Borkar. Dynamic programming for ergodic control with partial observations. Stoch. Proc. Applns., 103(2):293-310, 2003.
5. V. S. Borkar and S. P. Meyn. The O.D.E. method for convergence of stochastic approximation and reinforcement learning. SIAM J. Control Optim., 38(2):447-469, 2000. (also presented at the IEEE CDC, December, 1998).
6. A. Chorin, O. Hald, and R. Kupferman. Optimal prediction with memory. Physica D, 166:239-257, 2002.
7. A. J. Chorin, O. H. Hald, and R. Kupferman. Non-Markovian optimal prediction, 2001.
8. M.D. Donsker and S.R.S. Varadhan. Asymptotic evaluation of certain Markov process expectations for large time. I. II. Comm. Pure Appl. Math., 28:1-47; ibid. 28 (1975), 279-301, 1975.
9. R. J. Elliott, L. Aggoun, and J. B. Moore. Hidden Markov models, volume 29 of Applications of Mathematics (New York). Springer-Verlag, New York, 1995. Estimation and control.
10. E. A. Feinberg and A. Shwartz, editors. Handbook of Markov decision processes. International Series in Operations Research & Management Science, 40. Kluwer Academic Publishers, Boston, MA, 2002. Methods and applications.
11. R. A. Howard. Dynamic Programming and Markov Processes. John Wiley and Sons/MIT Press, New York, NY, 1960.
12. V. R. Konda and V. S. Borkar. Actor-critic-type learning algorithms for Markov decision processes. SIAM J. Control Optim., 38(1):94-123 (electronic), 1999.
13. V. R. Konda and J. N. Tsitsiklis. On actor-critic algorithms. SIAM J. Control Optim., 42(4):1143-1166 (electronic), 2003.
14. I. Kontoyiannis and S. P. Meyn. Spectral theory and limit theorems for geometrically ergodic Markov processes. Ann. Appl. Probab., 13:304-362, 2003. Presented at the INFORMS Applied Probability Conference, NYC, July, 2001.
15. S. Lippman. Applying a new device in the optimization of exponential queueing systems. Operations Res., 23:687-710, 1975.
16. G. Mathew and S. Meyn. Learning macroscopic dynamics for optimal prediction. Submitted to 2008 IEEE Conf. on Dec. and Control. Preliminary version presented at Info. Thy. & Appl. at ITA, UCSD 2008.
17. F. S. Melo, S. Meyn, and M. Isabel Ribeiro. An analysis of reinforcement learning with function approximation. In Proceedings of ICML, pages 664-671, 2008.
18. Francisco S. Melo and M. Isabel Ribeiro. Convergence of Q-learning with linear function approximation. In Proceedings of the 2007 European Control Conference, pages 2671-2678, 2007.
19. S. P. Meyn. The policy iteration algorithm for average reward Markov decision processes with general state space. IEEE Trans. Automat. Control, 42(12):1663-1680, 1997.
20. S. P. Meyn. Control techniques for complex networks. Cambridge University Press, Cambridge, 2007.
21. S. P. Meyn and R. L. Tweedie. Markov Chains and Stochastic Stability. Springer-Verlag, London, second edition, 1993. 2008 Edition to appear, Cambride University Press, Cambridge Mathematical Library. 1993 edition online: http://black.csl.uiuc.edu/~meyn/pages/book.html.
22. H. Mori. Transport, collective motion, and brownian motion. Progress of Theoretical Physics, 33:423-455, 1965.
23. Dirk Ormoneit and Śaunak Sen. Kernel-based reinforcement learning. Mach. Learn., 49(2-3):161-178, 2002.
24. P. J. Schweitzer. Perturbation theory and finite Markov chains. J. Appl. Prob., 5:401-403, 1968.
25. C.E. Shannon. A mathematical theory of communication. Bell System Tech. J., 27:379-423, 623-656, 1948.
26. R.S. Sutton and A.G. Barto. Reinforcement Learning: An Introduction. MIT Press, on-line edition at http://www.cs.ualberta.ca/ ~sutton/book/the-book.html edition, 1998.
27. L. Tassiulas and A. Ephremides. Jointly optimal routing and scheduling in packet radio networks. IEEE Trans. Inform. Theory, 38(1):165-168, 1992.
28. J. N. Tsitsiklis and B. Van Roy. Feature-based methods for large scale dynamic programming. Mach. Learn., 22(1-3):59-94, 1996.
29. R. Zwanzig. Nonequilibrium Statistical Mechanics. Oxford University Press, Oxford, England, 2001.