Adaptive importance sampling technique for Markov chains using stochastic approximation

Operations Research

Volumn 54, Issue 3, 2006, Pages 489-504

  Ahamed, T P I ^a   Borkar, V S ^b   Juneja, S ^b  

^a TKM COLLEGE OF ENGINEERING   (India)

^b TATA INSTITUTE OF FUNDAMENTAL RESEARCH   (India)

Author keywords

Probability: Markov processes; Queues: Markovian; Simulation: efficiency

Indexed keywords

ADAPTIVE ALGORITHMS; APPROXIMATION THEORY; ASYMPTOTIC STABILITY; MARKOV PROCESSES; OPERATIONS RESEARCH; RANDOM PROCESSES;

PROBABILITY: MARKOV PROCESSES; QUEUES: MARKOVIAN; SIMULATION: EFFICIENCY; ZERO-VARIANCE ESTIMATOR;

SAMPLING;

EID: 33744819512     PISSN: 0030364X     EISSN: 15265463     Source Type: Journal    
DOI: 10.1287/opre.1060.0291     Document Type: Article

References (36)

1. Ahamed, T. P. I., V. S. Borkar, S. Juneja. 2006. Adaptive importance sampling technique for Markov chains using stochastic approximation. Online companion. Oper. Res. http://or.pubs.informs.org/Pages/collect.html.
2. Andradottir, S., D. P. Heyman, T. Ott. 1995. On the choice of alternative measures in importance sampling with Markov chains. Oper. Res. 43(3) 509-519.
3. Bertsekas, D. P., J. N. Tsitsiklis. 1989. Parallel and Distributed Computation. Prentice Hall, Englewood Cliffs, NJ.
4. Bertsekas, D. P., J. N. Tsitsiklis. 1996. Neuro-Dynamic Programming. Athena Scientific, Belmont, MA.
5. Booth, T. E. 1985. Exponential convergence for Monte Carlo particle transport. Trans. Amer. Nuclear Soc. 50 267-268.
6. Borkar, V. S. 1998. Asynchronous stochastic approximation. SIAM J. Control Optim. 36 840-851.
7. Borkar, V. S. 2002. On the lock-in probability of stochastic approximation. Combinatorics, Probab. Comput. 11 11-20.
8. Borkar, V. S., S. P. Meyn. 2000. The O.D.E. method for convergence of stochastic approximation and reinforcement learning. SIAM J. Control Optim. 38 447-469.
9. Borkar, V. S., K. Soumyanath. 1997. An analog scheme for fixed point computation, Part 1: Theory. IEEE Trans. Circuits Systems 1: Fundamental Theory Appl. 44 351-355.
10. Bratley, P., B. L. Fox, L. E. Schrage. 1987. A Guide to Simulation, 2nd ed. Springer-Verlag, New York.
11. Cao, X. R., H. F. Chen. 1997. Perturbation realization, potentials, and sensitivity analysis of Markov processes. IEEE Trans. Automatic Control 42 1382-1393.
12. De Boer, P. T. 2001. Analysis and efficient simulation of queueing models of telecommunication systems. Ph.D. thesis, University of Twente, Enschede, The Netherlands.
13. De Boer, P. T., V. F. Nicola, R. Y. Rubinstein. 2000. Adaptive importance sampling simulation of queueing networks. J. A. Joines, R. R. Barton, K. Kang, P. A. Fishwick, eds. Proc. 2000 Winter Simulation Conf. IEEE Press, Piscataway, NJ, 646-655.
14. Derevitskii, D. P., A. L. Fradkov. 1974. Two models for analyzing the dynamics of adaptation algorithms. Automation Remote Control 35 59-67.
15. Desai, P. Y., P. W. Glynn. 2001. A Markov chain perspective on adaptive Monte Carlo algorithms. B. A. Peters, J. S. Smith, D. J. Medeiros, M. H. Rohrer, eds. Proc. 2001 Winter Simulation Conf. IEEE Press, Piscataway, NJ, 379-384.
16. Fishman, G. S. 1996. Monte Carlo: Concepts, Algorithms, and Applications. Springer, New York.
17. Glasserman, P., S. G. Kau. 1995. Analysis of an importance sampling estimator for tandem queues. ACM TOMACS 5 22-42.
18. Glynn, P. W., D. L. Iglehart. 1989. Importance sampling for stochastic simulations. Management Sci. 35 1367-1392.
19. Glynn, P. W., D. L. Iglehart. 1993. Conditions for the applicability of the regenerative method. Management Sci. 39 1108-1111.
20. Golub, G. H., C. F. Van Loan. 1996. Matrix Computations, 3rd ed. Johns Hopkins University Press, Baltimore, MD.
21. Heidelberger, P. 1995. Fast simulation of rare events in queuing and reliability models. ACM Trans. Model. Comput. Simulation 5(1) 43-85.
22. Hsieh, M., P. W. Glynn. 2002. Confidence regions for stochastic approximation algorithms. E. Yucesan, C.-H. Chen, J. L. Snowdon, J. M. Charnes, eds. Proc. 2002 Winter Simulation Conf. IEEE Press, Piscataway, NJ, 370-376.
23. Juneja, S., V. Nicola. 2005. Efficient simulation of buffer overflow probabilities in Jackson networks with feedback. ACM TOMACS 15(4) 281-315.
24. Juneja, S., P. Shahabuddin. 2005. Rare event simulation techniques: An introduction and recent advances. Shane Henderson, Barry Nelson, eds. Handbook on Simulation. Elsevier. Forthcoming.
25. Kollman, C., K. Baggerly, D. Cox, R. Picard. 1999. Adaptive importance sampling on discrete Markov chains. Ann. Appl. Probab. 9 391-412.
26. Konda, V. R., V. S. Borkar. 1999. Actor-critic-type learning algorithms for Markov decision processes. SIAM J. Control Optim. 38 94-123.
27. Kuruganti, I., S. Strickland. 1995. Optimal importance sampling for Markovian systems. Proc. 1995 IEEE Conf. Systems, Man Cybernetics. Vancouver, British Columbia, Canada, 195-200.
28. Kuruganti, I., S. Strickland. 1996. Importance sampling for Markov chains: Computing variance and determining optimal measures. Proc. 1996 Winter Simulation Conf. IEEE Press, Piscataway, NJ, 273-280.
29. Kushner, H. J., G. Yin. 1997. Stochastic Approximation Algorithms and Applications. Springer Verlag, New York.
30. Precup, D., R. S. Sutton, S. P. Singh. 2000. Eligibility traces for off-policy policy evaluation. Proc. 17th Internat. Conf. Machine Learn. (ICML'00). Morgan Kaufman, Stanford, CA, 759-766.
31. Randhawa, R. S., S. Juneja. 2004. Simulating rare events by combining temporal difference methods and importance sampling. ACM TOMACS 14 1-30.
32. Singh, S., P. Dayan. 1998. Analytical mean squared error curves for temporal difference learning. Machine Learn. 32 5-40.
33. Su, Y., M. C. Fu. 2000. Importance sampling in derivatives security pricing. J. A. Joines, R. R. Barton, K. Jang, P. A. Fishwick, eds. Proc. 2000 Winter Simulation Conf. IEEE Press, Piscataway, NJ, 587-596.
34. Su, Y., M. C. Fu. 2002. Optimal importance sampling in securities pricing. J. Comput. Finance 5 27-50.
35. Tsitsiklis, J. T. 1994. Asynchronous stochastic approximation and Q-learning. Machine Learn. 16 185-202.
36. Vázquez-Abad, F. J., D. Dufresne. 1998. Accelerated simulation for pricing Asian options. D. J. Medeiros, E. F. Watson, J. S. Carson, M. S. Manivannan, eds. Proc. 1998 Winter Simulation Conf. IEEE Press, Piscataway, NJ, 1493-1500.