Combining importance sampling and temporal difference control variates to simulate Markov chains

ACM Transactions on Modeling and Computer Simulation

Volumn 14, Issue 1, 2004, Pages 1-30

  Randhawa, R S ^a,c   Juneja, S ^b  

^a STANFORD UNIVERSITY   (United States)

^b TATA INSTITUTE OF FUNDAMENTAL RESEARCH   (India)

^c STANFORD UNIVERSITY   (United States)

Author keywords

Importance sampling; Markov chains; Rare events; Stochastic approximation; Temporal difference methods; Variance reduction

Indexed keywords

IMPORTANCE SAMPLING; RARE EVENTS; STOCHASTIC APPROXIMATION; TEMPORAL DIFFERENCE METHODS; VARIANCE REDUCTION;

ALGORITHMS; APPROXIMATION THEORY; COMPUTER SIMULATION; SAMPLING; SET THEORY;

MARKOV PROCESSES;

EID: 3042683131     PISSN: 10493301     EISSN: None     Source Type: Journal    
DOI: 10.1145/974734.974735     Document Type: Article

References (29)

1. ANDRADOTTIR, S., HEYMAN, D., AND OTT, T. J. 1993. Variance reduction through smoothing and control variates for Markov chain simulations. ACM Trans. Mod. Comput. Sim. 3, 3, 167-189.
2. ANDRADOTTIR, S., HEYMAN, D., AND OTT, T. J. 1995. On the choice of alternative measures in importance sampling with Markov chains. Operat. Res. 43, 3, 509-519.
3. BERTSEKAS, D. AND TSITSIKLIS, J. 1996. Neuro-Dynamic Programming. Athena Scientific, Belmont, MA.
4. BILLINOSLEY, P. 1995. Probability and Measure. John Wiley & Sons, New York, NY.
5. BORKAR, V. AND MEYN, S. 2000. The o.d.e. method for convergence of stochastic approximation and reinforcement learning. SIAM J. Contr. Opt. 38, 2, 347-469.
6. BRATLEY, P., Fox, B., AND SCHRAGE, L. 1987. A Guide to Simulation. Springer-Verlag, New York, NY.
7. CRANE, M. AND IGLEHART, D. 1975. Simulating stable stochastic systems, iii: Regenerative processes and discrete-event simulations. Operat. Res. 23, 33-45.
8. DEMBO, A. AND ZEITOUNI, O. 1992. Large Deviations Techniques and Applications. Jones and Bartlett, Boston, MA.
9. FRATER, M., LENNON, T., AND ANDERSON, B. 1991. Optimally efficient estimation of the statistics of rare events in queuing networks. IEEE Trans. Automat. Contr. 36, 12, 1395-1404.
10. GLASSERMAN, P. AND KOU, S. 1995. Analysis of an importance sampling estimator for tandem queues. ACM Trans. Mod. Comput. Sim. 5, 1, 22-42.
11. GLASSERMAN, P. AND WANG, Y. 1997. Counterexamples in importance sampling for large deviations probabilities. Ann. Appl. Prob. 7, 731-746.
12. GLYNN, P. AND IGLEHART, D. 1989. Importance sampling for stochastic simulations. Manage. Sci. 35, 11, 1367-1392.
13. HEIDELBERGER, P. 1977. Variance reduction techniques for simulating Markov chains. In Proceedings of the 1977 Winter Simulation Conference.
14. HEIDELBERGER, P. 1980a. Variance reduction techniques for the simulation of Markov processes, I: Multiple estimates. IBM J. Res. Develop. 24, 570-581.
15. HEIDELBERGER, P. 1980b. Variance reduction techniques for the simulation of Markov processes, ii: Matrix iterative methods. Acta Informatica 13, 21-37.
16. HEIDELBERGER, P. 1995. Fast simulation of rare events in queuing and reliability models. ACM Trans. Model. Comput. Sim. 5, 1, 43-85.
17. HSEIH, M. AND GLYNN, P. 2002. Confidence regions for stochastic approximation algorithms. In Proceedings of the 2002 Winter Simulation Conference. 370-376.
18. JUNEJA, S. 2001. Importance sampling and the cyclic approach. Operat. Res. 49, 6, 900-912.
19. JUNEJA, S. 2003. Efficient rare event simulation using importance sampling: An introduction. In Computational Mathematics, Modelling and Algorithms, J. C. Misra, Ed. Narosa Publishing House, New Delhi, India, 357-396.
20. JUNEJA, S. AND SHAHABUDDIN, P. 2001. Efficient simulation of Markov chains with small transition probabilities. Manage. Sci. 47, 4, 547-562.
21. KUSHNER, H. AND YIN, G. 1997. Stochastic Approximation Algorithms and Applications. Springer-Verlag, New York, NY.
22. PAREKH, S. AND WALRAND, J. 1989. A quick simulation method for excessive backlogs in networks of queues. IEEE Trans. Automat. Contr. 34, 1, 54-66.
23. PRECUP, D., SUTTON, R., AND SINGH, S. 2000. Eligibility traces for off-policy policy evaluation. In Proceedings of the 17th International Conference on Machine Learning (ICML'00). Morgan Kaufman, San Francisco, CA, 759-766.
24. RUBINSTEIN, R. 1997. Optimization of computer simulation models with rare events. European J. Operat. Res. 99, 89-112.
25. RUBINSTEIN, R. 1999. Rare event simulation via cross-entropy and importance sampling. Second Workshop on Rare Event Simulation (RESIM'99). 1-17.
26. SINGH, S. AND DAYAN, P. 1998. Analytical mean squared error curves for temporal difference learning. Mach. Learn. 32, 5-40.
27. SINGH, S. AND SUTTON, R. 1996. Reinforcement learning with replacing eligibility traces. Mach. Learn. 22, 123-158.
28. SUTTON, R. 1988. Learning to predict by the methods of temporal differences. Mach. Learn. 3, 9-44.
29. SUTTON, R. AND BARTO, A. 1998. Reinforcement Learning: An Introduction. The MIT Press, Cambridge, MA.