Measure-valued differentiation for stationary markov chains

Mathematics of Operations Research

Volumn 31, Issue 1, 2006, Pages 154-172

  Heidergott, Bernd ^a   Hordijk, Arie ^b   Weisshaupt, Heinz ^c  

^a VU UNIVERSITY AMSTERDAM   (Netherlands)

^b LEIDEN UNIVERSITY   (Netherlands)

^c UNIVERSITY OF VIENNA   (Austria)

Author keywords

Markov chains; Measure valued differentiation; Normed ergodicity; Perturbation analysis; Score function

Indexed keywords

DIFFERENTIAL EQUATIONS; FUNCTION EVALUATION; MAPPING; PARAMETER ESTIMATION; PERTURBATION TECHNIQUES; QUEUEING THEORY;

GRADIENT ESTIMATORS; MEASURE-VALUED DIFFERENTIATION; NORMED ERGODICITY; SCORE FUNCTION;

MARKOV PROCESSES;

EID: 33646741306     PISSN: 0364765X     EISSN: 15265471     Source Type: Journal    
DOI: 10.1287/moor.1050.0171     Document Type: Article

References (31)

1. Baccelli, F., D. Hong. 2000. Analytic expansions of (max,+) Lyapunov exponents. Ann. Appl. Probab. 10 779-827.
2. Braker, J. 1991. Max-algebra modelling and analysis of time-dependent transportation networks. Proc. 1st Eur: Control Conf., Grenoble, France, 1831-1836. Hermes, Paris, France.
3. Brémaud, P. 1992. Maximal coupling and rare perturbation analysis. Queueing Systems: Theory Appl. 11 307-333.
4. Brémaud, P., F. Vázquez-Abad. 1992. On the pathwise computation of derivatives with respect to the rate of a point process: The phantom RFA method. Queueing Systems: Theory Appl. 10 49-270.
5. Borovkov, A. A., A. Hordijk, 2000. On normed ergodicity of Markov chains. Technical report MI 2000-40 Leiden University, Leiden, The Netherlands.
6. Borovkov, A. A., A. Hordijk. 2004. Characterization and sufficient conditions for normed ergodicity of Markov chains. Adv. Appl. Probab. 36 227-242.
7. Cohen, J. W. 1969. The Single Server Queue. North-Holland, Amsterdam, The Netherlands.
8. Dekker, R., A. Hordijk. 1988. Average, sensitive and Blackwell optimal policies in denumerable Markov decision chains with unbounded rewards. Math. Oper. Res. 13 395-421.
9. Dekker, R., A. Hordijk, F. M. Spieksma. 1994. On the relation between recurrence and ergodicity properties in denumerable Markov decision chains. Math. Oper. Res. 19 539-559.
10. Glynn, P. 1990. Likelihood ratio gradient estimation for stochastic systems. Comm. ACM 33 75-84.
11. Glynn, P., P. L'Ecuyer, 1995. Likelihood ratio gradient estimation for stochastic recursions. Adv. Appl. Probab. 27 1019-1053.
12. Glynn, P., P. L'Ecuyer, M. Adès. 1991. Gradient estimation for ratios. B. Nelson, W. Kelton, G. Clark, eds. Proc. 1991 Winter Simulation Conf. Winter Simulation Conference. San Diego, CA, 986-993.
13. Heidergott, B., A. Hordijk, H. Weisshaupt. 2003. Derivatives of Markov kernels and their Jordan decomposition. EURANDOM Report 2003-001, EURANDOM, Eindhoven, The Netherlands.
14. Heidergott, B., A. Hordijk, H. Weisshaupt. 2002. Measure-valued differentaion for stationary Markov chains. EURANDOM Report 2002-027, EURANDOM, Eindhoven, The Netherlands.
15. Heidergott, B., X. Cao, 2002. A note on the relation between weak derivatives and perturbation realization. IEEE Trans. Automatic Control 47 1112-1115.
16. Heidergott, B., A. Hordijk. 2003. Taylor series expansions for stationary Markov chains. Adv. Appl. Probab. 35 1046-1070.
17. Heidergott, B., A. Hordijk. 2004. Single-run gradient estimators via measure-valued differentiation. IEEE Trans. Automatic Control 49 1843-1846.
18. Heidergott, B., F. Vázquez-Abad. 2000. Measure-valued differentiation for stochastic processes: The finite horizon case. EURANDOM Report 2000-033, EURANDOM, Eindhoven, The Netherlands.
19. Heidergott, B., F. Vázquez-Abad. 2004. Gradient estimation for a problem in public transportation: A comparison of SPA. SF and MVD. International Workshop on DES (WODES'04). Reims, France, 241-246.
20. Heidergott, B., R. de Vries. 2001. Towards a control theory for transportation networks. Discrete Event Dynam. Systems 11 371-398.
21. Hordijk, A., R. Dekker. 1983. Average, sensitive and Blaekwell optimal policies in denumerable Markov decision chains with unbounded rewards. Report No. 83-36, Institute of Applied Mathematics and Computing Science, Leiden University, Leiden, The Netherlands.
22. Hordijk, A., F. M. Spieksma. 1992. On ergodicity and recurrence properties of a Markov chain with an application to an open Jackson network. Adv. Appl. Probab. 24 343-376.
23. Hordijk, A., F. M. Spieksma. 1994. A new formula for the deviation matrix. F. P. Kelly, ed. Chapter 36 in Probability, Statistics and Optimization, Wiley, New York.
24. Hordijk, A., A. A. Yushkevich. 1999. Blaekwell optimality in the class of all policies in Markov decision chains with a Borel state space and unbounded rewards. Math. Methods Oper. Res. 50 421-448.
25. Kushner, H., F. Vázquez-Abad. 1992. Estimation of the derivative of a stationary measure with respect to a control parameter. J. Appl. Probab. 29 343-353.
26. Meyn, S. P., R. L. Tweedie. 1993. Markov Chains and Stochastic Stability. Springer, London, UK.
27. Pflug, G. 1992. Gradient estimates for the performance of Markov chains and discrete event processes. Ann. Oper. Res. 39 173-194.
28. Pflug, G. 1996. Optimisation of Stochastic Models. Kluwer Academic Publishers, Boston, MA.
29. Ross, S. M. 1970. Applied Probability Models with Optimization Applications. Holden-Day, San Francisco, CA.
30. Rubinstein, R., A. Shapiro. 1993. Discrete Event Systems: Sensitivity Analysis and Optimization by the Score Function Method, John Wiley and Sons, New York.
31. Suri, R., X. Cao. 1983. The phantom customer and marked customer methods for optimization of closed queueing networks with blocking and general service times. Performance Evaluation Rev. August 243-256.