Markov chains and stochastic stability, second edition

Markov Chains and Stochastic Stability, Second Edition

Volumn , Issue , 2009, Pages 1-504

  Meyn, Sean ^a   Tweedie, Richard L ^b  

^a UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

^b NONE

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84923551867     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1017/CBO9780511626630     Document Type: Book

References (413)

1. D. Aldous and P. Diaconis. Strong uniform times and finite random walks. Adv. Applied Maths., 8: 69-97, 1987.
2. E. Altman, P. Konstantopoulos, and Z. Liu. Stability, monotonicity and invariant quantities in general polling systems. Queueing Syst. Theory Appl., 11: 35-57, 1992.
3. B. D. O. Anderson and J. B. Moore. Optimal Control: Linear Quadratic Methods. Prentice-Hall, Englewood Cliffs, N. J., 1990.
4. W. J. Anderson. Continuous-Time Markov Chains: An Applications-Oriented Approach. Springer-Verlag, New York, 1991.
5. M. Aoki. State Space Modeling of Time Series. Springer-Verlag, Berlin, 1990.
6. A. Arapostathis, V. S. Borkar, E. Fernandez-Gaucherand, M. K. Ghosh, and S. I. Marcus. Discrete-time controlled Markov processes with average cost criterion: a survey. SIAM J. Control Optim., 31: 282-344, 1993.
7. E. Arjas and E. Nummelin. A direct construction of the R-invariant measure for a Markov chain on a general state space. Ann. Probab., 4: 674-679, 1976.
8. E. Arjas, E. Nummelin, and R. L. Tweedie. Uniform limit theorems for non-singular renewal and Markov renewal processes. J. Appl. Probab., 15: 112-125, 1978.
9. S. Asmussen. Applied Probability and Queues. John Wiley & Sons, New York, 1987.
10. S. Asmussen and P. W. Glynn. Stochastic Simulation: Algorithms and Analysis, volume 57 of Stochastic Modelling and Applied Probability. Springer-Verlag, New York, 2007.
11. R. Atar and O. Zeitouni. Lyapunov exponents for finite state nonlinear filtering. SIAM J. Control Optim., 35(1): 36-55, 1997.
12. K. B. Athreya and P. Ney. Branching Processes. Springer-Verlag, New York, 1972.
13. K. B. Athreya and P. Ney. A new approach to the limit theory of recurrent Markov chains. Trans. Amer. Math. Soc., 245: 493-501, 1978.
14. K. B. Athreya and P. Ney. Some aspects of ergodic theory and laws of large numbers for Harris recurrent Markov chains. Colloquia Mathematica Societatis János Bolyai. Nonparametric Statistical Inference, 32: 41-56, 1980. Budapest, Hungary.
15. K. B. Athreya and S. G. Pantula. Mixing properties of Harris chains and autoregressive processes. J. Appl. Probab., 23: 880-892, 1986.
16. K. E. Avrachenkov and J. B. Lasserre. The fundamental matrix of singularly perturbed Markov chains. Adv. Appl. Probab., 31(3): 679-697, 1999.
17. S. Balaji and S. P. Meyn. Multiplicative ergodicity and large deviations for an irreducible Markov chain. Stoch. Proc. Applns., 90(1): 123-144, 2000.
18. D. J. Bartholomew and D. J. Forbes. Statistical Techniques for Manpower Planning. John Wiley & Sons, New York, 1979.
19. R. Bartoszyński. On the risk of rabies. Math. Biosci., 24: 355-377, 1975.
20. P. Baxendale. Stochastic averaging and asymptotic behavior of the stochastic Duffingvan der Pol equation. Stoch. Proc. Applns., 113: 235-272, 2004.
21. P. H. Baxendale. Renewal theory and computable convergence rates for geometrically ergodic Markov chains. Adv. Appl. Probab., 15(1B): 700-738, 2005.
22. P. H. Baxendale and L. Goukasian. Lyapunov exponents for small random perturbations of Hamiltonian systems. Ann. Probab., 30(1): 101-134, 2002.
23. V. E. Beněs. Existence of finite invariant measures for Markov processes. Proc. Amer. Math. Soc., 18: 1058-1061, 1967.
24. V. E. Beněs. Finite regular invariant measures for Feller processes. J. Appl. Probab., 5: 203-209, 1968.
25. H. C. P. Berbee. Random walks with stationary increments and renewal theory. PhD thesis, Vrije University, Amsterdam, 1979.
26. J. R. L. Bernard, editor. Macquarie English Thesaurus. Macquarie Library, 1984.
27. D. P. Bertsekas. Dynamic Programming and Optimal Control. Athena Scientific, Cambridge, Mass., third edition, 2007.
28. D. P. Bertsekas and J. N. Tsitsiklis. An analysis of stochastic shortest path problems. Math. Oper. Res., 16(3): 580-595, 1991.
29. D. P. Bertsekas and J. N. Tsitsiklis. Neuro-Dynamic Programming. Athena Scientific, Cambridge, Mass., 1996.
30. D. Bertsimas, D. Gamarnik, and J. N. Tsitsiklis. Stability conditions for multiclass fluid queueing networks. IEEE Trans. Automat. Control, 41(11): 1618-1631, 1996.
31. D. Bertsimas, D. Gamarnik, and J. N. Tsitsiklis. Correction to: "Stability conditions for multiclass fluid queueing networks" [IEEE Trans. Automat. Control. 41 (1996), no. 11, 1618-1631; MR1419686 (97f: 90028)]. IEEE Trans. Automat. Control, 42(1): 128, 1997.
32. D. Bertsimas, D. Gamarnik, and J. N. Tsitsiklis. Performance of multiclass Markovian queueing networks via piecewise linear Lyapunov functions. Ann. Appl. Probab., 11(4): 1384-1428, 2001.
33. D. Bertsimas, I. Paschalidis, and J. N. Tsitsiklis. Optimization of multiclass queueing networks: polyhedral and nonlinear characterizations of achievable performance. Ann. Appl. Probab., 4: 43-75, 1994.
34. N. P. Bhatia and G. P. Szegö. Stability Theory of Dynamical Systems. Springer-Verlag, New York, 1970.
35. R. N. Bhattacharaya. On the functional Central Limit Theorem and the law of the iterated logarithm for Markov processes. Z. Wahrscheinlichkeitstheorie und Verw. Geb., 60: 185-201, 1982.
36. P. Billingsley. Convergence of Probability Measures. John Wiley & Sons, New York, 1968.
37. P. Billingsley. Probability and Measure. John Wiley & Sons, New York, 1995.
38. H. Bonsdorff. Characterisation of uniform recurrence for general Markov chains. Ann. Acad. Scientarum Fennicae Ser. A, I. Mathematica, Dissertationes, 32, 1980.
39. V. S. Borkar. Stochastic Approximation: A Dynamical Systems Viewpoint. Hindustan Publishing Agency and Cambridge University Press (jointly), Delhi and Cambridge, 2008.
40. V. S. Borkar and S. P. Meyn. The O. D. E. method for convergence of stochastic approximation and reinforcement learning. SIAM J. Control Optim., 38(2): 447-469, 2000. (also presented at the IEEE CDC, December 1998).
41. V. S. Borkar and S. P. Meyn. Risk-sensitive optimal control for Markov decision processes with monotone cost. Math. Oper. Res., 27(1): 192-209, 2002.
42. Vivek S. Borkar. Uniform stability of controlled Markov processes. In System Theory: Modeling, Analysis and Control (Cambridge, MA, 1999), volume 518 of Kluwer Internat. Ser. Engrg. Comput. Sci., pages 107-120. Kluwer Acad. Publ., Boston, 2000.
43. A. A. Borovkov. Limit theorems for queueing networks. Theory Probab. Appl., 31: 413-427, 1986.
44. A. Brandt. The stochastic equation yn+1 = an yn + bn with stationary coefficients. Adv. Appl. Probab., 18: 211-220, 1986.
45. A. Brandt, P. Franken, and B. Lisek. Stationary Stochastic Models. Akademie-Verlag, Berlin, 1990.
46. L. Breiman. The strong law of large numbers for a class of Markov chains. Ann. Math. Statist., 31: 801-803, 1960.
47. L. Breiman. Some probabilistic aspects of the renewal theorem. In Trans. 4th Prague Conf. on Inf. Theory, Statist. Dec. Functions and Random Procs., pages 255-261. Academia, Prague, 1967.
48. L. Breiman. Probability. Addison-Wesley, Reading, Mass., 1968.
49. L. A. Breyer and G. O. Roberts. Catalytic perfect simulation. Methodology and Computing in Applied Probability, 3(2): 161-177, 2001.
50. P. J. Brockwell, 1992. Personal communication.
51. P. J. Brockwell and R. A. Davis. Time Series: Theory and Methods. Springer-Verlag, New York, second edition, 1991.
52. P. J. Brockwell, J. Liu, and R. L. Tweedie. On the existence of stationary threshold autoregressive moving-average processes. J. Time Ser. Anal., 13: 95-107, 1992.
53. P. J. Brockwell, S. J. Resnick, and R. L. Tweedie. Storage processes with general release rule and additive inputs. Adv. Appl. Probab., 14: 392-433, 1982.
54. G. A. Brosamler. An almost everywhere Central Limit Theorem. Math. Proc. Camb. Phil. Soc., 104: 561-574, 1988.
55. J. B. Brown. Ergodic Theory and Topological Dynamics. Academic Press, New York, 1976.
56. S. Browne and K. Sigman. Work-modulated queues with applications to storage processes. J. Appl. Probab., 29: 699-712, 1992.
57. P. E. Caines. Linear Stochastic Systems. John Wiley & Sons, New York, 1988.
58. P. Cattiaux, A. Guillin, F.-Y. Wang, and L. Wu. Lyapunov conditions for logarithmic Sobolev and super Poincaré inequality. ArXiv 0712. 0235 [math. PR], 2007.
59. E. Çinlar. Introduction to Stochastic Processes. Prentice-Hall, Englewood Cliffs, N. J., 1975.
60. H. P. Chan and T. L. Lai. Saddlepoint approximations and nonlinear boundary crossing probabilities of Markov random walks. Ann. Appl. Probab., 13(2): 395-429, 2003.
61. K. S. Chan. Topics in Nonlinear Time Series Analysis. PhD thesis, Princeton University, 1986.
62. K. S. Chan. A note on the geometric ergodicity of a Markov chain. Adv. Appl. Probab., 21: 702-704, 1989.
63. K. S. Chan. Asymptotic behaviour of the Gibbs sampler. J. Amer. Statist. Assoc., 88: 320-326, 1993.
64. K. S. Chan, J. Petruccelli, H. Tong, and S. W. Woolford. A multiple threshold AR(1) model. J. Appl. Probab., 22: 267-279, 1985.
65. H. Chen. Fluid approximations and stability of multiclass queueing networks: workconserving disciplines. Ann. Appl. Probab., 5(3): 637-665, 1995.
66. R. Chen and R. S. Tsay. On the ergodicity of TAR(1) processes. Ann. Appl. Probab., 1: 613-634, 1991.
67. R-R. Chen and S. P. Meyn. Value iteration and optimization of multiclass queueing networks. Queueing Syst. Theory Appl., 32(1-3): 65-97, 1999.
68. P. Chigansky, R. Liptser, and R. van Handel. Intrinsic methods in filter stability. In Handbook of Nonlinear Filtering. Oxford University Press, 2008. To appear.
69. Y. S. Chow and H. Robbins. A renewal theorem for random variables which are dependent or non-identically distributed. Ann. Math. Statist., 34: 390-395, 1963.
70. K. L. Chung. The general theory of Markov processes according to Doeblin. Z. Wahrscheinlichkeitstheorie und Verw. Geb., 2: 230-254, 1964.
71. K. L. Chung. Markov Chains with Stationary Transition Probabilities. Springer-Verlag, Berlin, second edition, 1967.
72. K. L. Chung. A Course in Probability Theory. Academic Press, New York, second edition, 1974.
73. K. L. Chung and D. Ornstein. On the recurrence of sums of random variables. Bull. Amer. Math. Soc., 68: 30-32, 1962.
74. R. Cogburn. The Central Limit Theorem for Markov processes. In L. M. Le Cam, J. Neyman, and E. L. Scott, editors, Proceedings of the 6th Berkeley Symposium on Mathematical Statistics and Probability, pages 485-512. University of California Press, Berkeley, 1972.
75. R. Cogburn. A uniform theory for sums of Markov chain transition probabilities. Ann. Probab., 3: 191-214, 1975.
76. J. W. Cohen. The Single Server Queue. North-Holland, Amsterdam, second edition, 1982.
77. C. Constantinescu and A. Cornea. Potential Theory on Harmonic Spaces. Springer-Verlag, Berlin, 1972.
78. P. C. Consul. Evolution of surnames. Int. Statist. Rev., 59: 271-278, 1991.
79. J. N. Corcoran and R. L. Tweedie. Perfect sampling of ergodic Harris chains. Ann. Appl. Probab., 11(2): 438-451, 2001.
80. J. G. Dai. On positive Harris recurrence of multiclass queueing networks: a unified approach via fluid limit models. Ann. Appl. Probab., 5(1): 49-77, 1995.
81. J. G. Dai and S. P. Meyn. Stability and convergence of moments for multiclass queueing networks via fluid limit models. IEEE Trans. Automat. Control, 40: 1889-1904, 1995.
82. J. G. Dai and G. Weiss. Stability and instability of fluid models for reentrant lines. Math. Oper. Res., 21(1): 115-134, 1996.
83. D. Daley. The serial correlation coefficients of waiting times in a stationary single server queue. J. Austral. Math. Soc., 8: 683-699, 1968.
84. A. de Acosta and P. Ney. Large deviation lower bounds for arbitrary additive functionals of a Markov chain. Ann. Probab., 26(4): 1660-1682, 1998.
85. B. Delyon and O. Zeitouni. Lyapunov exponents for filtering problems. In Applied stochastic analysis (London, 1989), volume 5 of Stochastics Monogr., pages 511-521. Gordon and Breach, New York, 1991.
86. C. Derman. A solution to a set of fundamental equations in Markov chains. Proc. Amer. Math. Soc., 5: 332-334, 1954.
87. G. B. Di Masi and L. Stettner. Ergodicity of hidden Markov models. Math. Control Signals Systems, 17(4): 269-296, 2005.
88. P. Diaconis. Group Representations in Probability and Statistics. Institute of Mathematical Statistics, Hayward, Calif., 1988.
89. P. Diaconis and L. Saloff-Coste. Logarithmic sobolev inequalities for finite Markov chains. Ann. Appl. Probab., 6(3): 695-750, 1996.
90. P. Diaconis and D. Stroock. Geometric bounds for eigenvalues of Markov chains. Ann. Appl. Probab., 1: 36-61, 1991.
91. J. Diebolt. Loi stationnaire et loi des fluctuations pour le processus autorégressif général d'ordre un. C. R. Acad. Sci., 310: 449-453, 1990.
92. J. Diebolt and D. Guégan. Probabilistic properties of the general nonlinear Markovian process of order one and applications to time series modeling. Technical report 125, Laboratoire de Statistique Théorique et Appliquée, Université Paris, 1990.
93. W. Doeblin. Sur les propriétés asymptotiques de mouvement régis par certain types de châınes simples. Bull. Math. Soc. Roum. Sci., 39(1): 57-115; 39(2), 3-61, 1937.
94. W. Doeblin. Exposé de la théorie des châınes simples constantes de Markov à un nombre fini d'états. Revue Mathematique de l'Union Interbalkanique, 2: 77-105, 1938.
95. W. Doeblin. Eléments d'une théorie générale des châınes simples constantes de Markoff. Ann. Sci. Ec. Norm. Sup., 57: 61-111, 1940.
96. M. D. Donsker and S. R. S. Varadhan. Asymptotic evaluation of certain Markov process expectations for large time. I. II. Comm. Pure Appl. Math., 28: 1-47; 28 (1975), 279-301, 1975.
97. M. D. Donsker and S. R. S. Varadhan. Asymptotic evaluation of certain Markov process expectations for large time. III. Comm. Pure Appl. Math., 29(4): 389-461, 1976.
98. M. D. Donsker and S. R. S. Varadhan. Asymptotic evaluation of certain Markov process expectations for large time. IV. Comm. Pure Appl. Math., 36(2): 183-212, 1983.
99. J. L. Doob. Stochastic Processes. John Wiley & Sons, New York, 1953.
100. R. Douc, G. Fort, E. Moulines, and P. Soulier. Practical drift conditions for subgeometric rates of convergence. Ann. Appl. Probab., 14(3): 1353-1377, 2004.
101. D. Down, S. P. Meyn, and R. L. Tweedie. Exponential and uniform ergodicity of Markov processes. Ann. Probab., 23(4): 1671-1691, 1995.
102. M. Duflo. Méthodes Récursives Aléatoires. Masson, Paris, 1990.
103. W. T. M. Dunsmuir, S. P. Meyn, and G. Roberts. Obituary: Richard Lewis Tweedie. J. Appl. Probab., 39(2): 441-454, 2002.
104. P. Dupuis and R. S. Ellis. A Weak Convergence Approach to the Theory of Large Deviations. John Wiley & Sons Inc., New York, 1997.
105. E. B. Dynkin. Markov Processes I, II. Academic Press, New York, 1965.
106. R. J. Elliott, L. Aggoun, and J. B. Moore. Hidden Markov Models. Springer-Verlag, New York, 1995.
107. Y. Ephraim and N. Merhav. Hidden Markov processes. IEEE Trans. Inform. Theory, 48(6): 1518-1569, 2002.
108. S. N. Ethier and T. G. Kurtz. Markov Processes: Characterization and Convergence. John Wiley & Sons, New York, 1986.
109. G. Fayolle. On random walks arising in queueing systems: ergodicity and transience via quadratic forms as Lyapounov functions. I. Queueing Systems, 5: 167-183, 1989.
110. G. Fayolle, V. A. Malyshev, M. V. Menshikov, and A. F. Sidorenko. Lyapunov functions for Jackson networks. Math. Oper. Res., 18(4): 916-927, 1993.
111. P. D. Feigin and R. L. Tweedie. Random coefficient autoregressive processes: a Markov chain analysis of stationarity and finiteness of moments. J. Time Ser. Anal., 6: 1-14, 1985.
112. P. D. Feigin and R. L. Tweedie. Linear functionals and Markov chains associated with the Dirichlet process. Math. Proc. Camb. Phil. Soc., 105: 579-585, 1989.
113. E. Feinberg and A. Shwartz, editors. Markov Decision Processes: Models, Methods, Directions, and Open Problems. Kluwer Acad. Publ., Holland, 2001.
114. W. Feller. An Introduction to Probability Theory and Its Applications. I. John Wiley & Sons, New York, third edition, 1968.
115. W. Feller. An Introduction to Probability Theory and Its Applications. II. John Wiley & Sons, New York, second edition, 1971.
116. J. Feng. Martingale problems for large deviations of Markov processes. Stoch. Proc. Applns., 81: 165-212, 1999.
117. J. Feng and T. G. Kurtz. Large Deviations for Stochastic Processes, volume 131 of Mathematical Surveys and Monographs. American Mathematical Society, Providence, R. I., 2006.
118. P. A. Ferrari, H. Kesten, and S. Martínez. R-positivity, quasi-stationary distributions and ratio limit theorems for a class of probabilistic automata. Ann. Appl. Probab., 6: 577-616, 1996.
119. J. A. Fill. Eigenvalue bounds on convergence to stationarity for nonreversible Markov chains, with an application to the exclusion process. Ann. Appl. Probab., 1(1): 62-87, 1991.
120. W. H. Fleming. Exit probabilities and optimal stochastic control. App. Math. Optim., 4: 329-346, 1978.
121. S. R. Foguel. Positive operators on C(X). Proc. Amer. Math. Soc., 22: 295-297, 1969.
122. S. R. Foguel. The Ergodic Theory of Markov Processes. Van Nostrand Reinhold, New York, 1969.
123. S. R. Foguel. The ergodic theory of positive operators on continuous functions. Ann. Scuola Norm. Sup. Pisa, 27: 19-51, 1973.
124. S. R. Foguel. Selected Topics in the Study of Markov Operators. Carolina Lecture Series. Dept. of Mathematics, University of North Carolina at Chapel Hill, 1980.
125. G. Fort, S. Meyn, E. Moulines, and P. Priouret. ODE methods for skip-free Markov chain stability with applications to MCMC. Ann. Appl. Probab., 18(2): 664-707, 2008.
126. G. Fort and E. Moulines. Polynomial ergodicity of Markov transition kernels. Stoch. Proc. Applns., 103(1): 57-99, 2003.
127. S. G. Foss and R. L. Tweedie. Perfect simulation and backward coupling. Comm. Statist. Stochastic Models, 14(1-2): 187-203, 1998. Special issue in honor of Marcel F. Neuts.
128. S. G. Foss, R. L. Tweedie, and J. N. Corcoran. Simulating the invariant measures of Markov chains using backward coupling at regeneration times. Probab. Engrg. Inform. Sci., 12(3): 303-320, 1998.
129. F. G. Foster. On the stochastic matrices associated with certain queuing processes. Ann. Math. Statist., 24: 355-360, 1953.
130. J. J. Fuchs and B. Delyon. Adaptive control of a simple time-varying system. IEEE Trans. Automat. Control, 37: 1037-1040, 1992.
131. Cheng-Der Fuh. Asymptotic operating characteristics of an optimal change point detection in hidden Markov models. Ann. Statist., 32(5): 2305-2339, 2004.
132. Cheng-Der Fuh and Tze Leung Lai. Asymptotic expansions in multidimensional Markov renewal theory and first passage times for Markov random walks. Adv. in Appl. Probab., 33(3): 652-673, 2001.
133. Cheng-Der Fuh and Cun-Hui Zhang. Poisson equation, moment inequalities and quick convergence for Markov random walks. Stoch. Proc. Applns., 87(1): 53-67, 2000.
134. H. Furstenberg and H. Kesten. Products of random matrices. Ann. Math. Statist., 31: 457-469, 1960.
135. D. Gamarnik. Extension of the PAC framework to finite and countable Markov chains. IEEE Trans. Inform. Theory, 49(1): 338-345, 2003.
136. J. M. Gani and I. W. Saunders. Some vocabulary studies of literary texts. Sankhyā Ser. B, 38: 101-111, 1976.
137. L. Georgiadis, W. Szpankowski, and L. Tassiulas. A scheduling policy with maximal stability region for ring networks with spatial reuse. Queueing Syst. Theory Appl., 19(1-2): 131-148, 1995.
138. P. Glynn and R. Szechtman. Some new perspectives on the method of control variates. In K. T. Fang, F. J. Hickernell, and H. Niederreiter, editors, Monte Carlo and Quasi-Monte Carlo Methods 2000: Proceedings of a Conference held at Hong Kong Baptist University, Hong Kong SAR, China, pages 27-49. Springer-Verlag, Berlin, 2002.
139. P. W. Glynn and S. P. Meyn. A Liapounov bound for solutions of the Poisson equation. Ann. Probab., 24(2): 916-931, 1996.
140. P. W. Glynn and D. Ormoneit. Hoeffding's inequality for uniformly ergodic Markov chains. Statistics and Probability Letters, 56: 143-146, 2002.
141. F. Z. Gong and L. M. Wu. Spectral gap of positive operators and applications. J. Math. Pure Appl., 85: 151-191, 2006.
142. G. C. Goodwin, P. J. Ramadge, and P. E. Caines. Discrete time stochastic adaptive control. SIAM J. Control Optim., 19: 829-853, 1981.
143. C. W. J. Granger and P. Andersen. An Introduction to Bilinear Time Series Models. Vandenhoeck and Ruprecht, Göttingen, 1978.
144. R. M. Gray. Entropy and Information Theory. Springer-Verlag, New York, 1990.
145. J. A. Gubner, B. Gopinath, and S. R. S. Varadhan. Bounding functions of Markov processes and the shortest queue problem. Adv. in Appl. Probab., 21(4): 842-860, 1989.
146. D. Guégan. Different representations for bilinear models. J. Time Ser. Anal., 8: 389-408, 1987.
147. A. Guillin, C. Leonard, L. Wu, and N. Yao. Transportation inequalities for Markov processes. Unpublished manuscript. Preprint available at http://www. latp. univmrs. fr/-guillin/index3. html, 2007.
148. L. Guo and S. P. Meyn. Adaptive control for time-varying systems: a combination of martingale and Markov chain techniques. Int. J. Adaptive Control and Signal Processing, 3: 1-14, 1989.
149. M. Guo and J. Petruccelli. On the null recurrence and transience of a first-order SETAR model. J. Appl. Probab., 28: 584-592, 1991.
150. Olle Häggström. On the Central Limit Theorem for geometrically ergodic Markov chains. Prob. Theory Related Fields, 132(1): 74-82, 2005.
151. P. Hall and C. C. Heyde. Martingale Limit Theory and Its Application. Academic Press, New York, 1980.
152. P. R. Halmos. Measure Theory. Van Nostrand, Princeton, 1950.
153. A. Handwerk and H. Willems. Wolfgang Doeblin: A mathematician rediscovered. Video-MATH. Springer, Berlin, 2007.
154. L. P. Hansen and J. Scheinkman. Long term risk: an operator approach. In preparation, 2008.
155. T. E. Harris. The existence of stationary measures for certain Markov processes. In Proceedings of the 3rd Berkeley Symposium on Mathematical Statistics and Probability, volume 2, pages 113-124. University of California Press, Berkeley, 1956.
156. J. M. Harrison. Discrete dynamic programming with unbounded rewards. Ann. Math. Statist., 43: 636-644, 1972.
157. J. M. Harrison and S. I. Resnick. The stationary distribution and first exit probabilities of a storage process with general release rule. Math. Oper. Res., 1: 347-358, 1976.
158. S. G. Henderson and P. W. Glynn. Approximating martingales for variance reduction in Markov process simulation. Math. Oper. Res., 27(2): 253-271, 2002.
159. S. G. Henderson, S. P. Meyn, and V. B. Tadić. Performance evaluation and policy selection in multiclass networks. Discrete Event Dynamic Systems: Theory and Applications, 13(1-2): 149-189, 2003.
160. S. G. Henderson. Variance Reduction via an Approximating Markov Process. PhD thesis, Stanford University, Stanford, Calif., 1997.
161. I. N. Herstein. Topics in Algebra. John Wiley & Sons, New York, second edition, 1975.
162. G. Högnas. On random walks with continuous components. Preprint no 26, Aarhus Universitet, 1977.
163. A. Hordijk and F. M. Spieksma. On ergodicity and recurrence properties of a Markov chain with an application. Adv. Appl. Probab., 24: 343-376, 1992.
164. C. Huang and D. Isaacson. Ergodicity using mean visit times. J. Lond. Math. Soc., 14: 570-576, 1976.
165. J. Huang, I. Kontoyiannis, and S. P. Meyn. The ODE method and spectral theory of Markov operators. In T. E. Duncan and B. Pasik-Duncan, editors, Proceedings of the workshop held at the University of Kansas, Lawrence, October 18-20, 2001, pages 205-222. Springer-Verlag, Berlin, 2002.
166. W. Huisinga, S. Meyn, and C. Schütte. Phase transitions and metastability in Markovian and molecular systems. Ann. Appl. Probab., 14(1): 419-458, 2004.
167. K. Ichihara and H. Kunita. A classification of the second order degenerate elliptic operator and its probabilistic characterization. Z. Wahrscheinlichkeitstheorie und Verw. Geb., 30: 235-254, 1974.
168. N. Ikeda and S. Watanabe. Stochastic Differential Equations and Diffusion Processes. North-Holland, Amsterdam, 1981.
169. R. Isaac. Some topics in the theory of recurrent Markov processes. Duke Math. J., 35: 641-652, 1968.
170. D. Isaacson and R. L. Tweedie. Criteria for strong ergodicity of Markov chains. J. Appl. Probab., 15: 87-95, 1978.
171. N. Jain. Some limit theorem for a general Markov process. Z. Wahrscheinlichkeitstheorie und Verw. Geb., 6: 206-223, 1966.
172. N. Jain and B. Jamison. Contributions to Doeblin's theory of Markov processes. Z. Wahrscheinlichkeitstheorie und Verw. Geb., 8: 19-40, 1967.
173. B. Jakubczyk and E. D. Sontag. Controllability of nonlinear discrete-time systems: a lie-algebraic approach. SIAM J. Control Optim., 28: 1-33, 1990.
174. B. Jamison. Asymptotic behavior of successive iterates of continuous functions under a Markov operator. J. Math. Anal. Appl., 9: 203-214, 1964.
175. B. Jamison. Ergodic decomposition induced by certain Markov operators. Trans. Amer. Math. Soc., 117: 451-468, 1965.
176. B. Jamison. Irreducible Markov operators on C(S). Proc. Amer. Math. Soc., 24: 366-370, 1970.
177. B. Jamison and S. Orey. Markov chains recurrent in the sense of Harris. Z. Wahrscheinlichkeitstheorie und Verw. Geb., 8: 41-48, 1967.
178. B. Jamison and R. Sine. Sample path convergence of stable Markov processes. Z. Wahrscheinlichkeitstheorie und Verw. Geb., 28: 173-177, 1974.
179. S. F. Jarner and S. Hansen. Geometric ergodicity of Metropolis algorithms. Stoch. Proc. Applns., 85(2): 341-361, 2000.
180. S. F. Jarner and G. O. Roberts. Polynomial convergence rates of Markov chains. Ann. Appl. Probab., 12(1): 224-247, 2002.
181. A. A. Johnson and G. L. Jones. Gibbs sampling for a Bayesian hierarchical general linear model. ArXiv: 0712. 3056 [math. PR], 2007.
182. D. A. Jones. Non-linear autoregressive processes. Proc. Roy. Soc. A, 360: 71-95, 1978.
183. G. L. Jones. On the Markov chain Central Limit Theorem. Probab. Surv., 1: 299-320 (electronic), 2004.
184. G. L. Jones and J. P. Hobert. Honest exploration of intractable probability distributions via Markov chain Monte Carlo. Statist. Sci., 16(4): 312-334, 2001.
185. A. F. Veinott Jr. Discrete dynamic programming with sensitive discount optimality criteria. Ann. Math. Statist., 40(5): 1635-1660, 1969.
186. M. Kac. On the notion of recurrence in discrete stochastic processes. Bull. Amer. Math. Soc., 53: 1002-1010, 1947.
187. V. V. Kalashnikov. Analysis of ergodicity of queueing systems by Lyapunov's direct method (in russian). Avtomatica i Telemechanica, 4: 46-54, 1971.
188. V. V. Kalashnikov. The property of gamma-reflexivity for Markov sequences. Soviet Math. Dokl., 14: 1869-1873, 1973.
189. V. V. Kalashnikov. Stability analysis in queueing problems by the method of test functions. Theory Probab. Appl., 22: 86-103, 1977.
190. V. V. Kalashnikov. Qualitative Analysis of Complex Systems Behaviour by the Test Functions Method (in Russian). Nauka, Moscow, 1978. [in Russian].
191. V. V. Kalashnikov and S. T. Rachev. Mathematical Methods for Construction of Queueing Models. Wadsworth and Brooks/Cole, New York, 1990.
192. R. E. Kalman and J. E. Bertram. Control system analysis and design by the second method of Lyapunov. Trans. ASME Ser. D: J. Basic Eng., 82: 371-400, 1960.
193. M. Kaplan. A sufficient condition for nonergodicity of a Markov chain. IEEE Trans. Inform. Theory, 25: 470-471, 1979.
194. S. Karlin and H. M. Taylor. A First Course in Stochastic Processes. Academic Press, New York, second edition, 1975.
195. H. A. Karlsen. Existence of moments in a stationary stochastic difference equation. Adv. Appl. Probab., 22: 129-146, 1990.
196. N. V. Kartashov. Criteria for uniform ergodicity and strong stability of Markov chains with a common phase space. Theory Probab. Appl., 30: 71-89, 1985.
197. N. V. Kartashov. Inequalities in theorems of ergodicity and stability for Markov chains with a common phase space. Theory Probab. Appl., 30: 247-259, 1985.
198. J. L. Kelley. General Topology. Van Nostrand, Princeton, N. J., 1955.
199. F. P. Kelly. Reversibility and Stochastic Networks. John Wiley & Sons, Chichester, U. K., 1979.
200. D. G. Kendall. Some problems in the theory of queues. J. Roy. Statist. Soc. Ser. B, 13: 151-185, 1951.
201. D. G. Kendall. Stochastic processes occurring in the theory of queues and their analysis by means of the imbedded Markov chain. Ann. Math. Statist., 24: 338-354, 1953.
202. D. G. Kendall. Unitary dilations of Markov transition operators and the corresponding integral representation for transition-probability matrices. In U. Grenander, editor, Probability and Statistics, pages 139-161. Almqvist and Wiksell, Stockholm, 1959.
203. D. G. Kendall. Geometric ergodicity in the theory of queues. In K. J. Arrow, S. Karlin, and P. Suppes, editors, Mathematical Methods in the Social Sciences, pages 176-195. Stanford University Press, Stanford, 1960.
204. D. G. Kendall. Kolmogorov as I remember him. Statist. Sci., 6: 303-312, 1991.
205. G. Kersting. On recurrence and transience of growth models. J. Appl. Probab., 23: 614-625, 1986.
206. R. Z. Khas'minskii. Stochastic Stability of Differential Equations. Sijthoff & Noordhoff, Netherlands, 1980.
207. J. F. C. Kingman. The ergodic behaviour of random walks. Biometrika, 48: 391-396, 1961.
208. J. F. C. Kingman. Regenerative Phenomena. John Wiley & Sons, London, 1972.
209. C. Kipnis and S. R. S. Varadhan. Central limit theorem for additive functionals of reversible Markov processes and applications to simple exclusions. Comm. Math. Phys., 104(1): 1-19, 1986.
210. A. Klein, L. J. Landau, and D. S. Shucker. Decoupling inequalities for stationary Gaussian processes. Ann. Probab., 10: 702-708, 1981.
211. W. Kliemann. Recurrence and invariant measures for degenerate diffusions. Ann. Probab., 15: 690-707, 1987.
212. W. Kliemann and N. Sri Namachchivaya, editors. Nonlinear Dynamics and Stochastic Mechanics. CRC Press, Boca Raton, Fla., 1995.
213. F. Kochman and J. Reeds. A simple proof of Kaijser's unique ergodicity result for hidden Markov α-chains. Ann. Appl. Probab., 16(4): 1805-1815, 2006.
214. P. V. Kokotovic, J. O'Reilly, and J. K. Khalil. Singular Perturbation Methods in Control: Analysis and Design. Academic Press, Orlando, Fla., 1986.
215. A. N. Kolmogorov. Über die analytischen methoden in der wahrscheinlichkeitsrechnung. Math. Ann., 104: 415-458, 1931.
216. A. N. Kolmogorov. Anfangsgründe der theorie der Markoffschen ketten mit unendlichen vielen möglichen zuständen. Mat. Sbornik N. S. Ser, pages 607-610, 1936.
217. V. R. Konda and J. N. Tsitsiklis. On actor-critic algorithms. SIAM J. Control Optim., 42(4): 1143-1166 (electronic), 2003.
218. I. Kontoyiannis and S. P. Meyn. Spectral theory and limit theorems for geometrically ergodic Markov processes. Ann. Appl. Probab., 13: 304-362, 2003.
219. I. Kontoyiannis and S. P. Meyn. Large deviations asymptotics and the spectral theory of multiplicatively regular Markov processes. Electron. J. Probab., 10(3): 61-123 (electronic), 2005.
220. I. Kontoyiannis and S. P. Meyn. Finite state-space Markov chain approximations for diffusions. In preparation, 2008.
221. U. Krengel. Ergodic Theorems. Walter de Gruyter, Berlin, New York, 1985.
222. P. Krugman and M. Miller, editors. Exchange Rate Targets and Currency Bands. Cambridge University Press, Cambridge, 1992.
223. P. R. Kumar and S. P. Meyn. Stability of queueing networks and scheduling policies. IEEE Trans. Automat. Control, 40(2): 251-260, 1995.
224. P. R. Kumar and S. P. Meyn. Duality and linear programs for stability and performance analysis queueing networks and scheduling policies. IEEE Trans. Automat. Control, 41(1): 4-17, 1996.
225. P. R. Kumar and P. P. Varaiya. Stochastic Systems: Estimation, Identification and Adaptive Control. Prentice-Hall, Englewood Cliffs, N. J., 1986.
226. S. Kumar and P. R. Kumar. Performance bounds for queueing networks and scheduling policies. IEEE Trans. Automat. Control, AC-39: 1600-1611, 1994.
227. H. Kunita. Diffusion processes and control systems. Course at the University of Paris VI, 1974.
228. H. Kunita. Supports of diffusion processes and controllability problems. In K. Itô, editor, Proceedings of the International Symposium on Stochastic Differential Equations, pages 163-185. John Wiley & Sons, New York, 1978.
229. H. Kunita. Stochastic Flows and Stochastic Differential Equations. Cambridge University Press, Cambridge, 1990.
230. B. C. Kuo. Automatic Control Systems. Prentice-Hall, Englewood Cliffs, N. J., sixth edition, 1990.
231. T. G. Kurtz. The Central Limit Theorem for Markov chains. Ann. Probab., 9: 557-560, 1981.
232. H. J. Kushner. Stochastic Stability and Control. Academic Press, New York, 1967.
233. M. T. Lacey andW. Philipp. A note on the almost sure Central Limit Theorem. Statistics and Probability Letters, 9: 201-205, 1990.
234. J. Lamperti. Criteria for the recurrence or transience of stochastic processes I. J. Math. Anal. Appl., 1: 314-330, 1960.
235. J. Lamperti. Criteria for stochastic processes II: passage time moments. J. Math. Anal. Appl., 7: 127-145, 1963.
236. C. Landim. Central limit theorem for Markov processes. In From Classical to Modern Probability, volume 54 of Progr. Probab., pages 145-205. Birkhäuser, Basel, 2003.
237. G. M. Laslett, D. B. Pollard, and R. L. Tweedie. Techniques for establishing ergodic and recurrence properties of continuous-valued Markov chains. Nav. Res. Log. Quart., 25: 455-472, 1978.
238. M. Lin. Conservative Markov processes on a topological space. Israel J. Math., 8: 165-186, 1970.
239. T. Lindvall. Lectures on the Coupling Method. John Wiley & Sons, New York, 1992.
240. R. S. Liptster and A. N. Shiryayev. Statistics of Random Processes, II: Applications. Springer-Verlag, New York, 1978.
241. R. Lund and R. L. Tweedie. Geometric convergence rates for stochastically ordered Markov chains. Math. Oper. Res., 20: 182-194, 1996.
242. N. Maigret. Théorème de limite centrale pour une chaine de Markov récurrente Harris positive. Ann. Inst. Henri Poincaré Ser. B, 14: 425-440, 1978.
243. V. A. Malyšev and M. V. Menšikov. Ergodicity, continuity and analyticity of countable Markov chains. Trudy Moskov. Mat. Obshch., 39: 3-48, 235, 1979.
244. Trans. Moscow Math. Soc., pp. 1-48, 1981.
245. V. A. Malyšhev. Classification of two-dimensional positive random walks and almost linear semi-martingales. Soviet Math. Dokl., 13: 136-139, 1972.
246. R. S. Mamon and R. J. Elliott. Hidden Markov Models in Finance, volume 104 of International Series in Operations Research & Management Science. Springer-Verlag, New York, 2007.
247. P. Marbach and J. N. Tsitsiklis. Simulation-based optimization of Markov reward processes. IEEE Trans. Automat. Control, 46(2): 191-209, 2001.
248. I. M. Y. Mareels and R. R. Bitmead. Bifurcation effects in robust adaptive control. IEEE Trans. Circuits and Systems, 35: 835-841, 1988.
249. A. A. Markov. Extension of the law of large numbers to dependent quantities (in Russian). Izv. Fiz.-Matem. Obsch. Kazan Univ. (2nd Ser.), 15: 135-156, 1906.
250. P. G. Marlin. On the ergodic theory of Markov chains. Operations Res., 21: 617-622, 1973.
251. P. Mathé. Numerical integration using V-uniformly ergodic Markov chains. J. Appl. Probab., 41(4): 1104-1112, 2004.
252. J. G. Mauldon. On non-dissipative Markov chains. Math. Proc. Camb. Phil. Soc., 53: 825-835, 1958.
253. M. Maxwell and M. Woodroofe. Central limit theorems for additive functionals of Markov chains. Ann. Probab., 28(2): 713-724, 2000.
254. D. Q. Mayne. Optimal nonstationary estimation of the parameters of a linear system with Gaussian inputs. J. Electron. Contr., 14: 101, 1963.
255. A. Medio. Invariant probability distributions in economic models: a general result. Macroeconomic Dynamics, 8(2): 162-187, 2004. Available at http://ideas. repec. org/a/cup/macdyn/v8y2004i02p162-187 03. html.
256. A. I. Mees, editor. Nonlinear Dynamics and Statistics. Birkhäuser, Boston, 2001. Selected papers from the workshop held at Cambridge University, Cambridge, September 1998.
257. K. L. Mengersen and R. L. Tweedie. Rates of convergence of the Hastings and Metropolis algorithms. Ann. Statist., 24: 101-121, 1996.
258. M. V. Men'šikov. Ergodicity and transience conditions for random walks in the positive octant of space. Soviet Math. Dokl., 15: 1118-1121, 1974.
259. J.-F. Mertens, E. Samuel-Cahn, and S. Zamir. Necessary and sufficient conditions for recurrence and transience of Markov chains, in terms of inequalities. J. Appl. Probab., 15: 848-851, 1978.
260. S. P. Meyn. Ergodic theorems for discrete time stochastic systems using a stochastic Lyapunov function. SIAM J. Control Optim., 27: 1409-1439, 1989.
261. S. P. Meyn. Stability of Markov chains on topological spaces with applications to adaptive control and time series analysis. In L. Gerencsér and P. E. Caines, editors, Topics in Stochastic Systems: Modelling, Estimation and Adaptive Control, pages 369-401. Springer-Verlag, New York, 1991.
262. S. P. Meyn. The policy iteration algorithm for average reward Markov decision processes with general state space. IEEE Trans. Automat. Control, 42(12): 1663-1680, 1997.
263. S. P. Meyn. Stability and optimization of queueing networks and their fluid models. In Mathematics of Stochastic Manufacturing Systems (Williamsburg, VA, 1996), pages 175-199. American Mathematical Society, Providence, R. I., 1997.
264. S. P. Meyn. Algorithms for optimization and stabilization of controlled Markov chains. Sādhanā, 24(4-5): 339-367, 1999.
265. S. P. Meyn. Sequencing and routing in multiclass queueing networks I: feedback regulation. SIAM J. Control Optim., 40(3): 741-776, 2001.
266. S. P. Meyn. Large deviation asymptotics and control variates for simulating large functions. Ann. Appl. Probab., 16(1): 310-339, 2006.
267. S. P. Meyn. Myopic policies and MaxWeight policies for stochastic networks. In Proc. of the 46th Conf. on Dec. and Control, pages 639-646, 2007.
268. S. P. Meyn. Control Techniques for Complex Networks. Cambridge University Press, Cambridge, 2008.
269. S. P. Meyn. Stability and asymptotic optimality of generalized MaxWeight policies. To appear in SIAM J. Control Optim., 2008.
270. S. P. Meyn and L. J. Brown. Model reference adaptive control of time varying and stochastic systems. IEEE Trans. Automat. Control, 38: 1738-1753, 1993.
271. S. P. Meyn and P. E. Caines. A new approach to stochastic adaptive control. IEEE Trans. Automat. Control, AC-32: 220-226, 1987.
272. S. P. Meyn and P. E. Caines. Stochastic controllability and stochastic Lyapunov functions with applications to adaptive and nonlinear systems. In Stochastic Differential Systems. Proc. 4th Bad Honnef Conference, pages 235-257. Springer-Verlag, Berlin, 1989.
273. S. P. Meyn and P. E. Caines. Asymptotic behavior of stochastic systems processing Markovian realizations. SIAM J. Control Optim., 29: 535-561, 1991.
274. S. P. Meyn and D. G. Down. Stability of generalized Jackson networks. Ann. Appl. Probab., 4: 124-148, 1994.
275. S. P. Meyn and L. Guo. Stability, convergence, and performance of an adaptive control algorithm applied to a randomly varying system. IEEE Trans. Automat. Control, AC-37: 535-540, 1992.
276. S. P. Meyn and L. Guo. Geometric ergodicity of a doubly stochastic time series model. J. Time Ser. Analysis, 14(1): 93-108, 1993.
277. S. P. Meyn, G. Hagen, G. Mathew, and A. Banasuk. On complex spectra and metastability of Markov models. In Proc. of the 47th Conf. on Dec. and Control, 2008. More information at https://css. paperplaza. net/conferences/scripts/abstract. pl?ConfID=32&Number=1318.
278. S. P. Meyn and R. L. Tweedie. Stability of Markovian processes I: discrete time chains. Adv. Appl. Probab., 24: 542-574, 1992.
279. S. P. Meyn and R. L. Tweedie. Generalized resolvents and Harris recurrence of Markov processes. Contemporary Mathematics, 149: 227-250, 1993.
280. S. P. Meyn and R. L. Tweedie. Stability of Markovian processes II: continuous time processes and sampled chains. Adv. Appl. Probab., 25: 487-517, 1993.
281. S. P. Meyn and R. L. Tweedie. Stability of Markovian processes III: Foster-Lyapunov criteria for continuous time processes. Adv. Appl. Probab., 25: 518-548, 1993.
282. S. P. Meyn and R. L. Tweedie. The Doeblin decomposition. Contemporary Mathematics, 149: 211-225, 1993.
283. S. P. Meyn and R. L. Tweedie. Computable bounds for convergence rates of Markov chains. Ann. Appl. Probab., 4: 981-1011, 1994.
284. S. P. Meyn and R. L. Tweedie. State-dependent criteria for convergence of Markov chains. Ann. Appl. Probab., 4: 149-168, 1994.
285. H. D. Miller. Geometric ergodicity in a class of denumerable Markov chains. Z. Wahrscheinlichkeitstheorie und Verw. Geb., 4: 354-373, 1966.
286. S. Mittnik. Nonlinear time series analysis with generalized autoregressions: a state space approach. Working paperWP-91-06, State University of New York at Stony Brook, Stony Brook, N. Y., 1991.
287. A. Mokkadem. Criteres de melange pour des processus stationnaires. Estimation sous des hypotheses de melange. Entropie de processus lineaires. PhD thesis, Université Paris Sud, Centre d'Orsay, 1987.
288. P. A. P. Moran. The statistical analysis of the Canadian lynx cycle I: structure and prediction. Aust. J. Zool., 1: 163-173, 1953.
289. P. A. P. Moran. The Theory of Storage. Methuen, London, 1959.
290. M. D. Moustafa. Input-output Markov processes. Proc. Koninkl. Ned. Akad. Wetensch., A60: 112-118, 1957.
291. P. Mykland, L. Tierney, and B. Yu. Regeneration in Markov chain samplers. J. Amer. Statist. Assoc., 90(429), 1995.
292. E. Nelson. The adjoint Markoff process. Duke Math. J., 25: 671-690, 1958.
293. M. F. Neuts. Two Markov chains arising from examples of queues with state dependent service times. Sankhyā Ser. A, 297: 259-264, 1967.
294. M. F. Neuts. Markov chains with applications in queueing theory, which have a matrixgeometric invariant probability vector. Adv. Appl. Probab., 10: 185-212, 1978.
295. Marcel F. Neuts. Matrix-Geometric Solutions in Stochastic Models: An Algorithmic Approach. Dover Publications, New York, 1994. Corrected reprint of the 1981 original.
296. J. Neveu. Potentiel Markovien récurrent des châınes de Harris. Ann. Inst. Fourier, Grenoble, 22: 7-130, 1972.
297. J. Neveu. Discrete-Parameter Martingales. North-Holland, Amsterdam, 1975.
298. P. Ney and E. Nummelin. Markov additive processes I: eigenvalue properties and limit theorems. Ann. Probab., 15(2): 561-592, 1987.
299. P. Ney and E. Nummelin. Markov additive processes II: large deviations. Ann. Probab., 15(2): 593-609, 1987.
300. D. F. Nicholls and B. G. Quinn. Random Coefficient Autoregressive Models: An Introduction. Springer-Verlag, New York, 1982.
301. S. Niemi and E. Nummelin. Central limit theorems for Markov random walks. Commentationes Physico-Mathematicae, 54, 1982.
302. E. Nummelin. A splitting technique for Harris recurrent chains. Z. Wahrscheinlichkeitstheorie und Verw. Geb., 43: 309-318, 1978.
303. E. Nummelin. Uniform and ratio limit theorems for Markov renewal and semiregenerative processes on a general state space. Ann. Inst. Henri Poincaré Ser. B, 14: 119-143, 1978.
304. E. Nummelin. General Irreducible Markov Chains and Nonnegative Operators. Cambridge University Press, Cambridge, 1984.
305. E. Nummelin. On the Poisson equation in the potential theory of a single kernel. Math. Scand., 68: 59-82, 1991.
306. E. Nummelin and P. Tuominen. Geometric ergodicity of Harris recurrent Markov chains with applications to renewal theory. Stoch. Proc. Applns., 12: 187-202, 1982.
307. E. Nummelin and P. Tuominen. The rate of convergence in Orey's theorem for Harris recurrent Markov chains with applications to renewal theory. Stoch. Proc. Applns., 15: 295-311, 1983.
308. E. Nummelin and R. L. Tweedie. Geometric ergodicity and R-positivity for general Markov chains. Ann. Probab., 6: 404-420, 1978.
309. S. Orey. Recurrent Markov chains. Pacific J. Math., 9: 805-827, 1959.
310. S. Orey. Limit Theorems for Markov Chain Transition Probabilities. Van Nostrand Reinhold, London, 1971.
311. D. Ornstein. Random walks I. Trans. Amer. Math. Soc., 138: 1-43, 1969.
312. D. Ornstein. Random walks II. Trans. Amer. Math. Soc., 138: 45-60, 1969.
313. A. Pakes and D. Pollard. Simulation and the asymptotics of optimization estimators. Econometrica, 57: 1027-1057, 1989.
314. A. G. Pakes. Some conditions for ergodicity and recurrence of Markov chains. Operations Res., 17: 1048-1061, 1969.
315. K. R. Parthasarathy. Probability Measures on Metric Spaces. Academic Press, New York, 1967.
316. J. Petruccelli and S. W. Woolford. A threshold AR(1) model. J. Appl. Probab., 21: 270-286, 1984.
317. R. G. Phillips and P. V. Kokotovic. A singular pertubation approach to modeling and control of Markov chains. IEEE Trans. Automat. Control, 26(5), 1981.
318. J. W. Pitman. Uniform rates of convergence for Markov chain transition probabilities. Z. Wahrscheinlichkeitstheorie und Verw. Geb., 29: 193-227, 1974.
319. D. B. Pollard and R. L. Tweedie. R-theory for Markov chains on a topological state space I. J. London Math. Society, 10: 389-400, 1975.
320. D. B. Pollard and R. L. Tweedie. R-theory for Markov chains on a topological state space II. Z. Wahrscheinlichkeitstheorie und Verw. Geb., 34: 269-278, 1976.
321. N. Popov. Conditions for geometric ergodicity of countable Markov chains. Soviet Math. Dokl., 18: 676-679, 1977.
322. M. Pourahmadi. On stationarity of the solution of a doubly stochastic model. J. Time Ser. Anal., 7: 123-131, 1986.
323. N. U. Prabhu. Queues and Inventories. John Wiley & Sons, New York, 1965.
324. S. Rai, P. Glynn, and J. E. Glynn. Recurrence classification for a family of non-linear storage models. Submitted for publication., 2007.
325. R. Rayadurgam, S. P. Meyn, and L. Brown. Bayesian adaptive control of time varying systems. In Proc. of the 31st Conf. on Dec. and Control, Tucson, Ariz., 1992.
326. S. I. Resnick. Extreme Values, Regular Variation and Point Processes. Springer-Verlag, New York, 1987.
327. D. Revuz. Markov Chains. North-Holland, Amsterdam, second edition, 1984.
328. C. P. Robert and G. Casella. Monte Carlo Statistical Methods. Springer-Verlag, New York, second edition, 2004.
329. G. O. Roberts and N. Polson. A note on the geometric convergence of the Gibbs sampler. J. Roy. Statist. Soc. Ser. B, 56: 377-384, 1994.
330. G. O. Roberts and J. S. Rosenthal. Geometric ergodicity and hybrid markov chains. Electronic Comm. Probab., 2: 13-25, 1997.
331. G. O. Roberts and J. S. Rosenthal. Convergence of the slice sampler. J. Roy. Statist. Soc. Ser. B, 61: 643-660, 1999.
332. G. O. Roberts and J. S. Rosenthal. The polar slice sampler. Stoch. Models, 18: 257-280, 2002.
333. G. O. Roberts and J. S. Rosenthal. General state space Markov chains and MCMC algorithms. Probab. Surv., 1: 20-71 (electronic), 2004.
334. G. O. Roberts and A. F. M. Smith. Simple conditions for the convergence of the Gibbs sampler and Hastings-Metropolis algorithms. Stoch. Proc. Applns., 49(2): 207-216, 1994.
335. G. O. Roberts and R. L. Tweedie. Exponential convergence of Langevin distributions and their discrete approximations. Bernoulli, 2(4): 341-363, 1996.
336. G. O. Roberts and R. L. Tweedie. Geometric convergence and Central Limit Theorems for multidimensional Hastings and Metropolis algorithms. Biometrika, 83: 95-100, 1996.
337. Z. Rosberg. A note on the ergodicity of Markov chains. J. Appl. Probab., 18: 112-121, 1981.
338. M. Rosenblatt. Equicontinuous Markov operators. Teor. Verojatnost. i Primenen, 9: 205-222, 1964.
339. M. Rosenblatt. Invariant and subinvariant measures of transition probability functions acting on continuous functions. Z. Wahrscheinlichkeitstheorie und Verw. Geb., 25: 209-221, 1973.
340. M. Rosenblatt. Recurrent points and transition functions acting on continuous functions. Z. Wahrscheinlichkeitstheorie und Verw. Geb., 30: 173-183, 1974.
341. W. A. Rosenkrantz. Ergodicity conditions for two-dimensional Markov chains on the positive quadrant. Prob. Theory and Related Fields, 83: 309-319, 1989.
342. J. S. Rosenthal. Rates of Convergence for Gibbs Sampler and Other Markov Chains. PhD thesis, Harvard University, 1992.
343. J. S. Rosenthal. Correction: "Minorization conditions and convergence rates for Markov chain Monte Carlo." J. Amer. Statist. Assoc., 90(431): 1136, 1995.
344. J. S. Rosenthal. Minorization conditions and convergence rates for Markov chain Monte Carlo. J. Amer. Statist. Assoc., 90(430): 558-566, 1995.
345. J. S. Rosenthal. Quantitative convergence rates of Markov chains: a simple account. Electron. Comm. Probab., 7: 123-128 (electronic), 2002.
346. W. Rudin. Real and Complex Analysis. McGraw-Hill, New York, 2nd edition, 1974.
347. S. H. Saperstone. Semidynamical Systems in Infinite Dimensional Spaces. Springer-Verlag, New York, 1981.
348. P. J. Schweitzer. Perturbation theory and finite Markov chains. J. Appl. Prob., 5: 401-403, 1968.
349. E. Seneta. Non-Negative Matrices and Markov Chains. Springer, New York, second edition, 1981.
350. L. I. Sennott, P. A. Humblet, and R. L. Tweedie. Mean drifts and the non-ergodicity of Markov chains. Operations Res., 31: 783-789, 1983.
351. J. G. Shanthikumar and D. D. Yao. Second-order properties of the throughput of a closed queueing network. Math. Oper. Res., 13: 524-533, 1988.
352. T. Shardlow and A. M. Stuart. A perturbation theory for ergodic Markov chains and application to numerical approximations. SIAM J. Numer. Anal., 37(4): 1120-1137, 2000.
353. M. Sharpe. General Theory of Markov Processes. Academic Press, New York, 1988.
354. Z. Šidák. Classification of Markov chains with a general state space. In Trans. 4th Prague Conf. Inf. Theory Stat. Dec. Functions, Random Proc, pages 547-571. Academia, Prague, 1967.
355. K. Sigman. The stability of open queueing networks. Stoch. Proc. Applns., 35: 11-25, 1990.
356. G. F. Simmons. Introduction to Topology and Modern Analysis. McGraw Hill, New York, 1963.
357. R. Sine. Convergence theorems for weakly almost periodic Markov operators. Israel J. Math., 19: 246-255, 1974.
358. R. Sine. On local uniform mean convergence for Markov operators. Pacific J. Math., 60: 247-252, 1975.
359. R. Sine. Sample path convergence of stable Markov processes II. Indiana University Math. J., 25: 23-43, 1976.
360. A. F. M. Smith and A. E. Gelfand. Bayesian statistics without tears: a samplingresampling perspective. Amer. Statist., 46: 84-88, 1992.
361. A. F. M. Smith and G. O. Roberts. Bayesian computation via the Gibbs sampler and related Markov chain Monte Carlo methods (with discussion). J. Roy. Statist. Soc. Ser. B, 55: 3-23, 1993.
362. W. L. Smith. Asymptotic renewal theorems. Proc. Roy. Soc. Edinburgh (A), 64: 9-48, 1954.
363. W. L. Smith. Regenerative stochastic processes. Proc. Roy. Soc. London (A), 232: 6-31, 1955.
364. W. L. Smith. Remarks on the paper "Regenerative stochastic processes." Proc. Roy. Soc. London (A), 256: 296-301, 1960.
365. J. Snyders. Stationary probability distributions for linear time-invariant systems. SIAM J. Control Optim., 15: 428-437, 1977.
366. V. Solo. Stochastic adaptive control and martingale limit theory. IEEE Trans. Automat. Control, 35: 66-70, 1990.
367. F. M. Spieksma. Geometrically Ergodic Markov Chains and the Optimal Control of Queues. PhD thesis, University of Leiden, 1991.
368. F. M. Spieksma. Spectral conditions and bounds for the rate of convergence of countable Markov chains. Technical report, University of Leiden, 1993.
369. F. M. Spieksma and R. L. Tweedie. Strengthening ergodicity to geometric ergodicity for Markov chains. Stochastic Models, 10: 45-75, 1994.
370. F. Spitzer. Principles of Random Walk. Van Nostrand, Princeton, N. J., 1964.
371. D. Steinsaltz. Locally contractive iterated function systems. Ann. Probab., 27(4): 1952-1979, 1999.
372. Ö. Stenflo. Uniqueness of invariant measures for place-dependent random iterations of functions. In Fractals in Multimedia (Minneapolis, MN, 2001), volume 132 of IMA Vol. Math. Appl., pages 13-32. Springer, New York, 2002.
373. L. Stettner. On the existence and uniqueness of invariant measures for continuous time Markov processes. Technical report LCDS 86-18, Brown University, Providence, R. I., 1986.
374. A. L. Stolyar. On the stability of multiclass queueing networks: a relaxed sufficient condition via limiting fluid processes. Markov Process. Related Fields, 1(4): 491-512, 1995.
375. C. R. Stone. On absolutely continuous components and renewal theory. Ann. Math. Statist., 37: 271-275, 1966.
376. O. Stramer and R. L. Tweedie. Langevin-type models I: diffusions with given stationary distributions and their discretizations. Methodol. Comput. Appl. Probab., 1(3): 283-306, 1999.
377. O. Stramer and R. L. Tweedie. Langevin-type models II: self-targeting candidates for MCMC algorithms. Methodol. Comput. Appl. Probab., 1(3): 307-328, 1999.
378. D. W. Stroock and S. R. Varadhan. On degenerate elliptic-parabolic operators of second order and their associated diffusions. Comm. Pure Appl. Math., 25: 651-713, 1972.
379. D. W. Stroock and S. R. Varadhan. On the support of diffusion processes with applications to the strong maximum principle. In L. M. Le Cam, J. Neyman, and E. L. Scott, editors, Proceedings of the 6th Berkeley Symposium on Mathematical Statistics and Probability, pages 333-368. University of California Press, Berkeley, 1972.
380. R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, Mass., 1998.
381. W. Szpankowski. Some sufficient conditions for non-ergodicity of Markov chains. J. Appl. Probab., 22: 138-147, 1985.
382. M. A. Tanner and W. H. Wong. The calculation of posterior distributions by data augmentation. J. Amer. Statist. Assoc., 82: 528-540, 1987.
383. L. Tassiulas. Adaptive back-pressure congestion control based on local information. IEEE Trans. Automat. Control, 40(2): 236-250, 1995.
384. L. Tassiulas and A. Ephremides. Stability properties of constrained queueing systems and scheduling policies for maximum throughput in multihop radio networks. IEEE Trans. Automat. Control, 37(12): 1936-1948, 1992.
385. J. L. Teugels. An example on geometric ergodicity of a finite Markov chain. J. Appl. Probab., 9: 466-469, 1972.
386. Luke Tierney. Markov chains for exploring posterior distributions. Ann. Statist., 22(4): 1701-1762, 1994. With discussion and a rejoinder by the author.
387. D. Tjostheim. Non-linear time series and Markov chains. Adv. Appl. Probab., 22: 587-611, 1990.
388. H. Tong. A note on a Markov bilinear stochastic process in discrete time. J. Time Ser. Anal., 2: 279-284, 1981.
389. H. Tong. Non-linear Time Series: A Dynamical System Approach. Oxford University Press, Oxford, 1990.
390. J. N. Tsitsiklis and B. Van Roy. An analysis of temporal-difference learning with function approximation. IEEE Trans. Automat. Control, 42(5): 674-690, 1997.
391. P. Tuominen. Notes on 1-recurrent Markov chains. Z. Wahrscheinlichkeitstheorie und Verw. Geb., 36: 111-118, 1976.
392. P. Tuominen and R. L. Tweedie. Markov chains with continuous components. Proc. London Math. Soc. (3), 38: 89-114, 1979.
393. P. Tuominen and R. L. Tweedie. The recurrence structure of general Markov processes. Proc. London Math. Soc. (3), 39: 554-576, 1979.
394. P. Tuominen and R. L. Tweedie. Subgeometric rates of convergence of f-ergodic Markov chains. Adv. Appl. Probab., 26: 775-798, 1994.
395. R. L. Tweedie. R-theory for Markov chains on a general state space I: solidarity properties and R-recurrent chains. Ann. Probab., 2: 840-864, 1974.
396. R. L. Tweedie. R-theory for Markov chains on a general state space II: R-subinvariant measures for R-transient chains. Ann. Probab., 2: 865-878, 1974.
397. R. L. Tweedie. Relations between ergodicity and mean drift for Markov chains. Austral. J. Statist., 17: 96-102, 1975.
398. R. L. Tweedie. Sufficient conditions for ergodicity and recurrence of Markov chains on a general state space. Stoch. Proc. Applns., 3: 385-403, 1975.
399. R. L. Tweedie. Criteria for classifying general Markov chains. Adv. Appl. Probab., 8: 737-771, 1976.
400. R. L. Tweedie. Operator geometric stationary distributions for Markov chains with applications to queueing models. Adv. Appl. Probab., 14: 368-391, 1981.
401. R. L. Tweedie. Criteria for rates of convergence of Markov chains with application to queueing and storage theory. In J. F. C. Kingman and G. E. H. Reuter, editors, Probability, Statistics and Analysis. Cambridge University Press, Cambridge, 1983.
402. R. L. Tweedie. The existence of moments for stationary Markov chains. J. Appl. Probab., 20: 191-196, 1983.
403. R. L. Tweedie. Invariant measures for Markov chains with no irreducibility assumptions. J. Appl. Probab., 25A: 275-285, 1988.
404. D. Vere-Jones. Geometric ergodicity in denumerable Markov chains. Quart. J. Math. Oxford (2nd Ser.), 13: 7-28, 1962.
405. D. Vere-Jones. A rate of convergence problem in the theory of queues. Theory Probab. Appl., 9: 96-103, 1964.
406. D. Vere-Jones. Ergodic properties of nonnegative matrices. I. Pacific J. Math., 22: 361-386, 1967.
407. D. Vere-Jones. Ergodic properties of nonnegative matrices II. Pacific J. Math., 26: 601-620, 1968.
408. L. Wu and N. Yao. Large deviation principles for Markov processes via Φ-Sobolev inequalities. Electron. Commun. Probab., 13: 10-23, 2008.
409. Liming Wu. Essential spectral radius for Markov semigroups I: discrete time case. Prob. Theory Related Fields, 128(2): 255-321, 2004.
410. L. M. Wu. Large deviations for Markov processes under superboundedness. C. R. Acad. Sci Paris Série I, 324: 777-782, 1995.
411. B. E. Ydstie. Bifurcations and complex dynamics in adaptive control systems. In Proc. of the 25th Conf. on Dec. and Control, Athens, Greece, 1986.
412. M. Yor and B. Bru. Comments on the life and mathematical legacy of Wolfgang Doeblin. Finance and Stochastics, 6(1): 3-47, 2002.
413. K. Yosida and S. Kakutani. Operator-theoretical treatment of Markov's process and mean ergodic theorem. Ann. Math., 42: 188-228, 1941.