Gibbs sampling, exponential families and orthogonal polynomials

Statistical Science

Volumn 23, Issue 2, 2008, Pages 151-178

  Diaconis, Persi ^a   Khare, Kshitij ^a   Saloff Coste, Laurent ^b  

^a STANFORD UNIVERSITY   (United States)

^b Department of Obstetrics Gynecology   (United States)

Author keywords

Conjugate priors; Exponential families; Gibbs sampler; Location families; Orthogonal polynomials; Running time analyses; Singular value decomposition

Indexed keywords

EID: 50849137884     PISSN: 08834237     EISSN: None     Source Type: Journal    
DOI: 10.1214/07-STS252     Document Type: Article

References (108)

1. AKHIEZER, N. and GLAZMAN, I. (1993). Theory of Linear Operators in Hilbert Space. Dover, New York. MR1255973
2. AMIT, Y. (1996). Convergence properties of the Gibbs sampler for perturbations of Gaussians. Ann. Statist. 24 122-140. MR1389883
3. ANDERSON, W. (1991). Continuous-Time Markov Chains. An Applications-Oriented Approach. Springer, New York. MR1118840
4. ATHREYA, K., DOSS, H. and SETHURAMAN, J. (1996). On the convergence of the Markov chain simulation method. Ann. Statist. 24 89-100. MR1389881
5. BAIK, J., KRIECHERBAUER, T., MCL AUGHLIN, K. and MILLER, P. (2003). Uniform asymptotics for polynomials orthogonal with respect to a general class of weights and universality results for associated ensembles. Intern. Math. Res. Not. 15 821-858.
6. BAKRY, D. and MAZET, O. (2003). Characterization of Markov semigroups on ℝ associated to some families of orthogonal polynomials. Séminaire de Probabilités XXXVII 60-80. Lecture Notes in Math. 1832. Springer, Berlin. MR2053041
7. BAR-LEV, S., BSHOUTY, D., ENIS, P., LETAC, G., LU, I.-L. and RICHARDS, D. (1994). The diagonal multivariate natural exponential families and their classification. J. Theoret. Probab. 7 883-929. MR1295545
8. BARNDORFF-NIELSEN, O. (1978). Information and Exponential Families in Statistical Theory. Wiley, Chichester. MR0489333
9. BAXENDALE, P. (2005). Renewal theory and computable convergence rates for geometrically ergodic Markov chains. Ann. Appl. Probab. 15 700-738. MR2114987
10. BEN AROUS, G., BOVIER, A. and GAYARD, V. (2003). Glauber dynamics of the random energy model. I, II. Comm. Math Phys. 235 379-425,
11. 236 1-54. MR1977880
12. BROWN, L. (1986). Fundamentals of Statistical Exponential Families. IMS, Hayward, CA.
13. BRYC, W. (2006). Approximation operators, exponential, and free exponential families. Preprint, Dept. Math. Sci., Univ. Cincinnati.
14. BUJA, A. C. (1990). Remarks on functional canonical variates, alternating least square methods and ACE. Ann. Statist. 18 1032-1069. MR1062698
15. CANNINGS, C. (1974). The latent roots of certain Markov chains asrising in genetics: A new approach. I. Haploid models. Adv. in Appl. Probab. 6 260-290. MR0343949
16. d. Ann. Statist. 24 1828-1854. MR1416663
17. CASELLA, G. and GEORGE, E. (1992). Explaining the Gibbs sampler. Amer. Statist. 46 167-174. MR1183069
18. CHAMAYOU, J. and LETAC, G. (1991). Explicit stationary distributions for compositions of random functions and products of random matrices. J. Theoret. Probab. 4 3-36. MR1088391
19. CHIHARA, T. (1978). An Introduction to Orthogonal Polynomials. Gordon and Breach, New York. MR0481884
20. CONSONNI, G. and VERONESE, P. (1992). Conjugate priors for exponential families having quadratic variance functions. J. Amer. Statist. Assoc. 87 1123-1127. MR1209570
21. COOPER, R., HOARE, M. and RAHMAN, M. (1977). Stochastic Processes and special functions: On the probabilistic origin of some positive kernels associated with classical orthogonal polynomials. J. Math. Anal. Appl. 61 262-291. MR0486694
22. DAUXOIS, J. and POUSSE, A. (1975). Une extension de l'analyse canonique. Quelques applications. Ann. Inst. H. Poincaré Sect. B (N.S.) 11 355-379. MR0408118
23. DEUTSCH, F. (2001). Best Approximation in Inner Product Spaces. Springer, New York. MR1823556
24. DIACONIS, P. and FREEDMAN, D. (1999). Iterated random functions. SIAM Rev. 41 45-76. MR1669737
25. DIACONIS, P., KHARE, K. and SALOFF-COSTE, L. (2006). Stochastic alternating projections. Preprint, Dept. Statistics, Stanford Univ.
26. DIACONIS, P., KHARE, K. and SALOFF-COSTE, L. (2006). Gibbs sampling, exponential families and coupling. Preprint, Dept. of Statistics, Stanford Univ.
27. DIACONIS, P. and SALOFF-COSTE, L. (1993). Comparison theorems for Markov chains. Ann. Appl. Probab. 3 696-730. MR1233621
28. DIACONIS, P. and SALOFF-COSTE, L. (2006). Separation cut-offs for birth and death chains. Ann. Appl. Probab. 16 2098-2122. MR2288715
29. DIACONIS, P. and STANTON, D. (2006). A hypergeometric walk. Preprint, Dept. Statistics, Stanford Univ.
30. DIACONIS, P. and YLVISAKER, D. (1979). Conjugate priors for exponential families. Ann. Statist. 7 269-281. MR0520238
31. DIACONIS, P. and YLVISAKER, D. (1985). Quantifying prior opinion. In Bayesian Statistics 2 (J. Bernardo et al. eds.) 133-156. North-Holland, Amsterdam. MR0862488
32. DONOHO, D. and JOHNSTONE, I. (1989). Projection-based approximation and a duality with kernel methods. Ann. Statist. 17 58-106. MR0981438
33. DYER, M., GOLDBERG, L., JERRUM, M. and MARTIN, R. (2005). Markov chain comparison. Probab. Surv. 3 89-111. MR2216963
34. EATON, M. L. (1992). A statistical diptych: Admissible inferences - Recurrence of symmetric Markov chains. Ann. Statist. 20 1147-1179. MR1186245
35. EATON, M. L. (1997). Admissibility in quadratically regular problems and recurrence of symmetric Markov chains: Why the connection? J. Statist. Plann. Inference 64 231-247. MR1621615
36. EATON, M. L. (2001). Markov chain conditions for admissibility on estimation problems with quadratic loss. In State of the Art in Statistics and Probability. A Festschrift for Willen von Zwet (M. de Gunst, C. Klaassen and A. van der Vrad, eds.) 223-243. IMS, Beachwood, OH. MR1836563
37. EAGLESON, G. (1964). Polynomial expansions of bivariate distributions. Ann. Math. Statist. 25 1208-1215. MR0168055
38. ESCH, D. (2003). The skew-t distribution: Properties and computations. Ph.D. dissertation, Dept. Statistics, Harvard Univ.
39. EWENS, W. (2004). Mathematical Population Genetics. I. Theoretical Introduction, 2nd ed. Springer, New York. MR2026891
40. FEINSILVER, P. (1986). Some classes of orthogonal polynomials associated with martingales. Proc. Amer. Math. Soc. 98 298-302. MR0854037
41. FEINSILVER, P. (1991). Orthogonal polynomials and coherent states. In Symmetries in Science V 159-172. Plenum Press, New York. MR1143591
42. FEINSILVER, P. and SCHOTT, R. (1993). Algebraic Structures and Operator Calculus. I. Representations and Probability Theory. Kluwer Academic Press, Dordrecht. MR1227095
43. FELLER, W. (1951). Diffusion processes in genetics. Proc. Second Berkeley Symp. Math. Statist. Probab. 227-246. Univ. California Press, Berkeley. MR0046022
44. FELLER, W. (1968). An Introduction to Probability Theory and Its Applications. I, 3rd ed. Wiley, NewYork. MR0228020
45. FELLER, W. (1971). An Introduction to Probability Theory and Its Applications. II, 2nd ed. Wiley, New York.
46. GELFAND, A. E. and SMITH, A. F. M. (1990). Sampling based approaches to calculating marginal densities. J. Amer. Statist. Assoc. 85 398-409. MR1141740
47. GEMAN, S. and GEMAN, D. (1984). Stochastic relaxation Gibbs distributions and the Bayesian restoration of images. IEEE Trans. Pattern Anal. Machine Intelligence 6 721-741.
48. GILKS, W. RICHARDSON, S. and SPIEGELHALTER, D. (1996). Markov Chain Monte Carlo in Practice. Chapman and Hall, London. MR1397966
49. GILL, J. (2002). Bayesian Methods: A Social and Behavioral Sciences Approach. Chapman and Hall, Boca Raton, FL.
50. GLAUBER, R. (1963). Time dependent statistics of the Ising model. J. Math. Phys. 4 294-307. MR0148410
51. GOODMAN, J. and SOKAL, A. (1984). Multigrid Monte Carlo method conceptual foundations. Phys. Rev. D 40 2035-2071.
52. GRIFFITHS, R. C. (1971). Orthogonal polynomials on the multinomial distribution. Austral. J. Statist. 13 27-35. MR0275548
53. GRIFFITHS, R. C. (1978). On a bivariate triangular distribution. Austral. J. Statist. 20 183-185. MR0548185
54. GRIFFITHS, R. C. (2006). n-kernel orthogonal polynomials on the Dirichlet, Dirichlet-multinomial, Ewens' sampling distribution and Poisson Dirichlet processes. Lecture notes available at http://www.stats.ox.ac. uk/̃griff/.
55. GROSS, L. (1979). Decay of correlations in classical lattice models at high temperatures. Comm. Math. Phys. 68 9-27. MR0539733
56. GUTIERREZ-PENA, E. and SMITH, A. (1997). Exponential and Bayes in conjugate families: Review and extensions. Test 6 1-90. MR1466433
57. HARKNESS, W. and HARKNESS, M. (1968). Generalized hyperbolic secant distributions. J. Amer. Statist. Assoc. 63 329-337. MR0234503
58. HASSAIRI, A. and ZARAI, M. (2004). Characterization of the cubic exponential families by orthogonality of polynomials. Ann. Statist. 32 2463-2476. MR2078547
59. HOARE, M. and RAHMAN, M. (1979). Distributed processes in discrete systems. Physica 97A 1-41.
60. HOARE, M. and RAHMAN, M. (1983). Cumulative Bernoulli trials and Krawtchouk processes. Stochastic Process. Appl. 16 113-139. MR0724060
61. HOARE, M. and RAHMAN, M. (2007). A probabilistic origin for a new class of bivariate polynomials. Preprint, School of Math. and Stat. Carleton Univ., Ottawa.
62. HOBERT, J. P. and ROBERT, C. P. (1999). Eaton's Markov chain, its conjugate partner and P-admissibility. Ann. Statist. 27 361-373. MR1701115
63. ISMAIL, M. (1977). Connection relations and bilinear formulas for the classical orthogonal polynomials. J. Math. Anal. Appl. 57 487-496. MR0430361
64. ISMAIL, M. (2005). Classical and Quantum Orthogonal Polynomials. Cambridge Univ. Press. MR2191786
65. JONES, G. and HOBERT, J. (2001). Honest exploration of intractable probability distributions via Markov chain Monte Carlo. Statist. Sci. 16 312-334. MR1888447
66. JONES, G. L. and HOBERT, J. P. (2004). Sufficient burn-in for Gibbs samplers for a hierarchical random effects model. Ann. Statist. 32 784-817. MR2060178
67. JORGENSEN, C. (1997). The Theory of Dispersion Models. Chapman and Hall, London. MR1462891
68. KARLIN, S. and MCGREGOR, J. (1961). The Hahn polynomials, formulas and application. Scripta. Math. 26 33-46. MR0138806
69. KOEKOEK, R. and SWARTTOUW, R. (1998). The Askeyscheme of hypergeometric orthogonal polynomials and its q-analog. Available at http://math.nist.gov/opsf/projects/koekoek.html.
70. KOUDOU, A. and POMMERET, D. (2000). A construction of Lancaster probabilities with margins in the multidimensional Meixner class. Austr. N. Z. J. Stat. 42 59-66. MR1747462
71. KOUDOU, A. (1996). Probabilities de Lancaster. Exp. Math. 14 247-275. MR1409004
72. KOUDOU, A. (1998). Lancaster bivariate probability distributions with Poisson, negative binomial and gamma margins. Test 7 95-110. MR1650839
73. LANCASTER, H. O. (1969). The Chi-Squared Distribution. Wiley, New York. MR0253452
74. LEHMANN, E. and ROMANO, J. (2005). Testing Statistical Hypotheses, 3rd ed. Springer, New York. MR2135927
75. LETAC, G. (1992). Lectures on Natural Exponential Families and Their Variance Functions. Monografías de Matemática 50. IMPA, Rio de Janeiro. MR1182991
76. LETAC, G. (2002). Donkey walk and Dirichlet distributions. Statist. Probab. Lett. 57 17-22. MR1911808
77. LETAC, G. and MORA, M. (1990). Natural real exponential families with cubic variance functions. Ann. Statist. 18 1-37. MR1041384
78. LIU, J., WONG, W. and KONG, A. (1995). Covariance structure and convergence rates of the Gibbs sampler with various scans. J. Roy. Statist. Soc. Ser. B 57 157-169. MR1325382
79. LIU, J. (2001). Monte Carlo Strategies in Scientific Computing. Springer, New York. MR1842342
80. MALOUCHE, D. (1998). Natural exponential families related to Pick functions. Test 7 391-412. MR1667006
81. MCKENZIE, E. (1988). Some ARMA models for dependent sequences of Poisson counts. Adv. in Appl. Probab. 20 822-835. MR0968000
82. MARCHEV, D. and HOBERT, J. P. (2004). Geometric ergodicity of van Dyk and Meng's algorithm for the multivariate Student's t model. J. Amer. Statist. Assoc. 99 228-238. MR2054301
83. MENG, X. L. and ZASLAVSKY, A. (2002). Single observation unbiased priors. Ann. Statist. 30 1345-1375. MR1936322
84. MEIXNER, J. (1934). Orthogonal polynom system mit einer Besonderth Gestalt der Erzengerder function. J. London Math. Soc. 9 6-13.
85. MEYER, P. A. (1966). Probability and Potentials. Blaisdell, Waltham, MA. MR0205288
86. MORENO, E. and GIRÓN, F. (1998). Estimating with incomplete count data: A Bayesian approach. J. Statist. Plann. Inference 66 147-159. MR1617002
87. MORRIS, C. (1982). Natural exponential families with quadratic variance functions. Ann. Statist. 10 65-80. MR0642719
88. MORRIS, C. (1983). Natural exponential families with quadratic variance functions: Statistical theory. Ann. Statist. 11 515-589. MR0696064
89. NEWMAN, M. and BARKEMA, G. (1999). Monte Carlo Methods in Statistical Physics. Oxford Univ. Press. MR1691513
90. PITT, M. and WALKER, S. (2006). Extended constructions of stationary autoregressive processes. Statist. Probab. Lett. 76 1219-1224. MR2269348
91. PITT, M. and WALKER, S. (2005). Constructing stationary time series models using auxiliary variables with applications. J. Amer. Statist. Assoc. 100 554-564. MR2160559
92. PITT, M., CHATFIELD, C. and WALKER, S. (2002). Constructing first order stationary autoregressive models via latent processes. Scand. J. Statist. 29 657-663. MR1988417
93. POMMERET, D. (1996). Natural exponential families and Lie algebras. Exp. Math. 14 353-381. MR1418028
94. POMMERET, D. (2001). Posterior variance for quadratic natural exponential families. Statist. Probab. Lett. 53 357-362. MR1856159
95. RINGROSE, J. (1971). Compact Non-Self-Adjoint Operators. Van Nostrand, New York.
96. ROSENTHAL, J. (1995). Minorization conditions and convergence rates for Markov chain Monte Carlo. J. Amer. Statist. Assoc. 90 558-566. MR1340509
97. ROSENTHAL, J. (1996). Analysis of the Gibbs sampler for a model related to James-Stein estimations. Statist. Comput. 6 269-275.
98. ROSENTHAL, J. S. (2002). Quantitative convergence rates of Markov chains: A simple account. Electron. Comm. Probab. 7 123-128. MR1917546
99. ROY, V. and HOBERT, J. P. (2007). Convergence rates and asymptotic standard errors for MCMC algorithms for Bayesian probit regression. J. Roy. Statist. Soc. Ser. B 69 607-623.
100. SALOFF-COSTE, L. (2004). Total variation lower bounds for finite Markov chains: Wilson's lemma. In Random Walks and Geometry (V. Kaimanovich and W. Woess, eds.) 515-532. de Gruyter, Berlin. MR2087800
101. SZEGO, G. (1959). Orthogonal Polynomials, rev. ed. Amer. Math. Soc., New York. MR0106295
102. TANNER, M. and WONG, W. (1987). The calculation of posterior distributions by data augmentation (with discussion). J. Amer. Statist. Assoc. 82 528-550. MR0898357
103. TIERNEY, L. (1994). Markov chains for exploring posterior distributions (with discussion). Ann. Statist. 22 1701-1762. MR1329166
104. TURCIN, V. (1971). On the computation of multidimensional integrals by the Monte Carlo method. Probab. Appl. 16 720-724. MR0292259
105. VAN DOORN, E. A. (2003). Birth-death processes and associated polynomials. In Proceedings of the Sixth International Symposium on Orthogonal Polynomials, Special Functions and their Applications (Rome, 2001). J. Comput. Appl. Math. 153 497-506. MR1985718
106. WALTER, G. and HAMEDANI, G. (1991). Bayes empirical Bayes estimation for natural exponential families with quadratic variance functions. Ann. Statist. 19 1191-1224. MR1126321
107. WILSON, D. (2004). Mixing times of Lozenge tiling and card shuffling Markov chains. Ann. Appl. Probab. 14 274-325. MR2023023
108. n. Stochastic Process. Appl. 87 1-23. MR1751162