MCMC for normalized random measure mixture models

Statistical Science

Volumn 28, Issue 3, 2013, Pages 335-359

  Favaro, Stefano ^a   Teh, Yee Whye ^b  

^a UNIVERSITY OF TURIN   (Italy)

^b UNIVERSITY OF OXFORD   (United Kingdom)

Author keywords

Algorithm 8; Bayesian nonparametrics; Completely random measure; Conditional sampler; Dirichlet process; Hierarchical mixture model; Marginalized sampler; MCMC posterior sampling method; Normalized generalized Gamma process; Normalized random measure; Slice sampling.

Indexed keywords

EID: 84878592020     PISSN: 08834237     EISSN: None     Source Type: Journal    
DOI: 10.1214/13-STS422     Document Type: Article

References (86)

1. ALDOUS, D. J. (1985). Exchangeability and related topics. In École D'ÉtÉ de ProbabilitÉs de Saint-Flour, XIII- 1983. Lecture Notes in Math. 1117 1-198. Springer, Berlin. MR0883646
2. BARRIOS, E., LIJOI, A., NIETO-BARAJAS, L. E. and PRüENSTER, I. (2012). Modeling with normalized random measure mixture models. Unpublished manuscript.
3. BINDER, D. A. (1978). Bayesian cluster analysis. Biometrika 65 31-38. MR0501592
4. BLACKWELL, D. and MACQUEEN, J. B. (1973). Ferguson distributions via Pólya urn schemes. Ann. Statist. 1 353-355. MR0362614
5. BRIX, A. (1999). Generalized gamma measures and shotnoise Cox processes. Adv. in Appl. Probab. 31 929-953. MR1747450
6. BRODERICK, T., JORDAN, M. I. and PITMAN, J. (2012). Clusters and features from combinatorial stochastic processes. Available at arXiv:1206.5862 [math.ST].
7. BUSH, C. A. and MACEACHERN, S. N. (1996). A semiparametric Bayesian model for randomised block designs. Biometrika 83 275-285.
8. DAHL, D. B. (2006). Model-based clustering for expression data via a Dirichlet process mixture model. In Bayesian Inference for Gene Expression and Proteomics (K. Do, P. Müller and M. Vannucci, eds.). Cambridge Univ. Press, Cambridge.
9. DALEY, D. J. and VERE-JONES, D. (2002). An Introduction to the Theory of Point Processes. Springer, New York. MR0950166
10. DIEBOLT, J. and ROBERT, C. P. (1994). Estimation of finite mixture distributions through Bayesian sampling. J. R. Stat. Soc. Ser. B Stat. Methodol. 56 363-375. MR1281940
11. ESCOBAR, M. D. (1988). Estimating the means of several normal populations by nonparametric estimation of the distribution of the means. Ph.D. thesis, Yale Univ. MR2637324
12. ESCOBAR, M. D. (1994). Estimating normal means with a Dirichlet process prior. J. Amer. Statist. Assoc. 89 268-277. MR1266299
13. ESCOBAR, M. D. and WEST, M. (1995). Bayesian density estimation and inference using mixtures. J. Amer. Statist. Assoc. 90 577-588. MR1340510
14. EWENS, W. J. (1972). The sampling theory of selectively neutral alleles. Theoret. Population Biology 3 87-112; erratum, ibid. 3 (1972), 240, 376. MR0325177
15. FAVARO, S. and WALKER, S. G. (2013). Slice sampling s - stable Poisson-Kingman mixture models. J. Comput. Graph. Statist. To appear.
16. FERGUSON, T. S. (1973). A Bayesian analysis of some nonparametric problems. Ann. Statist. 1 209-230. MR0350949
17. FERGUSON, T. S. and KLASS, M. J. (1972). A representation of independent increment processes without Gaussian components. Ann. Math. Statist. 43 1634-1643. MR0373022
18. FRALEY, C. and RAFTERY, A. E. (2007). Bayesian regularization for normal mixture estimation and model-based clustering. J. Classification 24 155-181. MR2415725
19. FRITSCH, A. and ICKSTADT, K. (2009). Improved criteria for clustering based on the posterior similarity matrix. Bayesian Anal. 4 367-391. MR2507368
20. GASTHAUS, J., WOOD, F., GöRüR, D. and TEH, Y. W. (2009). Dependent Dirichlet process spike sorting. In Advances in Neural Information Processing Systems 21 497- 504.
21. GILKS, W. R. andWILD, P. (1992). Adaptive rejection sampling for Gibbs sampling. Appl. Statist. 41 337-348.
22. GNEDIN, A. and PITMAN, J. (2006). Exchangeable Gibbs partitions and Stirling triangles. J. Math. Sci. 138 5674-5684.
23. GöRüR, D., RASMUSSEN, C. E., TOLIAS, A. S., SINZ, F. and LOGOTHETIS, N. K. (2004).Modelling spikes with mixtures of factor analysers. In Proceedings of the Conference of the German Association for Pattern Recognition (DAGM).
24. GREEN, P. J. (1995). Reversible jump Markov chain Monte Carlo computation and Bayesian model determination. Biometrika 82 711-732. MR1380810
25. GREEN, P. J. and RICHARDSON, S. (2001). Modelling heterogeneity with and without the Dirichlet process. Scand. J. Stat. 28 355-375. MR1842255
26. GRIFFIN, J. E., KOLOSSIATIS, M. and STEEL, M. F. J. (2013). Comparing distributions using dependent normalized random measure mixtures. J. R. Stat. Soc. Ser. B Stat. Methodol. 75 499-529.
27. GRIFFIN, J. E. and WALKER, S. G. (2011). Posterior simulation of normalized random measure mixtures. J. Comput. Graph. Statist. 20 241-259. MR2816547
28. GRIFFITHS, T. L. and GHAHRAMANI, Z. (2011). The Indian buffet process: An introduction and review. J. Mach. Learn. Res. 12 1185-1224. MR2804598
29. HJORT, N. L. (1990). Nonparametric Bayes estimators based on beta processes in models for life history data. Ann. Statist. 18 1259-1294. MR1062708
30. HJORT, N. L., HOLMES, C., MüLLER, P. and WALKER, S. G., eds. (2010). Bayesian Nonparametrics. Cambridge Series in Statistical and Probabilistic Mathematics 28. Cambridge Univ. Press, Cambridge. MR2722987
31. ISHWARAN, H. and JAMES, L. F. (2001). Gibbs sampling methods for stick-breaking priors. J. Amer. Statist. Assoc. 96 161-173. MR1952729
32. JAIN, S. and NEAL, R. M. (2000). A split-merge Markov chain Monte Carlo procedure for the Dirichlet process mixture model. Unpublished manuscript.
33. JAMES, L. F. (2002). Poisson process partition calculus with applications to exchangeable models and Bayesian nonparametrics. Available at arXiv:math/0205093v1.
34. JAMES, L. F. (2003). A simple proof of the almost sure discreteness of a class of random measures. Statist. Probab. Lett. 65 363-368. MR2039881
35. JAMES, L. F., LIJOI, A. and PRüNSTER, I. (2006). Conjugacy as a distinctive feature of the Dirichlet process. Scand. J. Stat. 33 105-120. MR2255112
36. JAMES, L. F., LIJOI, A. and PRüNSTER, I. (2009). Posterior analysis for normalized random measures with independent increments. Scand. J. Stat. 36 76-97. MR2508332
37. JAMES, L. F., LIJOI, A. and PRüNSTER, I. (2010). On the posterior distribution of classes of random means. Bernoulli 16 155-180. MR2648753
38. JASRA, A., HOLMES, C. C. and STEPHENS, D. A. (2005). Markov chain Monte Carlo methods and the label switching problem in Bayesian mixture modeling. Statist. Sci. 20 50-67. MR2182987
39. KALLI, M., GRIFFIN, J. E. and WALKER, S. G. (2011). Slice sampling mixture models. Stat. Comput. 21 93-105. MR2746606
40. KINGMAN, J. F. C. (1967). Completely random measures. Pacific J. Math. 21 59-78. MR0210185
41. KINGMAN, J. F. C. (1993). Poisson Processes. Oxford Studies in Probability 3. Clarendon Press, Oxford. MR1207584
42. KINGMAN, J. F. C., TAYLOR, S. J., HAWKES, A. G., WALKER, A. M., COX, D. R., SMITH, A. F. M., HILL, B. M., BURVILLE, P. J. and LEONARD, T. (1975). Random discrete distribution. J. R. Stat. Soc. Ser. B Stat. Methodol. 37 1-22. MR0368264
43. LAU, J. W. and GREEN, P. J. (2007). Bayesian model-based clustering procedures. J. Comput. Graph. Statist. 16 526-558. MR2351079
44. LEWICKI, M. S. (1998). A review of methods for spike sorting: The detection and classification of neural action potentials. Network 9 53-78.
45. LEWIS, P. A. W. and SHEDLER, G. S. (1979). Simulation of nonhomogeneous Poisson processes by thinning. Naval Res. Logist. Quart. 26 403-413. MR0546120
46. LIJOI, A.,MENA, R. H. and PRüNSTER, I. (2005). Bayesian nonparametric analysis for a generalized Dirichlet process prior. Stat. Inference Stoch. Process. 8 283-309. MR2177315
47. LIJOI, A., MENA, R. H. and PRüNSTER, I. (2005). Hierarchical mixture modeling with normalized inverse-Gaussian priors. J. Amer. Statist. Assoc. 100 1278-1291. MR2236441
48. LIJOI, A.,MENA, R. H. and PRüNSTER, I. (2007). Controlling the reinforcement in Bayesian non-parametric mixture models. J. R. Stat. Soc. Ser. B Stat. Methodol. 69 715-740. MR2370077
49. LIJOI, A. and PRüNSTER, I. (2010). Models beyond the Dirichlet process. In Bayesian Nonparametrics (N. L. Hjort, C. C. Holmes, P. Müller and S. G.Walker, eds.) 80-136. Cambridge Univ. Press, Cambridge. MR2730661
50. LIJOI, A., PRüNSTER, I. and WALKER, S. G. (2008). Investigating nonparametric priors with Gibbs structure. Statist. Sinica 18 1653-1668. MR2469329
51. LO, A. Y. (1984). On a class of Bayesian nonparametric estimates. I. Density estimates. Ann. Statist. 12 351-357. MR0733519
52. MACEACHERN, S. N. (1994). Estimating normal means with a conjugate style Dirichlet process prior. Comm. Statist. Simulation Comput. 23 727-741. MR1293996
53. MACEACHERN, S. N. (1998). Computational methods for mixture of Dirichlet process models. In Practical Nonparametric and Semiparametric Bayesian Statistics (D. Dey, P. Müller and D. Sinha, eds.). Lecture Notes in Statist. 133 23- 43. Springer, New York. MR1630074
54. MACEACHERN, S. N. and MüLLER, P. (1998). Estimating mixture of Dirichlet process models. J. Comput. Graph. Statist. 7 223-238.
55. MCLACHLAN, G. J. and BASFORD, K. E. (1988). Mixture Models: Inference and Applications to Clustering. Statistics: Textbooks and Monographs 84. Dekker, New York. MR0926484
56. MEDVEDOVIC, M. and SIVAGANESAN, S. (2002). Bayesian infinite mixture model based clustering of gene expression profiles. Bioinformatics 18 1194-1206.
57. MENGERSEN, K. L. and ROBERT, C. P. (1996). Testing for mixtures: A Bayesian entropic approach. In Bayesian Statistics, 5 (Alicante, 1994) (J. O. Berger, J. M. Bernardo, A. P. Dawid, D. V. Lindley and A. F. M. Smith, eds.) 255- 276. Oxford Univ. Press, New York. MR1425410
58. MULIERE, P. and TARDELLA, L. (1998). Approximating distributions of random functionals of Ferguson-Dirichlet priors. Canad. J. Statist. 26 283-297. MR1648431
59. MüLLER, P., ERKANLI, A. andWEST, M. (1996). Bayesian curve fitting using multivariate normal mixtures. Biometrika 83 67-79. MR1399156
60. NEAL, R. M. (1992). Bayesian mixture modeling. In Proceedings of the 11th International Workshop on Maximum Entropy and Bayesian Methods of Statistical Analysis, Seattle. Kluwer, Dordrecht.
61. NEAL, R. M. (2000). Markov chain sampling methods for Dirichlet process mixture models. J. Comput. Graph. Statist. 9 249-265. MR1823804
62. NEAL, R. M. (2003). Slice sampling. Ann. Statist. 31 705- 767. MR1994729
63. NIETO-BARAJAS, L. E. and PRüNSTER, I. (2009). A sensitivity analysis for Bayesian nonparametric density estimators. Statist. Sinica 19 685-705. MR2514182
64. NIETO-BARAJAS, L. E., PRüNSTER, I. andWALKER, S. G. (2004). Normalized random measures driven by increasing additive processes. Ann. Statist. 32 2343-2360. MR2153987
65. NOBILE, A. (1994). Bayesian analysis of finite mixture distributions. Ph.D. thesis, Carnegie Mellon Univ. MR2692049
66. OGATA, Y. (1981). On Lewis' simulation method for Point processes. IEEE Trans. Inform. Theory 27 23-31.
67. PAPASPILIOPOULOS, O. (2008). A note on posterior sampling from Dirichlet mixture models.Working Paper 20, Centre for Research in Statistical Methodology, Univ. Warwick.
68. PAPASPILIOPOULOS, O. and ROBERTS, G. O. (2008). Retrospective Markov chain Monte Carlo methods for Dirichlet process hierarchical models. Biometrika 95 169-186. MR2409721
69. PERMAN, M., PITMAN, J. and YOR, M. (1992). Size-biased sampling of Poisson point processes and excursions. Probab. Theory Related Fields 92 21-39. MR1156448
70. PITMAN, J. (2003). Poisson-Kingman partitions. In Statistics and Science: A Festschrift for Terry Speed (D.R. Goldstein, ed.). Institute of Mathematical Statistics Lecture Notes-Monograph Series 40 1-34. IMS, Beachwood, OH. MR2004330
71. PITMAN, J. (2006). Combinatorial Stochastic Processes. Lecture Notes in Math. 1875. Springer, Berlin. MR2245368
72. PITMAN, J. and YOR, M. (1997). The two-parameter Poisson-Dirichlet distribution derived from a stable subordinator. Ann. Probab. 25 855-900. MR1434129
73. QUIROGA, R. Q. (2007). Spike sorting. Scholarpedia 2 3583.
74. RAFTERY, A. E. (1996). Hypothesis testing and model selection. In Markov Chain Monte Carlo in Practice (W. R. Gilks, S. Richardson and D. J. Spiegelhalter, eds.). Chapman & Hall, London.
75. RAFTERY, A. E. (1996). Hypothesis testing and model selection via posterior simulation. In Markov Chain Monte Carlo in Practice (W. R. Gilks, S. Richardson and D. J. Spiegelhalter, eds.). Chapman & Hall, London.
76. RASMUSSEN, C. E.,DE LA CRUZ, B. J.,GHAHRAMANI, Z. and WILD, D. L. (2009). Modeling and visualizing uncertainty in gene expression clusters using Dirichlet process mixtures. IEEE/ACM Trans. Comput. Biol. and Bioinform. 6 615- 628.
77. RASMUSSEN, C. E. andWILLIAMS, C. K. I. (2006). Gaussian Processes for Machine Learning. MIT Press, Cambridge, MA. MR2514435
78. REGAZZINI, E., LIJOI, A. and PRüNSTER, I. (2003). Distributional results for means of normalized random measures with independent increments. Ann. Statist. 31 560-585. MR1983542
79. RICHARDSON, S. and GREEN, P. J. (1997). On Bayesian analysis of mixtures with an unknown number of components. J. R. Stat. Soc. Ser. B Stat. Methodol. 59 731-792. MR1483213
80. ROEDER, K. (1994). A graphical technique for determining the number of components in a mixture of normals. J. Amer. Statist. Assoc. 89 487-495. MR1294074
81. ROEDER, K. and WASSERMAN, L. (1997). Practical Bayesian density estimation using mixtures of normals. J. Amer. Statist. Assoc. 92 894-902. MR1482121
82. STEPHENS, M. (2000). Bayesian analysis of mixture models with an unknown number of components-an alternative to reversible jump methods. Ann. Statist. 28 40-74. MR1762903
83. TITTERINGTON, D. M., SMITH, A. F. M. and MAKOV, U. E. (1985). Statistical Analysis of Finite Mixture Distributions. Wiley, Chichester. MR0838090
84. TRIPPA, L. and FAVARO, S. (2012). A class of normalized random measures with an exact predictive sampling scheme. Scand. J. Stat. 39 444-460. MR2971631
85. WALKER, S. G. (2007). Sampling the Dirichlet mixture model with slices. Comm. Statist. Simulation Comput. 36 45- 54. MR2370888
86. WOOD, F. and BLACK, M. J. (2008). A nonparametric Bayesian alternative to spike sorting. Journal of Neuroscience Methods 173 1-12.