Regression density estimation with variational methods and stochastic approximation

Journal of Computational and Graphical Statistics

Volumn 21, Issue 3, 2012, Pages 797-820

  Nott, David J ^a   Tan, Siew Li ^a   Villan, Mattias ^b   Kohn, Robert ^c  

^a NATIONAL UNIVERSITY OF SINGAPORE   (Singapore)

^b LINKÖPING UNIVERSITY   (Sweden)

^c UNIVERSITY OF NEW SOUTH WALES   (Australia)

Author keywords

Bayesian model selection; Heteroscedasticity; Mixtures of experts; Stochastic approximation; Variational bayes

Indexed keywords

EID: 84865363413     PISSN: 10618600     EISSN: None     Source Type: Journal    
DOI: 10.1080/10618600.2012.679897     Document Type: Article

References (47)

1. Biernacki, C., Celeux, G., and Govaert, G. (2003), "Choosing Starting Values for theEMAlgorithm for Getting the Highest Likelihood in Multivariate Gaussian Mixture Models," Computational Statistics and Data Analysis, 41, 561-575.[804]
2. Bishop, C. M. (2006), Pattern Recognition and Machine Learning, New York: Springer.[801,806]
3. Bishop, C. M., and Svenśen, M. (2003), "Bayesian Hierarchical Mixtures of Experts," in Proceedings of the 19th Conference on Uncertainty in Artificial Intelligence, eds. U. Kjaerulffand C. Meek, Waltham, MA: Morgan Kaufmann, p. 57-64.[800,801,805]
4. Blei, D. M., and Jordan, M. I. (2006), Variational Inference for Dirichlet Process Mixtures," Bayesian Analysis, 1, 121-144.[800,802]
5. Braun, M., and McAuliffe, J. (2010), "Variational Inference for Large-Scale Models of Discrete Choice," Journal of the American Statistical Association, 105, 324-335. [803]
6. Boughton, W. (2004), "The Australian Water Balance Model," Environmental Modelling and Software, 19, 943-956.[810]
7. Constantinopoulos, C., and Likas, A. (2007), "Unsupervised Learning of Gaussian Mixtures Based on Variational Component Splitting," IEEE Transactions on Neural Networks, 18, 745-755.[800]
8. Corduneanu, A., and Bishop, C. M. (2001), "Variational Bayesian Model Selection for Mixture Distributions," in Artificial Intelligence and Statistics, eds. T. Jaakkolaand T. Richardson, Waltham, MA: Morgan Kaufmann, pp. 27-34.[800]
9. de Freitas, N., Højen-Sørensen, P., Jordan, M. I., and Russell, S. (2001), "Variational MCMC," in Uncertainty in Artificial Intelligence (UAI): Proceedings of the 17th Conference, eds. J. Breeseand D. Koller, San Francisco, CA: Morgan Kaufmann, pp. 120-127.[818]
10. De Iorio, M., M̈uller, P., Rosner, G. L., and MacEAchern, S. N. (2004), "An ANOVA Model for Dependent Random Measures," Journal of the American Statistical Association, 99, 205-215.[799]
11. Dunson, D. B., Pillai, N., and Park, J.-H. (2007), "Bayesian Density Regression," Journal of the Royal Statistical Society, Series B, 69, 163-183.[800]
12. Fr̈uhwirth-Schnatter, S. (2004), "Estimating Marginal Likelihoods for Mixture and Markov Switching Models Using Bridge Sampling Techniques," The Econometrics Journal, 7, 143-167.[806]
13. Geweke, J., and Amisano, G. (2010), "Comparing and Evaluating Bayesian Predictive Distributions of Asset Returns," International Journal of Forecasting, 26, 216-230.[798]
14. Geweke, J., and Keane, M. (2007), "Smoothly Mixing Regressions," Journal of Econometrics, 138, 252-291.[799,807,815]
15. Ghahramani, Z., and Beal, M. J. (2000), "Variational Inference for Bayesian Mixtures of Factor Analysers," in Advances in Neural Information Processing Systems (Vol. 12), eds. S. A. Solla, T. K. Leen, and K-R M̈uller, Cambridge: MIT Press, pp. 831-864.[800]
16. Griffin, J. E., and Steel, M. F. J. (2006), "Order-Based Dependent Dirichlet Processes," Journal of the American Statistical Association, 101, 179-194.[799]
17. Honkela, A., and Valpola, H. (2003), "On-Line Variational Bayesian Learning," in Proceedings of the 4th International Symposium on Independent Component Analysis and Blind Signal Separation (ICA 2003), Berlin: Springer, pp. 803-808.[806]
18. Jacobs, R., Jordan, M., Nowlan, S., and Hinton, G. (1991), "Adaptive Mixtures of Local Experts," Neural Computation, 3, 79-87.[798,801]
19. Ji, C., Shen, H., and West, M. (2010), "Bounded Approximations for Marginal Likelihoods," Technical Report, ISDS, Duke University. Available at http://ftp.stat.duke.edu/WorkingPapers/10-05.html[800,808,809]
20. Jiang, W., and Tanner, M. (1999), "Hierarchical Mixtures-of-Experts for Exponential Family Regression Models: Approximation and Maximum Likelihood Estimation," The Annals of Statistics, 27, 987-1011.[799]
21. Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., and Saul, L. K. (1999), "An Introduction to Variational Methods for Graphical Models," in Learning in Graphical Models, ed. M. I. Jordan, Cambridge, MA: MIT Press, pp. 105-158.[801]
22. Jordan, M. I., and Jacobs, R. A. (1994), "Hierarchical Mixtures of Experts and the EM Algorithm," Neural Computation, 6, 181-214.[798,800,801]
23. Li, F., Villani, M., and Kohn, R. (2010a), "Modeling Conditional Densities Using Finite Smooth Mixtures," in Mixtures: Estimation and Applications, eds.K. L.Mengersen, C. P. Robert, and D. M. Titterington, Chichester, UK: John Wiley & Sons, Ltd.
24. Li, F., Villani, M., and Kohn, R. (2010b), "Flexible Modeling of Conditional Distributions Using SmoothMixtures of Asymmetric Student t Densities," Journal of Statistical Planning and Inference, 140, 3638-3654.[807,815]
25. MacEachern, S. N. (1999), "Dependent Nonparametric Processes," in ASA Proceedings of the Section on Bayesian Statistical Science, Alexandria, VA: American Statistical Association, pp. 50-55.[799]
26. McCullagh, P., and Nelder, J. A. (1989), Generalized Linear Models (2nd ed.), London: Chapman and Hall.[804]
27. McGrory, C. A., and Titterington, D. M. (2007), "Variational Approximations in Bayesian Model Selection for Finite Mixture Distributions," Computational Statistics and Data Analysis, 51, 5352-5367.[800,804]
28. Norets, A. (2010), "Approximation of Conditional Densities by Smooth Mixtures of Regressions," The Annals of Statistics, 38, 1733-1766.[799]
29. O'Hagan, A. (2006), "Bayesian Analysis of Computer Code Outputs: A Tutorial," Reliability Engineering and System Safety, 91, 1290-1300.[811]
30. Ormerod, J. T., andWand, M. P. (2010), "Explaining Variational Approximations," The American Statistician, 64 (2), 140-153.[801,805]
31. Peng, F., Jacobs, R. A., and Tanner, M. A. (1996), "Bayesian Inference in Mixtures-of-Experts and Hierarchical Mixtures-of-ExpertsModels with an Application to Speech Recognition," Journal of the American Statistician Association, 91, 953-960.[799]
32. Pepelyshev, A. (2010), "The Role of the Nugget Term in the Gaussian Process Method," in MODA 9-Advances in Model-Oriented Design and Analysis: Contributions to Statistics, New York: Springer, pp. 149-156.[811]
33. Pesaran, M. H., and Timmermann, A. (2002), "Market Timing and Return Prediction Under Model Instability," Journal of Empirical Finance, 9, 495-510.[798]
34. Robbins, H., and Monro, S. (1951), "A Stochastic Approximation Method," Annals of Mathematical Statistics, 22, 400-407.[808]
35. Spall, J. C. (2003), Introduction to Stochastic Search and Optimization: Estimation, Simulation and Control, Hoboken, NJ: Wiley.[798,810]
36. Spiegelhalter, D. J., Best, N. G., Carlin, B. P., and Van der Linde, A. (2002), "Bayesian Measures of Model Complexity and Fit" (with discussion), Journal of the Royal Statistical Society, Series B, 64, 583-616.[800]
37. Ueda, N., and Ghahramani, Z. (2002), "Bayesian Model Search for Mixture Models Based on Optimizing Variational Bounds," Neural Networks, 15, 1223-11241.[800,801]
38. Vehtari, A., and Lampinen, J. (2002), "Bayesian Model Assessment and Comparison Using Cross-Validation Predictive Densities," Neural Computation, 14, 2439-2468.[807]
39. Villani, M., Kohn, R., and Giordani, P. (2009), "Regression Density Estimation Using Smooth Adaptive Gaussian Mixtures," Journal of Econometrics, 153, 155-173.[798,799]
40. Waterhouse, S., MacKay, D., and Robinson, T. (1996), "Bayesian Methods for Mixtures of Experts," in Advances in Neural Information Processing Systems, (Vol. 8), eds. D. S. Touretzky, M. C. Mozer, and M. E. Hasselmo, Cambridge, MA: MIT Press, pp. 351-357.[800,801,805]
41. Wand, M. P. (2002), "Vector Differential Calculus in Statistics," The American Statistician, 56, 55-62.[803]
42. Wang, B., and Titterington, D. M. (2005), "Inadequacy of Interval Estimates Corresponding toVariationalBayesian Approximations," in Proceedings of the 10th International Workshop on Artificial Intelligence and Statistics, eds. R. G. Cowell and Z. Ghahramani, Society for Artificial Intelligence and Statistics, pp. 373-380.[808]
43. Wedel, M. (2002), "Concomitant Variables in Finite MixtureModels," Statistica Neerlandica, 56, 362-375.[798]
44. West,M. (1985), "Generalized Linear Models:Outlier Accommodation, Scale Parameters and PriorDistributions," in Bayesian Statistics 2, eds. J.M. Bernardo, M. H. DeGroot,D.V. Lindley, and A. F. M. Smith,North Holland: Amsterdam, pp. 531-538.[804]
45. Wood, S. A., Jiang, W., and Tanner, M. A. (2002), "Bayesian Mixture of Splines for Spatially Adaptive Nonparametric Regression," Biometrika, 89, 513-528.[799]
46. Wood, S. A., Kohn, R., Cottet, R., Jiang, W., and Tanner, M. (2008), "Locally Adaptive Nonparametric Binary Regression," Journal of Computational and Graphical Statistics, 17, 352-372.[799]
47. Wu, B., McGrory, C. A., and Pettitt, A. N. (2012), "A New Variational Bayesian Algorithm With Application to Human Mobility Pattern Modeling," Statistics and Computing, 22 (1), 185-203.[800]