Bayesian non-parametrics and the probabilistic approach to modelling

Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences

Volumn 371, Issue 1984, 2013, Pages

  Ghahramani, Zoubin ^a  

^a UNIVERSITY OF CAMBRIDGE   (United Kingdom)

Author keywords

Bayesian statistics; Machine learning; Non parametrics; Probabilistic modelling

Indexed keywords

HIDDEN MARKOV MODELS; LEARNING SYSTEMS; PROBABILITY DENSITY FUNCTION; SURVEYS;

BAYESIAN MODELLING; BAYESIAN NONPARAMETRICS; BAYESIAN STATISTICS; COVARIANCE STRUCTURES; DENSITY ESTIMATION; DIRICHLET; DIRICHLET PROCESS; GAUSSIAN PROCESSES; MODEL PARAMETERS; NON-PARAMETRICS; PROBABILISTIC APPROACHES; PROBABILISTIC MODELLING; PROBABILISTIC MODELS; PROBABILITY THEORY; SCIENCE AND ENGINEERING; TIME-SERIES MODELLING;

UNCERTAINTY ANALYSIS;

EID: 84874126433     PISSN: 1364503X     EISSN: None     Source Type: Journal    
DOI: 10.1098/rsta.2011.0553     Document Type: Article

References (86)

1. Wolpert DM, Ghahramani Z, Jordan MI. 1995 An internal model for sensorimotor integration. Science 269, 1880-1882. (doi:10.1126/science.7569931)
2. Knill D, Richards W. 1996 Perception as Bayesian inference. Cambridge, UK: Cambridge University Press.
3. Griffiths TL, Tenenbaum JB. 2006 Optimal predictions in everyday cognition. Psychol. Sci. 17, 767-773. (doi:10.1111/j.1467-9280.2006.01780.x)
4. Doob JL. 1949 Application of the theory of martingales. Coll. Int. Centre Nat. Res. Sci. 13, 23-27.
5. Le Cam L. 1986 Asymptotic methods in statistical decision theory. Berlin, Germany: Springer.
6. Van der Vaart A. 2000 Asymptotic statistics. Cambridge, UK: Cambridge University Press.
7. Jefferys WH, Berger JO. 1992 Ockham's razor and Bayesian analysis. Am. Sci. 80, 64-72.
8. MacKay DJC. 2003 Information theory, inference, and learning algorithms. Cambridge, UK: Cambridge University Press.
9. Rasmussen CE, Ghahramani Z. 2001 Occam's razor. In Advances in Neural Information Processing Systems (eds TK Leen, TG Dietterich and V Tresp), vol. 13, pp. 294-300. Cambridge, MA: MIT Press.
10. MacKay DJC. 1991 Bayesian methods for adaptive models. PhD thesis, California Institute of Technology.
11. Murray IA, Ghahramani, Z. 2005 A note on the evidence and Bayesian Occam's razor. Technical Report GCNU-TR 2005-003, Gatsby Unit, University College London, London, UK.
12. Roberts SJ, Husmeier D, Rezek I, Penny W. 1998 Bayesian approaches to Gaussian mixture modeling. IEEE Trans. Pattern Anal. Mach. Intell. 20, 1133-1142. (doi:10.1109/34.730550)
13. Ghahramani Z, Beal MJ. 2000 Variational inference for Bayesian mixtures of factor analysers. In Advances in Neural Information Processing Systems, vol. 12, pp. 449-455.
14. MacKay DJC. 1994 Bayesian nonlinear modeling for the prediction competition. ASHRAE Trans. 100, 1053-1062.
15. Neal RM. 1998 Assessing relevance determination methods using DELVE. In Neural networks and machine learning (ed. CM Bishop), pp. 97-129. Berlin, Germany: Springer.
16. MacKay DJC. 1997 Ensemble learning for hidden Markov models. Technical report, Cavendish Laboratory, University of Cambridge, Cambridge, UK.
17. Friedman N, Koller D. 2003 Being Bayesian about network structure: a Bayesian approach to structure discovery in Bayesian networks. Mach. Learn. 50, 95-126. (doi:10.1023/A: 1020249912095)
18. Zellner A. 1988 Optimal information processing and Bayes's theorem. Am. Stat. 42, 278-280. (doi:10.2307/2685148)
19. Kallenberg O. 2005 Probabilistic symmetries and invariance principles. Berlin, Germany: Springer.
20. Neal RM. 2000 Markov chain sampling methods for Dirichlet process mixture models. J. Comput. Graph. Stat. 9, 249-265. (doi:10.1080/10618600.2000. 10474879)
21. Rasmussen CE, Williams CKI. 2006 Gaussian processes for machine learning. Cambridge, MA: MIT Press.
22. Ferguson TS. 1973 A Bayesian analysis of some nonparametric problems. Ann. Stat. 1, 209-230. (doi:10.1214/aos/1176342360)
23. Teh YW. 2010 Dirichlet processes. In Encyclopedia of machine learning (eds C Sammut, GIWebb), pp. 280-287. Berlin, Germany: Springer.
24. Antoniak CE. 1974 Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems. Ann. Stat. 2, 1152-1174. (doi:10.1214/aos/ 1176342871)
25. Ferguson TS. 1983 Bayesian density estimation by mixtures of normal distributions. Recent Adv. Stat. 287-303.
26. Lo A. 1984 On a class of Bayesian nonparametric estimates. I. Density estimates. Ann. Stat. 12, 351-357. (doi:10.1214/aos/1176346412)
27. Rasmussen, CE. 2000 The infinite Gaussian mixture model. In Advances in Neural Information Processing Systems, vol. 12, pp. 554-560.
28. Duda RO, Hart PE. 1973 Pattern classification and scene analysis. London, UK:Wiley.
29. Aldous D. 1985 Exchangeability and related topics. In École d'été de probabilités de Saint-Flour XIII-1983, pp. 1-198. Berlin, Germany: Springer.
30. Chung SH, Moore JB, Xia L, Premkumar LS, Gage PW. 1990 Characterization of single channel currents using digital signal processing techniques based on Hidden Markov models. Phil. Trans. R. Soc. Lond. B 329, 265-285. (doi:10.1098/rstb.1990.0170)
31. Rabiner LR, Juang, BH. 1986 An introduction to hidden Markov models. IEEE Acoust. Speech Signal Process. Mag. 3, 4-16. (doi:10.1109/MASSP.1986.1165342)
32. Beal MJ, Ghahramani Z, Rasmussen CE. 2002 The infinite hidden Markov model. In Advances in Neural Information Processing Systems, vol. 14, pp. 577-584.
33. Fortini S, Petrone S. 2012 Hierarchical reinforced urn processes. Stat. Probab. Lett. 82, 1521- 1529. (doi:10.1016/j.spl.2012.04.012)
34. Teh YW, Jordan MI, BealMJ, Blei DM. 2006 Hierarchical Dirichlet processes. J. Am. Stat. Assoc. 101, 1566-1581. (doi:10.1198/016214506000000302)
35. Van Gael J, Saatci Y, Teh YW, Ghahramani Z. 2008 Beam sampling for the infinite hidden Markov model. In Proc. 25th Int. Conf. Machine Learning, Helsinki, Finland, 5-9 June 2008 (eds WW Cohen, A McCallum, ST Roweis), pp. 1088-1095.
36. Bratieres S, Van Gael J, Vlachos A, Ghahramani Z. 2010 Scaling the iHMM: parallelization versus Hadoop. In IEEE Int. Conf. Computing and Information Technology, Bradford, UK, 29 June-1 July 2010.
37. Stepleton T, Ghahramani Z, Gordon G, Lee T-S. 2009 The block diagonal infinite hidden Markov model. In 12th Int. Conf. Artificial Intelligence Statistics, vol. 5 (eds D van Dyk, M Welling), pp. 552-559. Brookline, MA: Microtome Publishing.
38. Van Gael J, Ghahramani Z. 2011 Nonparametric hidden Markov models. In Bayesian time series models (eds D Barber, A Cemgil, S Chiappa), pp. 317-340. Cambridge, UK: Cambridge University Press.
39. Blunsom P, Cohn T. 2011 A hierarchical Pitman-Yor process HMM for unsupervised part of speech induction. In Proc. 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, 19-24 June 2011, Portland, OR, vol. 1, pp. 865-874. Association for Computational Linguistics.
40. Fox EB, Sudderth EB, Jordan MI, Willsky AS. 2008 An HDP-HMM for systems with state persistence. In Proc. 25th Int. Conf. Machine Learning, Helsinki, Finland, vol. 25. ACM Press.
41. Hinton GE, Zemel RS. 1994 Autoencoders, minimum description length, and Helmholtz free energy. In Advances in Neural Information Processing Systems, vol. 6 (eds J Cowan, G Tesauro, J Alspector), pp. 3-10. San Francisco, CA: Morgan Kaufmann Publishers.
42. Saund E. 1994 Unsupervised learning of mixtures of multiple causes in binary data. In Advances in neural information processing systems, vol. 6 (eds J Cowan, G Tesauro, J Alspector), pp. 27-34. San Francisco, CA: Morgan Kaufmann.
43. Ghahramani Z. 1995 Factorial learning and the EM algorithm. In Advances in neural information processing systems, vol. 7 (eds G Tesauro, DS Touretzky, TK Leen), pp. 617-624. Cambridge, MA: MIT Press.
44. Griffiths TL, Ghahramani Z. 2006 Infinite latent feature models and the Indian buffet process. In Advances in neural information processing systems, vol. 18 (eds B Schölkopf, JC Platt, T Hoffman), pp. 475-482. Cambridge, MA: MIT Press.
45. Griffiths TL, Ghahramani Z. 2011 The Indian buffet process: an introduction and review. J. Mach. Learn. Res. 12, 1185-1224.
46. Knowles DA, Ghahramani Z. 2007 Infinite sparse factor analysis and infinite independent components analysis. In 7th Int. Conf. Independent Component Analysis and Signal Separation (eds ME Davies, CJ James, SA Abdallah, MD Plumbley), pp. 381-388. London, UK: Springer.
47. Meeds E, Ghahramani Z, Neal R, Roweis S. 2007 Modelling dyadic data with binary latent factors. In Advances in Neural Information Processing Systems, vol. 19 (eds B Schölkopf, J Platt, T Hofmann), pp. 977-984. Cambridge, MA: MIT Press.
48. Adams RP, Wallach H, Ghahramani Z. 2010 Learning the structure of deep sparse graphical models. In 13th Int. Conf. Artificial Intelligence and Statistics, Sardinia, Italy (eds YW Teh, M Titterington), pp. 1-8.
49. Thibaux R, Jordan MI. 2007 Hierarchical beta processes and the Indian buffet process. In 11th Int. Conf. Artificial Intelligence and Statistics (AISTATS 2007), San Juan, Puerto Rico, 21-24 March 2007 (eds M. Meila, X Shen).
50. Hjort NL. 1990 Nonparametric Bayes estimators based on Beta processes in models for life history data. Ann. Stat. 18, 1259-1294. (doi:10.1214/aos/ 1176347749)
51. Teh YW, Görür D. 2009 Indian buffet processes with power-law behavior. In Advances in Neural Information Processing Systems, vol. 22, pp. 1838-1846.
52. Newman, M. 2010 Networks: an introduction. Oxford, UK: Oxford University Press.
53. Miller K, Griffiths T, Jordan M. 2009 Nonparametric latent feature models for link prediction. In Advances in Neural Information Processing Systems, vol. 22, pp. 1276-1284.
54. Mørup M, Schmidt M, Hansen L. 2011 Infinite multiple membership relational modeling for complex networks. In IEEE Int. Workshop on Machine Learning for Signal Processing (MLSP), Beijing, China, 18-21 September 2011, pp. 1-6. IEEE.
55. Palla K, Knowles DA, Ghahramani Z. 2012 An infinite latent attribute model for network data. In Proc. 29th Int. Conf. Machine Learning, Edinburgh, UK, July 2012 (eds J Langford, J Pineau), pp. 1607-1614. Omnipress.
56. Van Gael J, Teh YW, Ghahramani Z. 2008 The infinite factorial hidden Markov model. In Advances in Neural Information Processing Systems, vol. 21, pp. 1697-1704.
57. Orbanz P. 2009 Construction of nonparametric Bayesian models from parametric Bayes equations. In Advances in Neural Information Processing Systems, vol. 22, pp. 1392-1400.
58. Kingman JFC. 1982 The coalescent. Stoch. Process. Appl. 13, 235-248. (doi:10.1016/0304- 4149(82)90011-4)
59. Donnelly P, Tavare S. 1995 Coalescents and genealogical structure under neutrality. Annu. Rev. Genet. 29, 401-421. (doi:10.1146/annurev.ge.29.120195. 002153)
60. Teh YW, Daumé H, Roy DM. 2008 Bayesian agglomerative clustering with coalescents. In Advances in Neural Information Processing Systems, vol. 22, pp. 1473-1480.
61. Neal RM. 2003. Density modeling and clustering using Dirichlet diffusion trees. In Bayesian statistics, vol. 7 (ed. Bernardo JM), pp. 619-629. Oxford University Press.
62. Knowles DA, Ghahramani Z. 2011 Pitman-Yor diffusion trees. In Uncertainty in Artificial Intelligence (UAI 2011), Barcelona, Spain, 14-17 July 2011 (eds FG Cozman, A Pfeffer), pp. 410- 418. AUAI Press.
63. Pitman J, YorM. 1997 The two-parameter Poisson-Dirichlet distribution derived from a stable subordinator. Ann. Probab. 25, 855-900. (doi:10.1214/aop/1024404422)
64. McCullagh P, Pitman J, Winkel M. 2008 Gibbs fragmentation trees. Bernoulli 14, 988-1002.(doi:10.3150/08-BEJ134)
65. Teh YW, Blundell C, Elliott LT. 2011 Modelling genetic variations with fragmentation- coagulation processes. In Advances in Neural Information Processing Systems, vol. 23, pp. 819- 827.
66. Muirhead R. 1982 Aspects of multivariate statistical theory, vol. 42. London, UK: Wiley.
67. Asai M, McAleer M, Yu J. 2006 Multivariate stochastic volatility: a review. Econ. Rev. 25, 145- 175. (doi:10.1080/07474930600713564)
68. Bru M. 1991Wishart processes. J. Theor. Probab. 4, 725-751. (doi:10.1007/BF01259552)
69. Philipov A, Glickman M. 2006 Multivariate stochastic volatility via Wishart processes. J. Bus. Econ. Stat. 24, 313-328. (doi:10.1198/ 073500105000000306)
70. Gouriéroux C, Jasiak J, Sufana R. 2009 The Wishart autoregressive process of multivariate stochastic volatility. J. Econ. 150, 167-181. (doi:10.1016/j.jeconom.2008.12.016)
71. Wilson AG, Ghahramani Z. 2011 Generalised Wishart processes. In Uncertainty in Artificial Intelligence, Barcelona, Spain, 14-17 July 2011 (eds FG Cozman, A Pfeffer), pp. 736-744. AUAI Press.
72. Neal RM. 1993 Probabilistic inference using Markov chain Monte Carlo methods. Technical Report CRG-TR-93-1, Department of Computer Science, University of Toronto.
73. Propp JG, Wilson DB. 1996 Exact sampling with coupled Markov chains and applications to statistical mechanics. Random Struct. Algorithms 9, 223-252. (doi:10.1002/(SICI)1098-2418 (199608/09)9:1/2<223::AID-RSA14>3.0.CO;2-O)
74. Doucet A, de Freitas JFG, Gordon NJ. 2000 Sequential Monte Carlo methods in practice. New York, NY: Springer.
75. Jordan MI, Ghahramani Z, Jaakkola TS, Saul LK. 1999 An introduction to variational methods for graphical models. Mach. Learn. 37, 183-233. (doi:10.1023/A:1007665907178)
76. Minka TP. 2001 Expectation propagation for approximate Bayesian inference. In Uncertainty in artificial intelligence, vol. 17, pp. 362-369. Morgan Kaufmann.
77. Winn J, Bishop CM. 2005 Variational message passing. J. Mach. Learn. Res. 6, 661-694.
78. Minka, TP. 2005 Divergence measures and message passing. Microsoft Research Technical Report MSR-TR-2005-173.
79. Koller D, Friedman N. 2009 Probabilistic graphical models: principles and techniques. Cambridge, MA: MIT Press.
80. Diaconis P. 1988 Bayesian numerical analysis. In Statistical decision theory and related topics IV, vol. 1, pp. 163-175. Berlin, Germany: Springer.
81. O'Hagan A. 1991 Bayes-Hermite quadrature. J. Stat. Plan. Inference 29, 245-260. (doi:10.1016/ 0378-3758(91)90002-V)
82. Rasmussen CE, Ghahramani Z. 2003 Bayesian Monte Carlo. In Advances in Neural Information Processing Systems, vol. 15, pp. 489-496. Cambridge, MA: MIT Press.
83. Osborne MA, Garnett R, Roberts S, Hart C, Aigrain S, Gibson N. 2012 Bayesian quadrature for ratios. In Proc. 15th Int. Conf. Artificial Intelligence and Statistics (AISTATS 2012), La Palma, Canary Islands, 21-23 April 2012 (eds N Lawrence, M Girolami), pp. 832-840.
84. Lunn DJ, Thomas A, Best N, Spiegelhalter D. 2000 WinBUGS: a Bayesian modelling framework: concepts, structure, and extensibility. Stat. Comput. 10, 325-337. (doi:10.1023/A: 1008929526011)
85. Goodman N, Mansinghka V, Roy D, Bonawitz K, Tenenbaum J. 2008 Church: a language for generative models. In Uncertainty in artificial intelligence, Helsinki, Finland, 9-12 July 2008, vol. 22, p. 23.
86. Minka TP, Winn JM, Guiver JP, Knowles DA. 2010 Infer.NET 2.4. Microsoft Research Cambridge. See http://research.microsoft.com/infernet.