Bayesian inference with posterior regularization and applications to infinite latent SVMs

Journal of Machine Learning Research

Volumn 15, Issue , 2014, Pages 1799-1847

  Zhu, Jun ^a   Chen, Ning ^a   Xing, Eric P ^b  

^a TSINGHUA UNIVERSITY   (China)

^b Carnegie Mellon University   (United States)

Author keywords

Bayesian inference; Bayesian nonparametrics; Classification; Large margin learning; Multi task learning; Posterior regularization

Indexed keywords

CLASSIFICATION (OF INFORMATION); INFERENCE ENGINES; MATHEMATICAL OPERATORS; SUPPORT VECTOR MACHINES;

BAYESIAN INFERENCE; BAYESIAN NONPARAMETRICS; LARGE-MARGIN LEARNING; MULTITASK LEARNING; POSTERIOR REGULARIZATION;

BAYESIAN NETWORKS;

EID: 84902818267     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (74)

1. Yasemin Altun and Alex Smola. Unifying divergence minimization and statistical inference via convex duality. In International Conference on Learning Theory, pages 139-153, 2006. (Pubitemid 44072191)
2. Rie Kubota Ando and Tong Zhang. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, (6):1817-1853, 2005.
3. Charles E. Antoniak. Mixture of Dirichlet process with applications to Bayesian nonpara-metric problems. Annals of Statistics, (273):1152-1174, 1974.
4. Andreas Argyriou, Theodoros Evgeniou, and Massimiliano Pontil. Multi-task feature learning. In Advances in Neural Information Processing Systems, pages 41-48, 2007.
5. Bart Bakker and Tom Heskes. Task clustering and gating for Bayesian multitask learning. Journal of Machine Learning Research, (4):83-99, 2003.
6. Andrew Barron, Mark Schervish, and Larry Wasserman. The consistency of posterior distributions in nonparametric problems. The Annals of Statistics, 27(2):536-561, 1999.
7. Matthew J. Beal, Zoubin Ghahramani, and Carl E. Rasmussen. The infinite hidden Markov model. In Advances in Neural Information Processing Systems, pages 577-584, 2002.
8. Kedar Bellare, Gregory Druck, and Andrew McCallum. Alternating projections for learning with expectation constraints. In Uncertainty in Artificial Intelligence, 2009.
9. David Blei and Peter Frazier. Distance dependent Chinese restaurant process. In International Conference on Machine Learning, pages 87-94, 2010.
10. David Blei, Andrew Ng, and Michael I. Jordan. Latent Dirichlet allocation. Journal of Machine Learning Research, (3):993-1022, 2003.
11. Edwin Bonilla, Kian Ming Chai, and Christopher Williams. Multi-task Gaussian process prediction. In Advances in Neural Information Processing Systems, pages 153-160, 2008.
12. Jonathan Borwein and Qiji Zhu. Techniques of Variational Analysis: An Introduction. Springer, New York, NY, 2005.
13. Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
14. Ming-Wei Chang, Lev Ratinov, and Dan Roth. Guiding semi-supervision with constraint-driven learning. In the 45th Annual Meeting of the Association of Computational Linguistics, pages 280-287, 2007.
15. Ning Chen, Jun Zhu, Fuchun Sun, and Eric P. Xing. Large-margin predictive latent subspace learning for multiview data analysis. IEEE Trans. on Pattern Analysis and Machine Intelligence (TPAMI), 34(12):2365-2378, 2012.
16. Ning Chen, Jun Zhu, Fuchun Sun, and Bo Zhang. Learning harmonium models with infinite latent features. IEEE Transactions on Neural Networks and Learning Systems, 25(3): 520-532, 2014.
17. Taeryon Choi and R. V. Ramamoorthi. Remarks on consistency of posterior distributions. IMS Collections 3. Pushing the limits of contemporary statistics: Contributions in honor of Jayanta K. Ghosh. eds. S. Ghosal and B. Clarke, pages 170-186, 2008.
18. Arthur P. Dempster, Nan M. Laird, and Donald B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B, 39(1):1-38, 1977.
19. Finale Doshi-Velez. The Indian buffet process: Scalable inference and extensions. Master's thesis, The University of Cambridge, Aug 2009.
20. Mirosla Dudík, Steven J. Phillips, and Robert E. Schapire. Maximum entropy density estimation with generalized regularization and an application to species distribution modeling. Journal of Machine Learning Research, (8):1217-1260, 2007. (Pubitemid 46945617)
21. David Dunson and Shyamal Peddada. Bayesian nonparametric inferences on stochastic ordering. ISDS Discussion Paper, 2, 2007.
22. Thomas Ferguson. A Bayesian analysis of some nonparametric problems. Annals of Statistics, 1(2):209-230, 1973.
23. Morten Frydenberg. The chain graph Markov property. Scandinavian Journal of Statistics, 17:333-353, 1990.
24. Kuzman Ganchev, João. Graca, Jennifer Gillenwater, and Ben Taskar. Posterior regular-ization for structured latent variable models. Journal of Machine Learning Research, (11):2001-2094, 2010.
25. Paul Garthwaite, Joseph Kadane, and Anthony O'Hagan. Statistical methods for eliciting probability distributions. Journal of the American Statistical Association, 100(470):680-700, 2005. (Pubitemid 40816603)
26. Jayanta K. Ghosh and R.V. Ramamoorthi. Bayesian Nonparametrics. Springer, New York, NY, 2003.
27. João Graca, Kuzman Ganchev, and Ben Taskar. Expectation maximization and posterior constraints. In Advances in Neural Information Processing Systems, pages 569-576, 2007.
28. Thomas Griffiths and Zoubin Ghahramani. Infinite latent feature models and the Indian buffet process. Technical report, University College London, GCNU TR2005-001, 2005.
29. Thomas L. Griffiths and Mark Steyvers. Finding scientific topics. Proceedings of the National Academy of Sciences, 101:5228-5235, 2004. (Pubitemid 38469131)
30. Peter D. Hoff. Bayesian methods for partial stochastic orderings. Biometrika, 90:303-317, 2003. (Pubitemid 39047159)
31. Seungil Huh and Stephen Fienberg. Discriminative topic modeling based on manifold learning. ACM Transactions on Knowledge Discovery from Data, 5(4), 2012.
32. Kazufumi Ito and Karl Kunisch. Lagrange Multiplier Approach to Variational Problems and Applications. Advances in Design and Control, Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 2008.
33. Tommi Jaakkola, Meila Meila, and Tony Jebara. Maximum entropy discrimination. In Advances in Neural Information Processing Systems, pages 470-476, 1999.
34. Tony Jebara. Discriminative, Generative and Imitative Learning. PhD thesis, Media Laboratory, MIT, Dec 2001.
35. Tony Jebara. Multitask sparsity via maximum entropy discrimination. Journal of Machine Learning Research, (12):75-110, 2011.
36. Qixia Jiang, Jun Zhu, Maosong Sun, and Eric P. Xing. Monte Carlo methods for maximum margin supervised topic models. In Advances in Neural Information Processing Systems, pages 1592-1600, 2012.
37. Thorsten Joachims. Transductive inference for text classification using support vector machines. In International Conference on Machine Learning, 1999.
38. Michael I. Jordan, Zoubin Ghahramani, Tommis Jaakkola, and Lawrence K. Saul. An introduction to variational methods for graphical models. Machine Learning, 37:183-233, 1999. (Pubitemid 30544678)
39. Mohammad E. Khan, Guillaume Bouchard, Benjamin Marlin, and Kevin Murphy. Variational bounds for mixed-data factor analysis. In Advances in Neural Information Processing Systems, pages 1108-1116, 2010.
40. John Lafferty, Andrew McCallum, and Fernando Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In International Conference on Machine Learning, pages 282-289, 2001.
41. Percy Liang, Michael I. Jordan, and Dan Klein. Learning from measurements in exponential families. In International Conference on Machine Learning, pages 641-648, 2009.
42. Steven N. MacEachern. Dependent nonparametric process. In the Section on Bayesian Statistical Science of ASA, 1999.
43. Thomas L. Magnanti. Fenchel and Lagrange duality are equivalent. Mathematical Programming, (7):253-258, 1974.
44. Gideon Mann and Andrew McCallum. Generalized expectation criteria for semi-supervised learning with weakly labeled data. Journal of Machine Learning Research, (11):955-984, 2010.
45. Shike Mei, Jun Zhu, and Xiaojin Zhu. Robust Reg Bayes: Selectively incorporating firstorder logic domain knowledge into bayesian models. In International Conference on Machine Learning, pages 253-261, 2014.
46. Kurt Miller, Thomas Griffiths, and Michael I. Jordan. Nonparametric latent feature models for link prediction. In Advances in Neural Information Processing Systems, pages 1276-1284, 2009.
47. Iain Murray and Zoubin Ghahramani. Bayesian learning in undirected graphical models: Approximate MCMC algorithms. In Uncertainty in Artificial Intelligence, 2004.
48. Peter Orbanz. Nonparametric priors on complete separable metric spaces. Working paper, 2012.
49. Yuan (Alan) Qi, Martin Szummer, and Thomas P. Minka. Bayesian conditional random fields. In International Conference on Artificial Intelligence and Statistics, pages 269-276, 2005.
50. Piyush Rai and Hal Daume III. Infinite predictor subspace models for multitask learning. In International Conference on Artificial Intelligence and Statistics, pages 613-620, 2010.
51. Charles E. Rasmussen and Zoubin Ghahramani. Infinite mixtures of Gaussian process experts. In Advances in Neural Information Processing Systems, pages 881-888, 2002.
52. Christian P. Robert. Simulation of truncated normal variables. Statistics and Computing, 5(2):121-125, 1995.
53. Edward Schofield. Fitting maximum-entropy models on large sample spaces. PhD thesis, PhD thesis, Department of Computing, Imperial College London, 2006.
54. Wolfgang Stummer and Igor Vajda. On bregman distances and divergences of probability measures. IEEE Trans. on Information Theory, 58(3):1277-1288, 2012.
55. Ben Taskar, Carlos Guestrin, and Daphne Koller. Max-margin Markov networks. In Advances in Neural Information Processing Systems, 2003.
56. Yee Whye Teh, Dilan Görür, and Zoubin Ghahramani. Stick-breaking construction of the Indian buffet process. In International Conference on Artificial Intelligence and Statistics, pages 556-563, 2007.
57. Yee Whye Teh, Michael I. Jordan, Matthew J. Beal, and David Blei. Hierarchical Dirichlet process. Journal of the American Statistical Association, 101(476):1566-1581, 2006.
58. Romain Thibaux and Michael I. Jordan. Hierarchical Beta processes and the Indian buffet process. In International Conference on Artificial Intelligence and Statistics, pages 564-571, 2007.
59. Vladimir Vapnik. The Nature of Statistical Learning Theory. Springer, New York, 1995.
60. Martin Wainright and Michael I. Jordan. Graphical models, exponential family, and variational methods. Foundations and Trends in Machine Learning, 1(1):1-305, 2008.
61. Max Welling and Sridevi Parise. Bayesian random fields: The Bethe-Laplace approximation. In Uncertainty in Artificial Intelligence, 2006.
62. Max Welling, Ian Porteous, and Kenichi Kurihara. Exchangeable inconsistent priors for Bayesian posterior inference. In Workshop on Information Theory and Applications, 2012.
63. Max Welling, M. Rosen-Zvi, and G. Hinton. Exponential family harmoniums with an application to information retrieval. In Advances in Neural Information Processing Systems, pages 1481-1488, 2004.
64. Peter M. Williams. Bayesian conditionalisation and the principle of minimum information. The British Journal for the Philosophy of Science, 31(2), 1980.
65. Sinead Williamson, Peter Orbanz, and Zoubin Ghahramani. Dependent Indian buffet processes. In International Conference on Artifficial Intelligence and Statistics, pages 924-931, 2010.
66. Minjie Xu, Jun Zhu, and Bo Zhang. Bayesian nonparametric max-margin matrix factorization for collaborative prediction. In Advances in Neural Information Processing Systems, pages 64-72, 2012.
67. Ya Xue, David Dunson, and Lawrence Carin. The matrix stick-breaking process for flexible multi-task learning. In International Conference on Machine Learning, pages 1063-1070, 2007.
68. Yu Zhang and Dit-Yan Yeung. A convex formulation for learning task relationships in multi-task learning. In Uncertainty in Artificial Intelligence, 2010.
69. Jun Zhu. Max-margin nonparametric latent feature relational models for link prediction. In International Conference on Machine Learning, pages 719-726, 2012.
70. Jun Zhu, Amir Ahmed, and Eric P. Xing. MedLDA: Maximum margin supervised topic models for regression and classification. In International Conference on Machine Learning, pages 1257-1264, 2009.
71. Jun Zhu, Ning Chen, Hugh Perkins, and Bo Zhang. Gibbs max-margin topic models with fast inference algorithms. In International Conference on Machine Learning, pages 124-132, 2013.
72. Jun Zhu, Ning Chen, and Eric P. Xing. Infinite latent SVM for classification and multitask learning. In Advances in Neural Information Processing Systems, pages 1620-1628, 2011a.
73. Jun Zhu, Ning Chen, and Eric P. Xing. Infinite SVM: a Dirichlet process mixture of large-margin kernel machines. In International Conference on Machine Learning, pages 617-624, 2011b.
74. Jun Zhu and Eric P. Xing. Maximum entropy discrimination Markov networks. Journal of Machine Learning Research, (10):2531-2569, 2009.