AD3: Alternating directions dual decomposition for map inference in graphical models

Journal of Machine Learning Research

Volumn 16, Issue , 2015, Pages 495-545

  Martins, André F T ^a,b   Figueiredo, Mário A T ^c   Aguiar, Pedro M Q ^c   Smith, Noah A ^d   Xing, Eric P ^d  

^a Priberam Alameda D Afonso Henriques   (Portugal)

^b INSTITUTO DE TELECOMUNICAÇÕES   (Portugal)

^c INSTITUTO SUPERIOR TÉCNICO   (Portugal)

^d Carnegie Mellon University ^*  (United States)

Author keywords

Alternating directions method of multipliers; Dual decomposition; Graphical models; MAP inference

Indexed keywords

FORMAL LOGIC; GRAPHIC METHODS;

ALTERNATING DIRECTIONS; ALTERNATING DIRECTIONS METHOD OF MULTIPLIERS; CLOSED FORM SOLUTIONS; DUAL DECOMPOSITION; GRAPHICAL MODEL; MAP INFERENCES; MAXIMUM A POSTERIORI; MODULAR ARCHITECTURES;

INFERENCE ENGINES;

EID: 84930668497     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (83)

1. M. B. Almeida and A. F. T. Martins. Fast and robust compressive summarization with dual decomposition and multi-task learning. In Proc. of the Annual Meeting of the Association for Computational Linguistics, 2013.
2. S. Barman, X. Liu, S. Draper, and B. Recht. Decomposition methods for large scale LP decoding. In 49th Annual Allerton Conference on Communication, Control, and Computing, pages 253-260. IEEE, 2011.
3. D. Batra, S. Nowozin, and P. Kohli. Tighter relaxations for MAP-MRF inference: A local primal-dual gap based separation algorithm. In International Conference on Artificial Intelligence and Statistics, pages 146-154, 2011.
4. D. Bertsekas, W. Hager, and O. Mangasarian. Nonlinear Programming. Athena Scientific, 1999.
5. D.P. Bertsekas, A. Nedic, and A.E. Ozdaglar. Convex Analysis and Optimization. Athena Scientific, 2003.
6. S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers. Now Publishers, 2011.
7. S. P. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
8. J.P. Boyle and R.L. Dykstra. A method for finding projections onto the intersections of convex sets in Hilbert spaces. In Advances in Order Restricted Statistical Inference, pages 28-47. Springer Verlag, 1986.
9. P. Brucker. An O(n) algorithm for quadratic knapsack problems. Operations Research Letters, 3(3):163-166, 1984.
10. M. Chang, L. Ratinov, and D. Roth. Constraints as prior knowledge. In International Conference of Machine Learning: Workshop on Prior Knowledge for Text and Language Processing, July 2008.
11. Y. J. Chu and T. H. Liu. On the shortest arborescence of a directed graph. Science Sinica, 14:1396-1400, 1965.
12. D. Das. Semi-Supervised and Latent-Variable Models of Natural Language Semantics. PhD thesis, Carnegie Mellon University, 2012.
13. D. Das, A.F.T. Martins, and N.A. Smith. An exact dual decomposition algorithm for shallow semantic parsing with constraints. In Proc. of First Joint Conference on Lexical and Computational Semantics (∗SEM), 2012.
14. J. Duchi, D. Tarlow, G. Elidan, and D. Koller. Using combinatorial optimization within max-product belief propagation. Advances in Neural Information Processing Systems, 19, 2007.
15. J. Duchi, S. Shalev-Shwartz, Y. Singer, and T. Chandra. Efficient projections onto the L1-ball for learning in high dimensions. In Proc. of International Conference of Machine Learning, 2008.
16. J. Eckstein and D. Bertsekas. On the Douglas-Rachford splitting method and the proximal point algorithm for maximal monotone operators. Mathematical Programming, 55(1): 293-318, 1992.
17. J. Edmonds. Optimum branchings. Journal of Research of the National Bureau of Standards, 71B:233-240, 1967.
18. J.M. Eisner. Three new probabilistic models for dependency parsing: An exploration. In Proc. of International Conference on Computational Linguistics, pages 340-345, 1996.
19. F. Facchinei and J.S. Pang. Finite-Dimensional Variational Inequalities and Complementarity Problems, volume 1. Springer Verlag, 2003.
20. P. F. Felzenszwalb and D. P. Huttenlocher. Efficient belief propagation for early vision. International Journal of Computer Vision, 70(1):41-54, 2006.
21. C.J. Fillmore. Frame semantics and the nature of language. Annals of the New York Academy of Sciences, 280(1):20-32, 1976.
22. Q. Fu, H. Wang, and A. Banerjee. Bethe-ADMM for tree decomposition based parallel MAP inference. In Proc. of Uncertainty in Artificial Intelligence, 2013.
23. D. Gabay and B. Mercier. A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Computers and Mathematics with Applications, 2(1): 17-40, 1976.
24. A. Globerson and T. Jaakkola. Fixing max-product: Convergent message passing algorithms for MAP LP-relaxations. Neural Information Processing Systems, 20, 2008.
25. R. Glowinski and A. Marroco. Sur l'approximation, par éléments finis d'ordre un, et la résolution, par penalisation-dualité, d'une classe de problèmes de Dirichlet non linéaires. Rev. Franc. Automat. Inform. Rech. Operat., 9:41-76, 1975.
26. T. Hazan and A. Shashua. Norm-product belief propagation: Primal-dual message-passing for approximate inference. IEEE Transactions on Information Theory, 56(12):6294-6316, 2010.
27. B.S. He and X.M. Yuan. On the O(1/t) convergence rate of alternating direction method. SIAM Journal of Numerical Analysis (to appear), 2011.
28. M. Hestenes. Multiplier and gradient methods. Journal of Optimization Theory and Applications, 4:302-320, 1969.
29. J.K. Johnson, D.M. Malioutov, and A.S. Willsky. Lagrangian relaxation for MAP estimation in graphical models. In 45th Annual Allerton Conference on Communication, Control and Computing, 2007.
30. V. Jojic, S. Gould, and D. Koller. Accelerated dual decomposition for MAP inference. In International Conference of Machine Learning, 2010.
31. J. Kappes, B. Savchynskyy, and C. Schnorr. A bundle approach to efficient MAP-inference by Lagrangian relaxation. In IEEE Conference on Computer Vision and Pattern Recognition, 2012.
32. D. Koller and N. Friedman. Probabilistic Graphical Models: Principles and Techniques. The MIT Press, 2009.
33. V. Kolmogorov. Convergent tree-reweighted message passing for energy minimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28:1568-1583, 2006.
34. N. Komodakis, N. Paragios, and G. Tziritas. MRF optimization via dual decomposition: Message-passing revisited. In Proc. of International Conference on Computer Vision, 2007.
35. T. Koo, A. M. Rush, M. Collins, T. Jaakkola, and D. Sontag. Dual decomposition for parsing with non-projective head automata. In Proc. of Empirical Methods for Natural Language Processing, 2010.
36. V.A. Kovalevsky and V.K. Koval. A diffusion algorithm for decreasing energy of max-sum labeling problem. Technical report, Glushkov Institute of Cybernetics, Kiev, USSR, 1975.
37. F. R. Kschischang, B. J. Frey, and H. A. Loeliger. Factor graphs and the sum-product algorithm. IEEE Transactions on Information Theory, 47, 2001.
38. A. Kulesza and F. Pereira. Structured learning with approximate inference. Neural Information Processing Systems, 2007.
39. S. Lauritzen. Graphical Models. Clarendon Press, Oxford, 1996. ISBN 0-19-852219-3.
40. Y. Low, J. Gonzalez, A. Kyrola, D. Bickson, C. Guestrin, and J.M. Hellerstein. Graphlab: A new parallel framework for machine learning. In International Conference on Uncertainty in Artificial Intelligence, 2010.
41. M. P. Marcus, M. A. Marcinkiewicz, and B. Santorini. Building a large annotated corpus of English: the Penn treebank. Computational Linguistics, 19(2):313-330, 1993.
42. A. F. T. Martins. The Geometry of Constrained Structured Prediction: Applications to Inference and Learning of Natural Language Syntax. PhD thesis, Carnegie Mellon University and Instituto Superior Técnico, 2012.
43. A. F. T. Martins, N. A. Smith, and E. P. Xing. Polyhedral outer approximations with application to natural language parsing. In Proc. of International Conference of Machine Learning, 2009.
44. A. F. T. Martins, N. A. Smith, E. P. Xing, P. M. Q. Aguiar, and M. A. T. Figueiredo. Augmented dual decomposition for MAP inference. In Neural Information Processing Systems: Workshop in Optimization for Machine Learning, 2010.
45. A. F. T. Martins, M. A. T. Figueiredo, P. M. Q. Aguiar, N. A. Smith, and E. P. Xing. An augmented Lagrangian approach to constrained MAP inference. In Proc. of International Conference on Machine Learning, 2011a.
46. A. F. T. Martins, N. A. Smith, P. M. Q. Aguiar, and M. A. T. Figueiredo. Dual decomposition with many overlapping components. In Proc. of Empirical Methods for Natural Language Processing, 2011b.
47. A. F. T. Martins, M. B. Almeida, and N. A. Smith. Turning on the turbo: Fast thirdorder non-projective turbo parsers. In Proc. of the Annual Meeting of the Association for Computational Linguistics, 2013.
48. R. McDonald and G. Satta. On the complexity of non-projective data-driven dependency parsing. In Proc. of International Conference on Parsing Technologies, 2007.
49. R. T. McDonald, F. Pereira, K. Ribarov, and J. Hajic. Non-projective dependency parsing using spanning tree algorithms. In Proc. of Empirical Methods for Natural Language Processing, 2005.
50. O. Meshi and A. Globerson. An alternating direction method for dual MAP LP relaxation. In European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2011.
51. C. Michelot. A finite algorithm for finding the projection of a point onto the canonical simplex of Rn . Journal of Optimization Theory and Applications, 50(1):195-200, 1986.
52. Y. Nesterov. A method of solving a convex programming problem with convergence rate O(1/k2) . Soviet Math. Doklady, 27:372-376, 1983.
53. J. Nocedal and S.J. Wright. Numerical Optimization. Springer verlag, 1999.
54. S. Nowozin and C.H. Lampert. Global connectivity potentials for random field models. In IEEE Conference on Computer Vision and Pattern Recognition, pages 818-825. IEEE, 2009.
55. P. M. Pardalos and N. Kovoor. An algorithm for a singly constrained class of quadratic programs subject to upper and lower bounds. Mathematical Programming, 46(1):321-328, 1990.
56. J. Pearl. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, 1988.
57. P. Pletscher and S. Wulff. LPQP for MAP: Putting LP Solvers to Better Use. In Proc. of International Conference on Machine Learning, 2012.
58. H. Poon and P. Domingos. Unsupervised semantic parsing. In Proc. of Empirical Methods in Natural Language Processing, 2009.
59. M. Powell. A method for nonlinear constraints in minimization problems. In R. Fletcher, editor, Optimization, pages 283-298. Academic Press, 1969.
60. V. Punyakanok, D. Roth, W. Yih, and D. Zimak. Learning and inference over constrained output. In Proc. of International Joint Conference on Artificial Intelligence, 2005.
61. P. Ravikumar, A. Agarwal, and M. Wainwright. Message-passing for graph-structured linear programs: Proximal methods and rounding schemes. Journal of Machine Learning Research, 11:1043-1080, 2010.
62. M. Richardson and P. Domingos. Markov logic networks. Machine Learning, 62(1):107-136, 2006.
63. T.J. Richardson and R.L. Urbanke. Modern Coding Theory. Cambridge University Press, 2008.
64. R.T. Rockafellar. Convex Analysis. Princeton University Press, 1970.
65. D. Roth and W. Yih. A linear programming formulation for global inference in natural language tasks. In International Conference on Natural Language Learning, 2004.
66. A.M. Rush and M. Collins. A tutorial on dual decomposition and Lagrangian relaxation for inference in natural language processing. Journal of Artificial Intelligence Research, 45:305-362, 2012.
67. B. Savchynskyy, S. Schmidt, J. Kappes, and C. Schnörr. A study of Nesterov's scheme for Lagrangian decomposition and MAP labeling. In IEEE Conference on Computer Vision and Pattern Recognition, 2011.
68. M. Schlesinger. Syntactic analysis of two-dimensional visual signals in noisy conditions. Kibernetika, 4:113-130, 1976.
69. A. Schwing, T. Hazan, M. Pollefeys, and R. Urtasun. Distributed message passing for large scale graphical models. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1833-1840, 2011.
70. A. Schwing, T. Hazan, M. Pollefeys, and R. Urtasun. Globally convergent dual MAP LP relaxation solvers using Fenchel-Young margins. In Advances in Neural Information Processing Systems 25, pages 2393-2401, 2012.
71. D. Smith and J. Eisner. Dependency parsing by belief propagation. In Proc. of Empirical Methods for Natural Language Processing, 2008.
72. D. Sontag, T. Meltzer, A. Globerson, Y. Weiss, and T Jaakkola. Tightening LP relaxations for MAP using message-passing. In Proc. of Uncertainty in Artificial Intelligence, 2008.
73. D. Sontag, A. Globerson, and T. Jaakkola. Introduction to dual decomposition for inference. In Optimization for Machine Learning. MIT Press, 2011.
74. M. Sun, M. Telaprolu, H. Lee, and S. Savarese. An efficient branch-and-bound algorithm for optimal human pose estimation. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1616-1623, 2012.
75. R. Tanner. A recursive approach to low complexity codes. IEEE Transactions on Information Theory, 27(5):533-547, 1981.
76. D. Tarlow, I. E. Givoni, and R. S. Zemel. HOP-MAP: Efficient message passing with high order potentials. In AISTATS, 2010.
77. M. Wainwright and M. Jordan. Graphical Models, Exponential Families, and Variational Inference. Now Publishers, 2008.
78. M. Wainwright, T. Jaakkola, and A. Willsky. MAP estimation via agreement on trees: message-passing and linear programming. IEEE Transactions on Information Theory, 51 (11):3697-3717, 2005.
79. H. Wang and A. Banerjee. Online alternating direction method. In Proc. of International Conference on Machine Learning, 2012.
80. T. Werner. A linear programming approach to max-sum problem: A review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29:1165-1179, 2007.
81. H. Yamada and Y. Matsumoto. Statistical dependency analysis with support vector machines. In Proc. of International Conference on Parsing Technologies, 2003.
82. C. Yanover, T. Meltzer, and Y. Weiss. Linear programming relaxations and belief propagation-an empirical study. Journal of Machine Learning Research, 7:1887-1907, 2006.
83. J.S. Yedidia, Y. Wang, and S.C. Draper. Divide and concur and difference-map BP decoders for LDPC codes. IEEE Transactions on Information Theory, 57(2):786-802, 2011.