A tutorial on dual decomposition and lagrangian relaxation for inference in natural language processing

Journal of Artificial Intelligence Research

Volumn 45, Issue , 2012, Pages 305-362

  Rush, Alexander M ^a   Collins, Michael ^b  

^a MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^b Columbia University ^*  (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CLASSICAL METHODS; COMBINATORIAL ALGORITHM; DUAL DECOMPOSITION; GRAPHICAL MODEL; INFERENCE PROBLEM; LAGRANGIAN RELAXATION; NATURAL LANGUAGE PROCESSING; PRACTICAL ISSUES;

COMBINATORIAL OPTIMIZATION; LAGRANGE MULTIPLIERS; LEARNING ALGORITHMS; NATURAL LANGUAGE PROCESSING SYSTEMS;

INFERENCE ENGINES;

EID: 84875138356     PISSN: None     EISSN: 10769757     Source Type: Journal    
DOI: 10.1613/jair.3680     Document Type: Article

References (56)

1. Auli, M., & Lopez, A. (2011). A comparison of loopy belief propagation and dual decomposition for integrated ccg supertagging and parsing. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 470-480, Portland, Oregon, USA. Association for Computational Linguistics.
2. Bar-Hillel, Y., Perles, M., & Shamir, E. (1964). On formal properties of simple phrase structure grammars. In Language and Information: Selected Essays on their Theory and Application, pp. 116-150.
3. Boyd, S., Parikh, N., Chu, E., Peleato, B., & Eckstein, J. (2011). Distributed optimization and statistical learning via the alternating direction method of multipliers. Now Publishers.
4. Boyd, S., & Vandenberghe, L. (2004). Convex optimization. Cambridge Univ Pr.
5. Boyd, S., & Mutapcic, A. (2007). Subgradient Methods. Course Notes for EE364b, Stanford University, Winter 2006-07.
6. Chang, Y., & Collins, M. (2011). Exact Decoding of Phrase-based Translation Models through Lagrangian Relaxation. In To appear proc. of EMNLP.
7. Collins, M. (1997). Three Generative, Lexicalised Models for Statistical Parsing. In Proc. ACL, pp. 16-23.
8. Dantzig, G., & Wolfe, P. (1960). Decomposition principle for linear programs. In Operations research, Vol. 8, pp. 101-111.
9. Das, D., Martins, A., & Smith, N. (2012). An exact dual decomposition algorithm for shallow semantic parsing with constraints. Proceedings of* SEM.[ii, 10, 50].
10. DeNero, J., & Macherey, K. (2011). Model-Based Aligner Combination Using Dual Decomposition. In Proc. ACL.
11. Duchi, J., Tarlow, D., Elidan, G., & Koller, D. (2007). Using combinatorial optimization within maxproduct belief propagation. In Advances in Neural Information Processing Systems (NIPS).
12. Everett III, H. (1963). Generalized lagrange multiplier method for solving problems of optimum allocation of resources. Operations Research, 399-417.
13. Felzenszwalb, P., & Huttenlocher, D. (2006). Efficient belief propagation for early vision. International journal of computer vision, 70(1), 41-54.
14. Fisher, M. L. (1981). The lagrangian relaxation method for solving integer programming problems. Management Science, 27(1), pp. 1-18.
15. Germann, U., Jahr, M., Knight, K., Marcu, D., & Yamada, K. (2001). Fast decoding and optimal decoding for machine translation. In Proceedings of the 39th Annual Meeting on Association for Computational Linguistics, pp. 228-235. Association for Computational Linguistics.
16. Globerson, A., & Jaakkola, T. (2007). Fixing max-product: Convergent message passing algorithms for map lp-relaxations. In NIPS.
17. Hanamoto, A., Matsuzaki, T., & Tsujii, J. (2012). Coordination structure analysis using dual decomposition. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pp. 430-438, Avignon, France. Association for Computational Linguistics.
18. Held, M., & Karp, R. M. (1971). The traveling-salesman problem and minimum spanning trees: Part ii. Mathematical Programming, 1, 6-25. 10.1007/BF01584070.
19. Johnson, J., Malioutov, D., & Willsky, A. (2007). Lagrangian relaxation for map estimation in graphical models. In 45th Annual Allerton Conference on Communication, Control and Computing.
20. Jojic, V., Gould, S., & Koller, D. (2010). Fast and smooth: Accelerated dual decomposition for MAP inference. In Proceedings of International Conference on Machine Learning (ICML).
21. Klein, D., & Manning, C. (2002). Fast exact inference with a factored model for natural language parsing. Advances in neural information processing systems, 15(2002).
22. Koehn, P., Och, F. J., & Marcu, D. (2003). Statistical phrase-based translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistic s on Human Language Technology, NAACL '03, pp. 48-54.
23. Kolmogorov, V. (2006). Convergent tree-reweighted message passing for energy minimization. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 28(10), 1568-1583.
24. Komodakis, N., Paragios, N., & Tziritas, G. (2007). MRF Optimization via Dual Decomposition: Message-Passing Revisited. In Proc. ICCV.
25. Komodakis, N., Paragios, N., & Tziritas, G. (2011). Mrf energy minimization and beyond via dual decomposition. Pattern Analysis and Machine Intelligence, IEEE Transactions on, pp. 1-1.
26. Koo, T., Carreras, X., & Collins, M. (2008). Simple semi-supervised dependency parsing. In Proc. ACL/HLT.
27. Koo, T., Rush, A. M., Collins, M., Jaakkola, T., & Sontag, D. (2010). Dual decomposition for parsing with non-projective head automata. In EMNLP.
28. Korte, B., & Vygen, J. (2008). Combinatorial Optimization: Theory and Algorithms. Springer Verlag.
29. Lacoste-Julien, S., Taskar, B., Klein, D., & Jordan, M. (2006). Word alignment via quadratic assignment. In Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, pp. 112-119. Association for Computational Linguistics.
30. Lafferty, J., McCallum, A., & Pereira, F. (2001). Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. In Proc. ICML, pp. 282-289.
31. Lemaréchal, C. (2001). Lagrangian Relaxation. In Computational Combinatorial Optimization, Optimal or Provably Near-Optimal Solutions [based on a Spring School], pp. 112-156, London, UK. Springer-Verlag.
32. Magnanti, T. L., & Wolsey, L. A. (1994). Optimal trees. Tech. rep. 290-94, Massachusetts Institute of Technology, Operations Research Center.
33. Martin, R., Rardin, R., & Campbell, B. (1990). Polyhedral characterization of discrete dynamic programming. Operations research, 38(1), 127-138.
34. Martins, A. (2012). The Geometry of Constrained Structured Prediction: Applications to Inference and Learning of Natural Language Syntax. Ph.D. thesis.
35. Martins, A., Figueiredo, M., Aguiar, P., Smith, N., & Xing, E. (2011). An augmented lagrangian approach to constrained map inference. In International Conference on Machine Learning.
36. Martins, A., Smith, N., & Xing, E. (2009). Concise Integer Linear Programming Formulations for Dependency Parsing. In Proc. ACL, pp. 342-350.
37. Martins, A., Smith, N., Figueiredo, M., & Aguiar, P. (2011). Dual decomposition with many overlapping components. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pp. 238-249, Edinburgh, Scotland, UK. Association for Computational Linguistics.
38. McDonald, R. (2006). Discriminative Training and Spanning Tree Algorithms for Dependency Parsing. Ph.D. thesis, University of Pennsylvania, Philadelphia, PA, USA.
39. Meshi, O., & Globerson, A. (2011). An alternating direction method for dual map lp relaxation. Machine Learning and Knowledge Discovery in Databases, 470-483.
40. Nedić, A., & Ozdaglar, A. (2009). Approximate primal solutions and rate analysis for dual subgradient methods. SIAM Journal on Optimization, 19(4), 1757-1780.
41. Nesterov, Y. (2005). Smooth minimization of non-smooth functions. Mathematical Programming, 103(1), 127-152.
42. Paul, M. J., & Eisner, J. (2012). Implicitly intersecting weighted automata using dual decomposition. In Proc. NAACL.
43. Pearl, J. (1988). Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference (2nd edition). Morgan Kaufmann Publishers.
44. Riedel, S., & Clarke, J. (2006). Incremental integer linear programming for non-projective dependency parsing. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pp. 129-137. Association for Computational Linguistics.
45. Riedel, S., & McCallum, A. (2011). Fast and robust joint models for biomedical event extraction. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pp. 1-12, Edinburgh, Scotland, UK. Association for Computational Linguistics.
46. Roth, D., & Yih, W. (2005). Integer linear programming inference for conditional random fields. In Proceedings of the 22nd international conference on Machine learning, pp. 736-743. ACM.
47. Rush, A., Reichart, R., Collins, M., & Globerson, A. (2012). Improved parsing and pos tagging using inter-sentence consistency constraints. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 1434-1444, Jeju Island, Korea. Association for Computational Linguistics.
48. Rush, A., & Collins, M. (2011). Exact Decoding of Syntactic Translation Models through Lagrangian Relaxation. In Proc. ACL.
49. Rush, A., Sontag, D., Collins, M., & Jaakkola, T. (2010). On Dual Decomposition and Linear Programming Relaxations for Natural Language Processing. In Proc. EMNLP.
50. Shor, N. Z. (1985). Minimization Methods for Non-differentiable Functions. Springer Series in Computational Mathematics. Springer.
51. Smith, D., & Eisner, J. (2008). Dependency parsing by belief propagation. In Proc. EMNLP, pp. 145-156.
52. Sontag, D., Globerson, A., & Jaakkola, T. (2010). Introduction to dual decomposition for inference. In Sra, S., Nowozin, S., & Wright, S. J. (Eds.), Optimization for Machine Learning. MIT Press.
53. Sontag, D., Meltzer, T., Globerson, A., Jaakkola, T., & Weiss, Y. (2008). Tightening LP relaxations for MAP using message passing. In Proc. UAI.
54. Toutanova, K., Klein, D., Manning, C. D., & Singer, Y. (2003). Feature-rich part-of-speech tagging with a cyclic dependency network. In HLT-NAACL.
55. Wainwright, M., Jaakkola, T., & Willsky, A. (2005). MAP estimation via agreement on trees: message-passing and linear programming.. In IEEE Transactions on Information Theory, Vol. 51, pp. 3697-3717.
56. Yanover, C., Meltzer, T., & Weiss, Y. (2006). Linear Programming Relaxations and Belief Propagation-An Empirical Study. In The Journal of Machine Learning Research, Vol. 7, p. 1907. MIT Press.