Piecewise training for structured prediction

Machine Learning

Volumn 77, Issue 2-3, 2009, Pages 165-194

  Sutton, Charles ^a,b   McCallum, Andrew ^a  

^a The Manning College of Information and Computer Sciences   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Belief propagation; Conditional random fields; Graphical models; Local training

Indexed keywords

BELIEF PROPAGATION; CARDINALITIES; COMPUTATIONALLY EFFICIENT; CONDITIONAL RANDOM FIELD; GRAPHICAL MODEL; LOCAL TRAINING; NATURAL LANGUAGE PROCESSING; PIECE-WISE; PSEUDO-LIKELIHOOD; REAL-WORLD; STRUCTURED PREDICTION; SUBGRAPHS; TRAINING ALGORITHMS;

COMPUTATIONAL LINGUISTICS; CONTENT BASED RETRIEVAL; GRAPHIC METHODS; NATURAL LANGUAGE PROCESSING SYSTEMS; PARAMETER ESTIMATION;

MAXIMUM LIKELIHOOD ESTIMATION;

EID: 72449149168     PISSN: 08856125     EISSN: 15730565     Source Type: Journal    
DOI: 10.1007/s10994-009-5112-z     Document Type: Article

References (51)

1. Abbeel, P., Koller, D., & Ng, A. Y. (2005). Learning factor graphs in polynomial time and sample complexity. In Twenty-first conference on uncertainty in artificial intelligence (UAI05).
2. Bernal, A., Crammer, K., Hatzigeorgiou, A., & Pereira, F. (2007). Global discriminative learning for higher-accuracy computational gene prediction. PLoS Computational Biology, 3(3).
3. J. Besag 1975 Statistical analysis of non-lattice data The Statistician 24 3 179 195 10.2307/2987782 391451
4. J. Besag 1977 Efficiency of pseudolikelihood estimation for simple Gaussian fields Biometrika 64 3 616 618 0372.62067 10.1093/biomet/64.3.616 494640
5. Choi, A., Chavira, M., & Darwiche, A. (2007). Node splitting: a scheme for generating upper bounds in Bayesian networks. In Conference on uncertainty in artificial intelligence (UAI).
6. D. R. Cox N. Reid 2004 A note on pseudolikelihood constructed from marginal densities Biometrika 91 729 737 1162.62365 10.1093/biomet/91.3.729 2090633 (Pubitemid 41309904)
7. K. Crammer Y. Singer 2003 Ultraconservative online algorithms for multiclass problems Journal of Machine Learning Research 3 951 991 1112.68497 10.1162/jmlr.2003.3.4-5.951 1983939
8. Daumé III, H., & Marcu, D. (2005). Learning as search optimization: Approximate large margin methods for structured prediction. In International Conference on Machine Learning (ICML), Bonn, Germany.
9. Finkel, J. R., Manning, C. D., & Ng, A. Y. (2006). Solving the problem of cascading errors: approximate Bayesian inference for linguistic annotation pipelines. In Conference on empirical methods in natural language processing (EMNLP).
10. W. T. Freeman E. C. Pasztor O. T. Carmichael 2000 Learning low-level vision International Journal of Computer Vision 40 1 24 57 10.1023/A: 1026501619075 (Pubitemid 32083682)
11. Freitag, D. (1998). Machine learning for information extraction in informal domains. PhD thesis, Carnegie Mellon University.
12. Ganapathi, V., Vickrey, D., Duchi, J., & Koller, D. (2008). Constrained approximate maximum entropy learning. In Conference on uncertainty in artificial intelligence (UAI).
13. Gidas, B. (1988). Consistency of maximum likelihood and pseudolikelihood estimators for Gibbs distributions. In W. Fleming & P.-L. Lions (Eds.), Stochastic differential systems, stochastic control theory and applications. New York: Springer.
14. Greiner, R., Guo, Y., & Schuurmans, D. (2005). Learning coordination classifiers. In International joint conference on artificial intelligence (IJCAI).
15. F. Huang Y. Ogata 1999 Improvements of the maximum pseudo-likelihood estimators in various spatial statistical models Journal of Computational and Graphical Statistics 8 3 510 530 10.2307/1390872 ISSN 10618600. URL http://www.jstor.org/stable/1390872
16. A. Hyvarinen 2006 Consistency of pseudolikelihood estimation of fully visible Boltzmann machines Neural Computation 18 10 2283 2292 10.1162/neco.2006.18.10.2283 2256106 (Pubitemid 44511765)
17. Kakade, S., Teh, Y. W., & Roweis, S. (2002). An alternative objective function for Markovian fields. In Proceedings of the nineteenth international conference on machine learning.
18. Lafferty, J., McCallum, A., & Pereira, F. (2001). Conditional random fields: probabilistic models for segmenting and labeling sequence data. In International conference on machine learning (ICML).
19. Li, S. Z. (2001). Markov random field modeling in image analysis. New York: Springer.
20. Liang, P., Taskar, B., & Klein, D. (2006). Alignment by agreement. In Human language technology and North American association for computational linguistics (HLT/NAACL).
21. Liang, P., Klein, D., & Jordan, M. I. (2008). Agreement-based learning. In Advances in neural information processing systems (NIPS).
22. Lindsay, B. G. (1988). Composite likelihood methods. Contemporary Mathematics, 221-239.
23. McCallum, A., & Wellner, B. (2005). Conditional models of identity uncertainty with application to noun coreference. In L. K. Saul, Y. Weiss, & L. Bottou (Eds.), Advances in neural information processing systems 17 (pp. 905-912). Cambridge: MIT Press.
24. McCallum, A., Freitag, D., & Pereira, F. (2000). Maximum entropy Markov models for information extraction and segmentation. In International conference on machine learning (ICML) (pp. 591-598). San Francisco: Morgan Kaufmann.
25. McDonald, R., Crammer, K., & Pereira, F. (2005). Spanning tree methods for discriminative training of dependency parsers (Technical Report MS-CIS-05-11). University of Pennsylvania CIS.
26. Minka, T. (2001a). Expectation propagation for approximate Bayesian inference. In 17th conference on uncertainty in artificial intelligence (UAI) (pp. 362-369).
27. Minka, T. P. (2001b). The EP energy function and minimization schemes. http://research.microsoft.com/~minka/papers/ep/minka-ep-energy.pdf.
28. Minka, T. (2005). Divergence measures and message passing (Technical Report MSR-TR-2005-173). Microsoft Research.
29. Parise, S., & Welling, M. (2005). Learning in Markov random fields: an empirical study. In Joint Statistical Meeting (JSM2005).
30. Punyakanok, V., Roth, D., Yih, W.-T., & Zimak, D. (2005). Learning and inference over constrained output. In Proceedings of the international joint conference on artificial intelligence (IJCAI) (pp. 1124-1129).
31. Ratnaparkhi, A. (1996). A maximum entropy model for part-of-speech tagging. In Conference on empirical methods in natural language proceeding (EMNLP).
32. Rosen-Zvi, M., Yuille, A. L., & Jordan, M. I. (2005). The dlr hierarchy of approximate inference. In Conference on uncertainty in artificial intelligence (UAI).
33. Shotton, J., Winn, J., Rother, C., & Criminisi, A. (2006). Textonboost: joint appearance, shape and context modeling for multi-class object recognition and segmentation. In European conference on computer vision (ECCV).
34. Stern, D. H., Graepel, T., & MacKay, D. J. C. (2005). Modelling uncertainty in the game of go. In L. K. Saul, Y. Weiss, & L. Bottou (Eds.), Advances in neural information processing systems 17 (pp. 1353-1360). Cambridge: MIT Press.
35. Sutton, C. (2008). Efficient training methods for conditional random fields. PhD thesis. University of Massachusetts.
36. Sutton, C., & McCallum, A. (2004). Collective segmentation and labeling of distant entities in information extraction. In ICML workshop on statistical relational learning and its connections to other fields.
37. Sutton, C., & McCallum, A. (2005). Piecewise training of undirected models. In Conference on uncertainty in artificial intelligence (UAI).
38. Sutton, C., & McCallum, A. (2007a). An introduction to conditional random fields for relational learning. In L. Getoor & B. Taskar (Eds.), Introduction to statistical relational learning. Cambridge: MIT Press.
39. Sutton, C., & McCallum, A. (2007b). Piecewise pseudolikelihood for efficient CRF training. In International conference on machine learning (ICML).
40. Sutton, C., & Minka, T. (2006). Local training and belief propagation (Technical Report TR-2006-121). Microsoft Research.
41. Sutton, C., Rohanimanesh, K., & McCallum, A. (2004). Dynamic conditional random fields: Factorized probabilistic models for labeling and segmenting sequence data. In International conference on machine learning (ICML).
42. C. Sutton A. McCallum K. Rohanimanesh 2007 Dynamic conditional random fields: factorized probabilistic models for labeling and segmenting sequence data Journal of Machine Learning Research 8 693 723
43. Taskar, B., Guestrin, C., & Koller, D. (2004a). Max-margin Markov networks. In S. Thrun, L. Saul, & B. Schölkopf (Eds.), Advances in neural information processing systems 16. Cambridge: MIT Press.
44. Taskar, B., Klein, D., Collins, M., Koller, D., & Manning, C. (2004b). Max-margin parsing. In Empirical methods in natural language processing (EMNLP04).
45. Toutanova, K., Klein, D., Manning, C. D., & Singer, Y. (2003). Feature-rich part-of-speech tagging with a cyclic dependency network. In HLT-NAACL.
46. Vishwanathan, S. V. N., Schraudolph, N. N., Schmidt, M. W., & Murphy, K. (2006). Accelerated training of conditional random fields with stochastic meta-descent. In International conference on machine learning (ICML) (pp. 969-976).
47. Wainwright, M. J., Jaakkola, T., & Willsky, A. S. (2002). A new class of upper bounds on the log partition function. In Uncertainty in artificial intelligence.
48. M. J. Wainwright T. Jaakkola A. S. Willsky 2003 Tree-based reparameterization framework for analysis of sum-product and related algorithms IEEE Transactions on Information Theory 45 9 1120 1146 10.1109/TIT.2003.810642 1984817
49. Wainwright, M. J., Jaakkola, T., & Willsky, A. S. (2003b). Tree-reweighted belief propagation and approximate ML estimation by pseudo-moment matching. In Ninth workshop on artificial intelligence and statistics.
50. Wellner, B., McCallum, A., Peng, F., & Hay, M. (2004). An integrated, conditional model of information extraction and coreference with application to citation graph construction. In 20th conference on uncertainty in artificial intelligence (UAI).
51. J. S. Yedidia W. T. Freeman Y. Weiss 2005 Constructing free-energy approximations and generalized belief propagation algorithms IEEE Transactions on Information Theory 51 7 2282 2312 10.1109/TIT.2005.850085 2246363 (Pubitemid 41136394)