Stochastic gradient descent training for L1-regularized log-linear models with cumulative penalty

ACL-IJCNLP 2009 - Joint Conf. of the 47th Annual Meeting of the Association for Computational Linguistics and 4th Int. Joint Conf. on Natural Language Processing of the AFNLP, Proceedings of the Conf.

Volumn , Issue , 2009, Pages 477-485

  Tsuruoka, Yoshimasa ^a   Tsujii, Jun'ichi ^a,b   Ananiadou, Sophia ^a  

^a UNIVERSITY OF MANCHESTER   (United Kingdom)

^b UNIVERSITY OF TOKYO   (Japan)

Author keywords

[No Author keywords available]

Indexed keywords

CHARACTER RECOGNITION; COMPUTATIONAL LINGUISTICS; GRADIENT METHODS; NEWTON-RAPHSON METHOD; REGRESSION ANALYSIS; SPEECH RECOGNITION; STOCHASTIC MODELS; STOCHASTIC SYSTEMS;

APPROXIMATE GRADIENT; BATCH TRAINING ALGORITHMS; LEARNING FRAMEWORKS; NAMED ENTITY RECOGNITION; NATURAL LANGUAGE PROCESSING; PART OF SPEECH TAGGING; QUASI-NEWTON METHODS; STOCHASTIC GRADIENT DESCENT;

NATURAL LANGUAGE PROCESSING SYSTEMS;

EID: 80052416457     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.3115/1687878.1687946     Document Type: Conference Paper

References (28)

1. Galen Andrew and Jianfeng Gao. 2007. Scalable training of L1-regularized log-linear models. In Proceedings of ICML, pages 33-40.
2. Bob Carpenter. 2008. Lazy sparse stochastic gradient descent for regularized multinomial logistic regression. Technical report, Alias-i.
3. Stephen Clark and James R. Curran. 2004. Parsing the WSJ using CCG and log-linear models. In Proceedings of COLING 2004, pages 103-110.
4. Trevor Cohn and Philip Blunsom. 2005. Semantic role labeling with tree conditional random fields. In Proceedings of CoNLL, pages 169-172.
5. Michael Collins, Amir Globerson, Terry Koo, Xavier Carreras, and Peter L. Bartlett. 2008. Exponentiated gradient algorithms for conditional random fields and max-margin markov networks. The Journal of Machine Learning Research (JMLR), 9:1775-1822.
6. Michael Collins. 2002. Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms. In Proceedings of EMNLP, pages 1-8.
7. Christian Darken and John Moody. 1990. Note on learning rate schedules for stochastic optimization. In Proceedings of NIPS, pages 832-838.
8. Juhn Duchi and Yoram Singer. 2008. Online and batch learning using forward-looking subgradients. In NIPS Workshop: OPT 2008 Optimization for Machine Learning.
9. Juhn Duchi, Shai Shalev-Shwartz, Yoram Singer, and Tushar Chandra. 2008. Efficient projections onto the l1-ball for learning in high dimensions. In Proceedings of ICML, pages 272-279.
10. Jenny Rose Finkel, Alex Kleeman, and Christopher D. Manning. 2008. Efficient, feature-based, conditional random field parsing. In Proceedings of ACL-08:HLT, pages 959-967.
11. Jianfeng Gao, Galen Andrew, Mark Johnson, and Kristina Toutanova. 2007. A comparative study of parameter estimation methods for statistical natural language processing. In Proceedings of ACL, pages 824-831.
12. Han-Shen Huang, Yu-Ming Chang, and Chun-Nan Hsu. 2007. Training conditional random fields by periodic step size adaptation for large-scale textmining. In Proceedings of ICDM, pages 511-516.
13. Jun'ichi Kazama and Jun'ichi Tsujii. 2003. Evaluation and extension of maximum entropy models with inequality constraints. In Proceedings of EMNLP 2003.
14. J.-D. Kim, T. Ohta, Y. Tsuruoka, Y. Tateisi, and N. Collier. 2004. Introduction to the bio-entity recognition task at JNLPBA. In Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications (JNLPBA), pages 70-75.
15. John Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of ICML, pages 282-289.
16. John Langford, Lihong Li, and Tong Zhang. 2009. Sparse online learning via truncated gradient. The Journal of Machine Learning Research (JMLR), 10:777-801.
17. Su-In Lee, Honglak Lee, Pieter Abbeel, and Andrew Y. Ng. 2006. Efficient l1 regularized logistic regression. In Proceedings of AAAI-06, pages 401-408.
18. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1994. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313-330.
19. Jorge Nocedal. 1980. Updating quasi-newton matrices with limited storage. Mathematics of Computation, 35(151):773-782.
20. Daisuke Okanohara, Yusuke Miyao, Yoshimasa Tsuruoka, and Jun'ichi Tsujii. 2006. Improving the scalability of semi-markov conditional random fields for named entity recognition. In Proceedings of COLING/ACL, pages 465-472.
21. Adwait Ratnaparkhi. 1996. A maximum entropy model for part-of-speech tagging. In Proceedings of EMNLP 1996, pages 133-142.
22. Libin Shen, Giorgio Satta, and Aravind Joshi. 2007. Guided learning for bidirectional sequence classification. In Proceedings of ACL, pages 760-767.
23. David Smith and Jason Eisner. 2008. Dependency parsing by belief propagation. In Proceedings of EMNLP, pages 145-156.
24. James C. Spall. 2005. Introduction to Stochastic Search and Optimization. Wiley-IEEE.
25. Christoph Tillmann and Tong Zhang. 2006. A discriminative global training algorithm for statistical MT. In Proceedings of COLING/ACL, pages 721-728.
26. Kristina Toutanova, Aria Haghighi, and Christopher Manning. 2005. Joint learning improves semantic role labeling. In Proceedings of ACL, pages 589-596.
27. S. V. N. Vishwanathan, Nicol N. Schraudolph,MarkW. Schmidt, and Kevin P. Murphy. 2006. Accelerated training of conditional random fields with stochastic gradient methods. In Proceedings of ICML, pages 969-976.
28. Ben Wellner and Marc Vilain. 2006. Leveraging machine readable dictionaries in discriminative sequence models. In Proceedings of LREC 2006.