Feature-frequency-adaptive on-line training for fast and accurate natural language processing

Computational Linguistics

Volumn 40, Issue 3, 2014, Pages 563-586

  Sun, Xu ^a,b   Li, Wenjie ^c   Wang, Houfeng ^a,b   Lu, Qin ^c  

^a Peking University   (China)

^b PEKING UNIVERSITY   (China)

^c HONG KONG POLYTECHNIC UNIVERSITY   (Hong Kong)

Author keywords

[No Author keywords available]

Indexed keywords

ONLINE SYSTEMS; SENTIMENT ANALYSIS; SPEED;

CLASSIFICATION TASKS; FAST CONVERGENCE RATE; FEATURE FREQUENCIES; HIGHER FREQUENCIES; NAMED ENTITY RECOGNITION; NATURAL LANGUAGE PROCESSING; STATE OF THE ART; WORD SEGMENTATION;

E-LEARNING;

EID: 84910067653     PISSN: 08912017     EISSN: 15309312     Source Type: Journal    
DOI: 10.1162/COLI_a_00193     Document Type: Article

References (42)

1. Berger Adam L., Vincent J. Della Pietra, and Stephen A. Della Pietra. 1996. A maximum entropy approach to natural language processing. Computational Linguistics, 22(1):39–71.
2. Blitzer John, Mark Dredze, and Fernando Pereira. 2007. Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 440–447, Prague.
3. Bottou Léon. 1998. Online algorithms and stochastic approximations. In D. Saad, editor. Online Learning and Neural Networks. Cambridge University Press, pages 9–42.
4. Chen Stanley F. and Ronald Rosenfeld. 1999. A Gaussian prior for smoothing maximum entropy models. Technical Report CMU-CS-99-108, Carnegie Mellon University.
5. Collins Michael. 2002. Discriminative training methods for hidden Markov models: Theory and experiments with perceptron algorithms. In Proceedings of EMNLP’02, pages 1–8, Philadelphia, PA.
6. Crammer Koby, Alex Kulesza, and Mark Dredze. 2009. Adaptive regularization of weight vectors. In NIPS’09, pages 414–422, Vancouver.
7. Dredze Mark, Koby Crammer, and Fernando Pereira. 2008. Confidenceweighted linear classification. In Proceedings of ICML’08, pages 264–271, Helsinki.
8. Duchi John, Elad Hazan, and Yoram Singer. 2010. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2,121–2,159.
9. Finkel Jenny, Shipra Dingare, Huy Nguyen, Malvina Nissim, Christopher Manning, and Gail Sinclair. 2004. Exploiting context for biomedical entity recognition: From syntax to the Web. In Proceedings of BioNLP’04, pages 91–94, Geneva.
10. Freund Yoav and Robert Schapire. 1999. Large margin classification using the perceptron algorithm. Machine Learning, 37(3):277–296.
11. Gao Jianfeng, Galen Andrew, Mark Johnson, and Kristina Toutanova. 2007. A comparative study of parameter estimation methods for statistical natural language processing. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics (ACL’07), pages 824–831, Prague.
12. Hsu Chun-Nan, Han-Shen Huang, Yu-Ming Chang, and Yuh-Jye Lee. 2009. Periodic step-size adaptation in second-order gradient descent for single-pass on-line structured learning. Machine Learning, 77(2-3):195–224.
13. Jacobs Robert A. 1988. Increased rates of convergence through learning rate adaptation. Neural Networks, 1(4):295–307.
14. Kim Jin-Dong, Tomoko Ohta, Yoshimasa Tsuruoka, and Yuka Tateisi. 2004. Introduction to the bio-entity recognition task at JNLPBA. In Proceedings of BioNLP’04, pages 70–75, Geneva.
15. Kudo Taku and Yuji Matsumoto. 2001. Chunking with support vector machines. In Proceedings of NAACL’01, pages 1–8, Pittsburgh, PA.
16. Lafferty John, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML’01, pages 282–289, Williamstown, MA.
17. McDonald Ryan, Koby Crammer, and Fernando Pereira. 2005. Flexible text segmentation with structured multilabel classification. In Proceedings of HLT/ EMNLP’05, pages 987–994, Vancouver.
18. McMahan H. Brendan and Matthew J. Streeter. 2010. Adaptive bound optimization for online convex optimization. In Proceedings of COLT’10, pages 244–256, Haifa.
19. Murata Noboru. 1998. A statistical study of on-line learning. In D. Saad, editor. Online Learning in Neural Networks. Cambridge University Press, pages 63–92.
20. Nocedal Jorge and Stephen J. Wright. 1999. Numerical optimization. Springer.
21. Okanohara Daisuke, Yusuke Miyao, Yoshimasa Tsuruoka, and Jun’ichi Tsujii. 2006. Improving the scalability of semi-Markov conditional random fields for named entity recognition. In Proceedings of COLING-ACL’06, pages 465–472, Sydney.
22. Ratnaparkhi Adwait. 1996. A maximum entropy model for part-of-speech tagging. In Proceedings of the Conference on Empirical Methods in Natural Language Processing 1996, pages 133–142, Pennsylvania.
23. Sang Erik Tjong Kim and Sabine Buchholz. 2000. Introduction to the CoNLL-2000 shared task: Chunking. In Proceedings of CoNLL’00, pages 127–132, Lisbon.
24. Schaul Tom, Sixin Zhang, and Yann LeCun. 2012. No more pesky learning rates. CoRR, abs/1206.1106.
25. Settles Burr. 2004. Biomedical named entity recognition using conditional random fields and rich feature sets. In Proceedings of BioNLP’04, pages 104–107, Geneva.
26. Shalev-Shwartz Shai, Yoram Singer, and Nathan Srebro. 2007. Pegasos: Primal estimated sub-gradient solver for SVM. In Proceedings of ICML’07, pages 807–814, Corvallis, OR.
27. Sperduti Alessandro and Antonina Starita. 1993. Speed up learning and network optimization with extended back propagation. Neural Networks, 6(3):365–383.
28. Sun Weiwei. 2010.Word-based and character-based word segmentation models: Comparison and combination. In COLING’10 (Posters), pages 1,211–1,219, Beijing.
29. Sun Xu, Takuya Matsuzaki, and Wenjie Li. 2013. Latent structured perceptrons for large-scale learning with hidden information. IEEE Transactions on Knowledge and Data Engineering, 25(9):2,063–2,075.
30. Sun Xu, Takuya Matsuzaki, Daisuke Okanohara, and Jun’ichi Tsujii. 2009. Latent variable perceptron algorithm for structured classification. In Proceedings of the 21st International Joint Conference on Artificial Intelligence (IJCAI 2009), pages 1,236–1,242, Pasadena, CA.
31. Sun Xu, Louis-Philippe Morency, Daisuke Okanohara, and Jun’ichi Tsujii. 2008. Modeling latent-dynamic in shallow parsing: A latent conditional model with improved inference. In Proceedings of COLING’08, pages 841–848, Manchester.
32. Sun Xu, Houfeng Wang, and Wenjie Li. 2012. Fast online training with frequencyadaptive learning rates for Chinese word segmentation and new word detection. In Proceedings of ACL’12, pages 253–262, Jeju Island.
33. Sun Xu, Yaozhong Zhang, Takuya Matsuzaki, Yoshimasa Tsuruoka, and Jun’ichi Tsujii. 2009. A discriminative latent variable Chinese segmenter with hybrid word/character information. In Proceedings of NAACL-HLT’09, pages 56–64, Boulder, CO.
34. Sun Xu, Yao Zhong Zhang, Takuya Matsuzaki, Yoshimasa Tsuruoka, and Jun’ichi Tsujii. 2013. Probabilistic Chinese word segmentation with non-local information and stochastic training. Information Processing & Management, 49(3):626–636.
35. Tseng Huihsin, Pichuan Chang, Galen Andrew, Daniel Jurafsky, and Christopher Manning. 2005. A conditional random field word segmenter for SIGHAN bakeoff 2005. In Proceedings of the Fourth SIGHAN Workshop, pages 168–171, Jeju Island.
36. Tsuruoka Yoshimasa, Jun’ichi Tsujii, and Sophia Ananiadou. 2009. Stochastic gradient descent training for l1-regularized log-linear models with cumulative penalty. In Proceedings of ACL’09, pages 477–485, Suntec.
37. Vishwanathan S. V. N., Nicol N. Schraudolph, Mark W. Schmidt, and Kevin P. Murphy. 2006. Accelerated training of conditional random fields with stochastic meta-descent. In Proceedings of ICML’06, pages 969–976, Pittsburgh, PA.
38. Zhang Ruiqiang, Genichiro Kikui, and Eiichiro Sumita. 2006. Subword-based tagging by conditional random fields for Chinese word segmentation. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, pages 193–196, New York City.
39. Zhang Yue and Stephen Clark. 2007. Chinese segmentation with a word-based perceptron algorithm. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 840–847, Prague.
40. Zhao, Hai, Changning Huang, Mu Li, and Bao-Liang Lu. 2010. A unified character-based tagging framework for Chinese word segmentation. ACM Transactions on Asian Language Information Processing, 9(2): Article 5.
41. Zhao Hai and Chunyu Kit. 2011. Integrating unsupervised and supervised word segmentation: The role of goodness measures. Information Sciences, 181(1):163–183.
42. Zinkevich Martin, Markus Weimer, Alexander J. Smola, and Lihong Li. 2010. Parallelized stochastic gradient descent. In Proceedings of NIPS’10, pages 2,595–2,603, Vancouver.