Neural networks leverage corpus-wide information for part-of-speech tagging

EMNLP 2014 - 2014 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference

Volumn , Issue , 2014, Pages 938-950

  Tsuboi, Yuta ^a  

^a IBM RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; NATURAL LANGUAGE PROCESSING SYSTEMS; SYNTACTICS;

ACTIVATION FUNCTIONS; CONTINUOUS FEATURES; ENGLISH POS TAGGING; LINEAR MODELING; NONLINEAR INTERACTIONS; PART OF SPEECH TAGGING; STATE OF THE ART;

FEEDFORWARD NEURAL NETWORKS;

EID: 84926040142     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.3115/v1/d14-1101     Document Type: Conference Paper

References (50)

1. Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2014. Tailoring continuous word representations for dependency parsing. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (ACL). The Association for Computer Linguistics.
2. James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13:281-305.
3. Bernd Bohnet and Joakim Nivre. 2012. A transitionbased system for joint part-of-speech tagging and labeled non-projective dependency parsing. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Com-putational Natural Language Learning (EMNLPCoNLL), pages 1455-1465.
4. Y-Lan Boureau, Jean Ponce, and Yann LeCun. 2010. A theoretical analysis of feature pooling in visual recognition. In Proceedings of the International Conference on Machine Learning (ICML), pages 111-118.
5. Jinho D. Choi and Martha Palmer. 2012. Fast and robust part-of-speech tagging using dynamic model selection. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (ACL), pages 363-367.
6. Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel P. Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493-2537.
7. Koby Crammer and Yoram Singer. 2001. On the algorithmic implementation of multiclass kernel-based vector machines. Journal of Machine Learning Research, 2:265-292.
8. Hal Daume III, John Langford, and Daniel Marcu. 2009. Search-based structured prediction. Machine Learning Journal, 75(3):297-325, June.
9. John C. Duchi, Elad Hazan, and Yoram Singer. 2010. Adaptive subgradient methods for online learning and stochastic optimization. In Proceedings of the Conference on Learning Theory (COLT), pages 257-269.
10. Jesus Gimenez and Lluis Marquez. 2003. Fast and accurate part-of-speech tagging: The SVM approach revisited. In Proceedings of Recent Advances in Natural Language Processing (RANLP), pages 153-163.
11. Yoav Goldberg and Joakim Nivre. 2012. A dynamic oracle for arc-eager dependency parsing. In Proceedings of International Conference on Computational Linguistics (COLING), pages 959-976.
12. Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron C. Courville, and Yoshua Bengio. 2013. Maxout networks. In Proceedings of the International Conference on Machine Learning (ICML), pages 1319-1327.
13. Ian J. Goodfellow, Mehdi Mirza, Xia Da, Aaron C. Courville, and Yoshua Bengio. 2014. An empirical investigation of catastrophic forgeting in gradientbased neural networks. In Proceedings of International Conference on Learning Representations (ICLR).
14. Caglar Gulcehre, Kyunghyun Cho, Razvan Pascanu, and Yoshua Bengio. 2014. Learned-norm pooling for deep feedforward and recurrent neural networks. In Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML/PKDD).
15. Jan Hajic, Massimiliano Ciaramita, Richard Johansson, Daisuke Kawahara, Maria Antonia Marti, Lluis Marquez, Adam Meyers, Joakim Nivre, Sebastian Pado, Jan Stepanek, Pavel Stranak, Mihai Surdeanu, Nianwen Xue, and Yi Zhang. 2009. The CoNLL- 2009 shared task: Syntactic and semantic dependencies in multiple languages. In Proceedings of the Conference on Computational Natural Language Learning (CoNLL), pages 1-18.
16. Fei Huang and Alexander Yates. 2009. Distributional representations for handling sparsity in supervised sequence-labeling. In Proceedings of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (ACL/IJCNLP), pages 495-503.
17. Quoc V. Le, Tamas Sarlos, and Alexander J. Smola. 2013. Fastfood - computing Hilbert space expansions in loglinear time. In Proceedings of the International Conference on Machine Learning (ICML), pages 244-252.
18. Omer Levy and Yoav Goldberg. 2014. Linguistic regularities in sparse and explicit word representations. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL).
19. Ji Ma, Yue Zhang, Tong Xiao, and Jingbo Zhu. 2014. Tagging the Web: Building a robust web tagger with neural network. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (ACL). The Association for Computer Linguistics.
20. Christopher D. Manning. 2011. Part-of-speech tagging from 97% to 100%: Is it time for some linguistics? In Proceedings of Conference on Intelligent Text Processing and Computational Linguistics (CICLing), pages 171-189.
21. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of english: The Penn treebank. Computational Linguistics, 19(2):313-330.
22. H. Brendan McMahan, Gary Holt, David Sculley, Michael Young, Dietmar Ebner, Julian Grady, Lan Nie, Todd Phillips, Eugene Davydov, Daniel Golovin, Sharat Chikkerur, Dan Liu, Martin Wattenberg, Arnar Mar Hrafnkelsson, Tom Boulos, and Jeremy Kubica. 2013. Ad click prediction: A view from the trenches. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pages 1222- 1230.
23. H. Brendan McMahan. 2011. Follow-the-regularizedleader and mirror descent: Equivalence theorems and L1 regularization. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (AISTATS), pages 525-533.
24. Gregoire Mesnil, Xiaodong He, Li Deng, and Yoshua Bengio. 2013. Investigation of recurrent-neuralnetwork architectures and learning methods for spoken language understanding. In Proceedings of Annual Conference of the International Speech Communication Association (INTERSPEECH), pages 3771-3775.
25. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of Advances in Neural Information Processing Systems (NIPS), pages 3111- 3119.
26. Vinod Nair and Geoffrey E. Hinton. 2010. Rectified linear units improve restricted Boltzmann machines. In Proceedings of the International Conference on Machine Learning (ICML), pages 807-814.
27. Tetsuji Nakagawa and Yuji Matsumoto. 2006. Guessing parts-of-speech of unknown words using global information. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (ACL).
28. Masami Nakamura, Katsuteru Maruyama, Takeshi Kawabata, and Kiyohiro Shikano. 1990. Neural network approach to word category prediction for English texts. In Proceedings of International Conference on Computational Linguistics (COLING), pages 213-218.
29. Hiroki Ouchi, Kevin Duh, and Yuji Matsumoto. 2014. Improving dependency parsers with supertags. In Proceedings of Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 154-158.
30. Neal Parikh and Stephen P. Boyd. 2013. Proximal algorithms. Foundations and Trends in Optimization, 1(3):123-231.
31. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: global vectors for word representation. In Proceedings of the Conference on Empirical Methods on Natural Language Processing (EMNLP).
32. Ninh Pham and Rasmus Pagh. 2013. Fast and scalable polynomial kernels via explicit feature maps. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pages 239-247.
33. Stephane Ross, Geoffrey J. Gordon, and Drew Bagnell. 2011. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (AISTATS), pages 627-635.
34. Cicero Dos Santos and Bianca Zadrozny. 2014. Learning character-level representations for part-ofspeech tagging. In Proceedings of the International Conference on Machine Learning (ICML), pages 1818-1826.
35. Helmut Schmid. 1994. Part-of-speech tagging with neural networks. In Proceedings of International Conference on Computational Linguistics (COLING), pages 172-176.
36. Tobias Schnabel and Hinrich Schutze. 2014. FLORS: Fast and simple domain adaptation for part-ofspeech tagging. Transactions of the Association for Computational Linguistics, 2:15-26, February.
37. Richard Socher, Eric H. Huang, Jeffrey Pennington, Andrew Y. Ng, and Christopher D. Manning. 2011. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Proceedings of Advances in Neural Information Processing Systems (NIPS), pages 801-809.
38. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Chris Manning, Andrew Ng, and Chris Potts. 2013. Recursive deep models for semantic compositionality over a sentiment tree bank. In Proceedings of the Conference on Empirical Methods on Natural Language Processing (EMNLP), pages 1631-1642.
39. Anders Søgaard. 2011. Semi-supervised condensed nearest neighbor for part-of-speech tagging. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (ACL), pages 48-52.
40. Drahomira johanka Spoustova, Jan Hajic, Jan Raab, and Miroslav Spousta. 2009. Semi-supervised training for the averaged perceptron pos tagger. In Proceedings of Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 763-771.
41. Weiwei Sun and Hans Uszkoreit. 2012. Capturing paradigmatic and syntagmatic lexical relations: Towards accurate Chinese part-of-speech tagging. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (ACL), pages 242-252.
42. Weiwei Sun, Xiaochang Peng, and Xiaojun Wan. 2013. Capturing long-distance dependencies in sequence models: A case study of Chinese part-ofspeech tagging. In Proceedings of the International Joint Conference on Natural Language Processing (IJCNLP), pages 180-188.
43. Ilya Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton. 2013. On the importance of initialization and momentum in deep learning. In Proceedings of the International Conference on Machine Learning (ICML), pages 1139-1147.
44. Jun Suzuki and Hideki Isozaki. 2008. Semi-supervised sequential labeling and segmentation using gigaword scale unlabeled data. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (ACL), pages 665-673.
45. Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. 2003. Feature-rich partof- speech tagging with a cyclic dependency network. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 252-259.
46. Yoshimasa Tsuruoka and Jun'ichi Tsujii. 2005. Bidirectional inference with the easiest-first strategy for tagging sequence data. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT-EMNLP), pages 467-474.
47. Joseph P. Turian, Lev-Arie Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (ACL), pages 384-394.
48. Mengqiu Wang and Christopher D. Manning. 2013. Effect of non-linear deep architecture in sequence labeling. In Proceedings of the International Joint Conference on Natural Language Processing (IJCNLP).
49. Xiaohui Zhang, Jan Trmal, Daniel Povey, and Sanjeev Khudanpur. 2014. Improving deep neural network acoustic models using generalized maxout networks. In Proceedings of International Conference on Acoustics, Speech and Signal Processing (ICASSP).
50. Alisa Zhila, Wen tau Yih, Christopher Meek, Geoffrey Zweig, and Tomas Mikolov. 2013. Combining heterogeneous models for measuring relational similarity. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 1000-1009.