A neural probabilistic structured-prediction method for transition-based natural language processing

Journal of Artificial Intelligence Research

Volumn 58, Issue , 2017, Pages 703-729

  Zhou, Hao ^a   Zhang, Yue ^b   Cheng, Chuan ^a   Huang, Shujian ^a   Dai, Xinyu ^a   Chen, Jiajun ^a  

^a NANJING UNIVERSITY   (China)

^b SINGAPORE UNIVERSITY OF TECHNOLOGY AND DESIGN   (Singapore)

Author keywords

[No Author keywords available]

Indexed keywords

GLOBAL OPTIMIZATION; NATURAL LANGUAGE PROCESSING SYSTEMS;

ACCURACY IMPROVEMENT; BEAM SEARCH; DEPENDENCY PARSING; LOCALLY NORMALIZED; NATURAL LANGUAGE PROCESSING; NONLOCAL; STRUCTURED PREDICTION;

LEARNING SYSTEMS;

EID: 85019543974     PISSN: 10769757     EISSN: None     Source Type: Journal    
DOI: 10.1613/jair.5259     Document Type: Article

References (78)

1. Andor, D., Alberti, C., Weiss, D., Severyn, A., Presta, A., Ganchev, K., Petrov, S., & Collins, M. (2016). Globally normalized transition-based neural networks. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2442-2452. Association for Computational Linguistics.
2. Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
3. Ballesteros, M., Dyer, C., & Smith, A. N. (2015). Improved transition-based parsing by modeling characters instead of words with lstms. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 349-359. Association for Computational Linguistics.
4. Chang, K.-W., He, H., Ross, S., Daume III, H., & Langford, J. (2016). A credit assignment compiler for joint prediction. In Advances In Neural Information Processing Systems, pp. 1705-1713.
5. Chang, K.-W., Krishnamurthy, A., Agarwal, A., Daume, H., & Langford, J. (2015). Learning to search better than your teacher. In Proceedings of The 32nd International Conference on Machine Learning, pp. 2058-2066.
6. Chen, D., & Manning, C. D. (2014). A fast and accurate dependency parser using neural networks. In Empirical Methods in Natural Language Processing (EMNLP).
7. Chen, W., Zhang, Y., & Zhang, M. (2014). Feature embedding for dependency parsing.. In COLING, pp. 816-826.
8. Chen, W., Zhang, Y., & Isahara, H. (2006). An empirical study of chinese chunking. In Proceedings of the COLING/ACL on Main conference poster sessions, pp. 97-104. Association for Computational Linguistics.
9. Choi, J. D., & McCallum, A. (2013). Transition-based dependency parsing with selectional branching.. In ACL (1), pp. 1052-1062.
10. Collins, M. (1999). HEAD-DRIVEN STATISTICAL MODELS FOR NATURAL LANGUAGE PARSING. Ph. D. thesis, University of Pennsylvania.
11. Collins, M. (2002). Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms. In Proceedings of the ACL-02 conference on Empirical methods in natural language processing-Volume 10, pp. 1-8. Association for Computational Linguistics.
12. Collins, M., & Roark, B. (2004). Incremental parsing with the perceptron algorithm. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, p. 111. Association for Computational Linguistics.
13. Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. (2011). Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12, 2493-2537.
14. Conti, M., Di Pietro, R., Mancini, L. V., & Mei, A. (2009). (old) distributed data source verification in wireless sensor networks. Inf. Fusion, 10(4), 342-353.
15. Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273-297.
16. Daumé III, H., Langford, J., & Marcu, D. (2009). Search-based structured prediction. Machine learning, 75(3), 297-325.
17. Daumé III, H., & Marcu, D. (2005). Learning as search optimization: Approximate large margin methods for structured prediction. In Proceedings of the 22nd international conference on Machine learning, pp. 169-176. ACM.
18. Devlin, J., Zbib, R., Huang, Z., Lamar, T., Schwartz, R. M., & Makhoul, J. (2014). Fast and robust neural network joint models for statistical machine translation.. In ACL (1), pp. 1370-1380. Citeseer.
19. Do, T., Arti, T., et al. (2010). Neural conditional random fields. In International Conference on Artificial Intelligence and Statistics, pp. 177-184.
20. Doppa, J. R., Fern, A., & Tadepalli, P. (2014). Hc-search: A learning framework for searchbased structured prediction.. J. Artif. Intell. Res. (JAIR), 50, 369-407.
21. Duan, X., Zhao, J., & Xu, B. (2007). Probabilistic parsing action models for multi-lingual dependency parsing..
22. Duchi, J., Hazan, E., & Singer, Y. (2011). Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12, 2121-2159.
23. Durrett, G., & Klein, D. (2015). Neural crf parsing. arXiv preprint arXiv:1507.03641.
24. Dyer, C., Ballesteros, M., Ling, W., Matthews, A., & Smith, N. A. (2015). Transitionbased dependency parsing with stack long short-term memory. arXiv preprint arXiv:1505.08075.
25. Goldberg, Y., & Nivre, J. (2013). Training deterministic parsers with non-deterministic oracles. Transactions of the Association for Computational Linguistics, 1, 403-414.
26. Hinton, G. (2002). Training products of experts by minimizing contrastive divergence. Neural computation, 14(8), 1771-1800.
27. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780.
28. Huang, L., Fayong, S., & Guo, Y. (2012). Structured perceptron with inexact search. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 142-151. Association for Computational Linguistics.
29. Huang, L., & Sagae, K. (2010). Dynamic programming for linear-time incremental parsing. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pp. 1077-1086. Association for Computational Linguistics.
30. Huang, Z., Xu, W., & Yu, K. (2015). Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991.
31. Kiperwasser, E., & Goldberg, Y. (2016a). Easy-first dependency parsing with hierarchical tree lstms. Transactions of the Association for Computational Linguistics, 4, 445-461.
32. Kiperwasser, E., & Goldberg, Y. (2016b). Simple and accurate dependency parsing using bidirectional lstm feature representations. Transactions of the Association for Computational Linguistics, 4, 313-327.
33. Koo, T., Carreras, X., & Collins, M. (2008). Simple semi-supervised dependency parsing..
34. Kudo, T., & Matsumoto, Y. (2001). Chunking with support vector machines. In Proceedings of the second meeting of the North American Chapter of the Association for Computational Linguistics on Language technologies, pp. 1-8. Association for Computational Linguistics.
35. Lafferty, J., McCallum, A., & Pereira, F. C. (2001). Conditional random fields: Probabilistic models for segmenting and labeling sequence data..
36. LeCun, Y., & Huang, F. (2005). Loss functions for discriminative training of energybased models.. AIStats.
37. Liang, P., & Jordan, M. I. (2008). An asymptotic analysis of generative, discriminative, and pseudolikelihood estimators. In Proceedings of the 25th international conference on Machine learning, pp. 584-591. ACM.
38. Ling, W., Luís, T., Marujo, L., Astudillo, R. F., Amir, S., Dyer, C., Black, A. W., & Trancoso, I. (2015). Finding function in form: Compositional character models for open vocabulary word representation. arXiv preprint arXiv:1508.02096.
39. Liu, Y., & Sun, M. (2014). Contrastive unsupervised word alignment with non-local features. arXiv preprint arXiv:1410.2082.
40. Lyu, C., Zhang, Y., & Ji, D. (2016). Joint word segmentation, pos-tagging and syntactic chunking. In Thirtieth AAAI Conference on Artificial Intelligence.
41. Ma, J., Zhang, Y., & Zhu, J. (2014a). Punctuation processing for projective dependency parsing. In Proc. of ACL.
42. Ma, J., Zhang, Y., & Zhu, J. (2014b). Tagging the web: Building a robust web tagger with neural network. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 144-154, Baltimore, Maryland. Association for Computational Linguistics.
43. Marcus, M. P., Marcinkiewicz, M. A., & Santorini, B. (1993). Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2), 313-330.
44. Nivre, J. (2004). Incrementality in deterministic dependency parsing. In Incremental Parsing: Bringing Engineering and Cognition Together (ACL Workshop), pp. 50-57.
45. Nivre, J. (2008). Algorithms for deterministic incremental dependency parsing. Computational Linguistics, 34(4), 513-553.
46. Nivre, J., Hall, J., Nilsson, J., Chanev, A., Eryigit, G., Kübler, S., Marinov, S., & Marsi, E. (2007). Maltparser: A language-independent system for data-driven dependency parsing. Natural Language Engineering, 13(2), 95-135.
47. Nivre, J., & Scholz, M. (2004). Deterministic dependency parsing of English text. In Proceedings of the 20th International Conference on Computational Linguistics (COLING), pp. 64-70.
48. Sha, F., & Pereira, F. (2003). Shallow parsing with conditional random fields. In Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, pp. 213-220.
49. Shen, H., & Sarkar, A. (2005). Voting between multiple data representations for text chunking. Springer.
50. Socher, R., Bauer, J., Manning, C. D., & Ng, A. Y. (2013). Parsing with compositional vector grammars. In In Proceedings of the ACL conference. Citeseer.
51. Sun, X., Morency, L.-P., Okanohara, D., & Tsujii, J. (2008). Modeling latent-dynamic in shallow parsing: a latent conditional model with improved inference. In Proceedings of the 22nd International Conference on Computational Linguistics-Volume 1, pp. 841-848. Association for Computational Linguistics.
52. Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104-3112.
53. Suzuki, J., Isozaki, H., Carreras, X., & Collins, M. (2009). An empirical study of semisupervised structured conditional models for dependency parsing. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 2-Volume 2, pp. 551-560. Association for Computational Linguistics.
54. Tjong Kim Sang, E. F., & Buchholz, S. (2000). Introduction to the conll-2000 shared task: Chunking. In Proceedings of the 2nd workshop on Learning language in logic and the 4th conference on Computational natural language learning-Volume 7, pp. 127-132. Association for Computational Linguistics.
55. Vickrey, D., Lin, C. C, & Koller, D. (2010). Non-local contrastive objectives. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 1103-1110.
56. Vinyals, O., Kaiser, L., Koo, T., Petrov, S., Sutskever, I., & Hinton, G. (2015). Grammar as a foreign language. In Advances in Neural Information Processing Systems, pp. 2773-2781.
57. Watanabe, T., & Sumita, E. (2015). Transition-based neural constituent parsing. Proceedings of A CL-IJCNLP.
58. Weiss, D., Alberti, C, Collins, M., & Petrov, S. (2015). Structured training for neural network transition-based parsing. arXiv preprint arXiv:1506.06158.
59. Wiseman, S., & Rush, M. A. (2016). Sequence-to-sequence learning as beam-search optimization. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1296-1306. Association for Computational Linguistics.
60. Xu, W., Auli, M., & Clark, S. (2016). Expected f-measure training for shift-reduce parsing with recurrent neural networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 210-220, San Diego, California. Association for Computational Linguistics.
61. Xu, W., Clark, S., & Zhang, Y. (2014). Shift-reduce ccg parsing with a dependency model. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 218-227, Baltimore, Maryland. Association for Computational Linguistics.
62. Xue, N., Xia, F., Chiou, F.-D., & Palmer, M. (2005). The penn chinese treebank: Phrase structure annotation of a large corpus. Natural language engineering, 11(2), 207-238.
63. Yamada, H., & Matsumoto, Y. (2003). Statistical dependency analysis with support vector machines. In Proceedings of IWPT, Vol. 3.
64. Yao, K., Peng, B., Zweig, G., Yu, D., Li, X., & Gao, F. (2014). Recurrent conditional random field for language understanding. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4077-4081. IEEE.
65. Zhang, M., Zhang, Y., Che, W., & Liu, T. (2013). Chinese parsing exploiting characters. In 51st Annual Meeting of the Association for Computational Linguistics.
66. Zhang, M., Zhang, Y., & Fu, G. (2016). Transition-based neural word segmentation. Proceedings of the 54nd ACL.
67. Zhang, Y., & Clark, S. (2007). Chinese segmentation with a word-based perceptron algorithm. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pp. 840-847, Prague, Czech Republic. Association for Computational Linguistics.
68. Zhang, Y., & Clark, S. (2008). A tale of two parsers: investigating and combining graphbased and transition-based dependency parsing using beam-search. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 562-571. Association for Computational Linguistics.
69. Zhang, Y., & Clark, S. (2009). Transition-based parsing of the chinese treebank using a global discriminative model. In Proceedings of the 11th International Conference on Parsing Technologies, pp. 162-171. Association for Computational Linguistics.
70. Zhang, Y., & Clark, S. (2011a). Shift-reduce ccg parsing. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pp. 683-692. Association for Computational Linguistics.
71. Zhang, Y., & Clark, S. (2011b). Syntactic processing using the generalized perceptron and beam search. Computational Linguistics, 37(1), 105-151.
72. Zhang, Y., & Nivre, J. (2011). Transition-based dependency parsing with rich non-local features. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2, pp. 188-193. Association for Computational Linguistics.
73. Zhang, Y., & Nivre, J. (2012). Analyzing the effect of global learning and beam-search on transition-based dependency parsing.. In COLING (Posters), pp. 1391-1400.
74. Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C., & Torr, P. H. (2015). Conditional random fields as recurrent neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1529-1537.
75. Zhou, H., Zhang, Y., Huang, S., & Chen, J. (2015). A neural probabilistic structuredprediction model for transition-based dependency parsing. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1213-1222, Beijing, China. Association for Computational Linguistics.
76. Zhou, H., Zhang, Y., Huang, S., Zhou, J., Dai, X.-Y., & Chen, J. (2016). A search-based dynamic reranking model for dependency parsing. In In Proceeding of ACL.
77. Zhou, J., Qu, W., & Zhang, F. (2012). Exploiting chunk-level features to improve phrase chunking. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 557-567. Association for Computational Linguistics.
78. Zhu, M., Zhang, Y., Chen, W., Zhang, M., & Zhu, J. (2013). Fast and accurate shift-reduce constituent parsing. In 51st Annual Meeting of the Association for Computational Linguistics.