Learning distributed word representations for bidirectional LSTM recurrent neural network

2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016 - Proceedings of the Conference

Volumn , Issue , 2016, Pages 527-533

  Wang, Peilu ^a,b   Qian, Yao ^c   Zhao, Hai ^a   Soong, Frank K ^b   He, Lei ^b   Wu, Ke ^a  

^a SHANGHAI JIAO TONG UNIVERSITY   (China)

^b MICROSOFT RESEARCH ASIA   (China)

^c Educational Testing Service Research   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; NATURAL LANGUAGE PROCESSING SYSTEMS; SPEECH RECOGNITION;

DISTRIBUTED REPRESENTATION; NAMED ENTITY RECOGNITION; PART OF SPEECH TAGGING; RECURRENT NEURAL NETWORK (RNN); UNLABELED DATA; WORD REPRESENTATIONS;

LONG SHORT-TERM MEMORY;

EID: 84994157362     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.18653/v1/n16-1064     Document Type: Conference Paper

References (32)

1. Samy Bengio and Georg Heigold. 2014. Word embeddings for speech recognition. In INTERSPEECH, pages 1053-1057.
2. Yoshua Bengio, Rejean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. In Journal of Machine Learning Research (JMLR), volume 3, pages 1137-1155.
3. Michael Collins. 2002. Discriminative training methods for Hidden Markov Models: Theory and experiments with perceptron algorithms. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP), pages 1-8.
4. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of International Conference on Machine Learning (ICML), pages 160-167.
5. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research (JMLR), 12:2493-2537.
6. Scott Deerwester, Susan Dumais, Thomas Landauer, George Furnas, and Richard Harshman. 1990. Indexing by latent semantic analysis. JASIS, 41(6):391-407.
7. Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In Proceedings of Association for Computational Linguistics (ACL), pages 334-343.
8. David Graff. 2008. North American News Text, Complete LDC2008T15. https://catalog.ldc.upenn.edu/LDC2008T15.
9. Michael Gutmann and Aapo Hyvärinen. 2012. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of Machine Learning Research (JMLR), 13:307-361.
10. Changning Huang and Hai Zhao. 2007. Chinese word segmentation: A decade review. Journal of Chinese Information Processing, 21(3):8-20.
11. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. arXiv:1508.01991.
12. Kevin Lund and Curt Burgess. 1996. Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, Instruments, & Computers, 28(2):203-208.
13. Mitchell Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational linguistics (CL), 19(2):313-330.
14. Grégoire Mesnil, Xiaodong He, Li Deng, and Yoshua Bengio. 2013. Investigation of recurrent-neuralnetwork architectures and learning methods for spoken language understanding. In INTERSPEECH, pages 3771-3775.
15. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In INTERSPEECH, pages 1045-1048.
16. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
17. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013b. word2vec. https://code.google.com/p/word2vec/.
18. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeff Dean. 2013c. Distributed representations of words and phrases and their compositionality. In Proceedings of Conference on Neural Information Processing Systems (NIPS), pages 3111-3119.
19. Andriy Mnih and Geoffrey Hinton. 2007. Three new graphical models for statistical language modelling. In Proceedings of International Conference on Machine Learning (ICML), pages 641-648.
20. Andriy Mnih and Yee Whye Teh. 2012. A fast and simple algorithm for training neural probabilistic language models. In Proceedings of International Conference on Machine Learning (ICML), pages 1751-1758.
21. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014a. Glove: Global vectors for word representation. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
22. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014b. GloVe. http://nlp.stanford.edu/projects/glove/.
23. Erik Tjong Kim Sang and Sabine Buchholz. 2000. Introduction to the CoNLL-2000 shared task: Chunking. In Proceedings of Conference on Natural Language Learning (CoNLL), pages 127-132.
24. Erik Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Languageindependent named entity recognition. In Proceedings of Conference on Natural Language Learning (CoNLL), pages 142-147.
25. Ashish Vaswani, Yinggong Zhao, Victoria Fossum, and David Chiang. 2013. Decoding with large-scale neural language models improves translation. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP), pages 1387-1392.
26. Peilu Wang, Yao Qian, Frank Soong, Lei He, and Hai Zhao. 2015. Word embedding for recurrent neural network based tts synthesis. In Proceedings of International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pages 4879-4883.
27. Felix Weninger, Johannes Bergmann, and Björn Schuller. 2014. Introducing CURRENNT-the Munich open-source CUDA recurrent Neural Network Toolkit. Journal of Machine Learning Research (JMLR), 16(1):547-551.
28. Hai Zhao and Chunyu Kit. 2008a. An empirical comparison of goodness measures for unsupervised chinese word segmentation with a unified framework. In Proceedings of International Joint Conference on Natural Language Processing (IJCNLP), volume 1, pages 9-16.
29. Hai Zhao and Chunyu Kit. 2008b. Unsupervised segmentation helps supervised learning of character tagging for word segmentation and named entity recognition. In SIGHAN Workshop on Chinese Language Processing, pages 106-111.
30. Hai Zhao and Chunyu Kit. 2011. Integrating unsupervised and supervised word segmentation: The role of goodness measures. Information Sciences, 181(1):163-183.
31. Hai Zhao, Chang-Ning Huang, and Mu Li. 2006. An improved chinese word segmentation system with conditional random field. In SIGHAN Workshop on Chinese Language Processing, pages 162-165.
32. Hai Zhao, Chang-Ning Huang, Mu Li, and Bao-Liang Lu. 2010. A unified character-based tagging framework for chinese word segmentation. ACM Transactions on Asian Language Information Processing (TALIP), 9(2):1-32.