Learning Representations for Weakly Supervised Natural Language Processing Tasks

Computational Linguistics

Volumn 40, Issue 1, 2014, Pages 85-120

  Huang, Fei ^a,c   Ahuja, Arun ^b,c   Downey, Doug ^b,c   Yang, Yi ^b,c   Guo, Yuhong ^a,c   Yates, Alexander ^a,c  

^a TEMPLE UNIVERSITY   (United States)

^b NORTHWESTERN UNIVERSITY   (United States)

^c NONE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

HIDDEN MARKOV MODELS; IMAGE SEGMENTATION; LABELED DATA; NATURAL LANGUAGE PROCESSING SYSTEMS;

DOMAIN SPECIFIC; EXTRACTING FEATURES; MARKOV RANDOM FIELD MODEL; NATURAL LANGUAGE PROCESSING; NOVEL TECHNIQUES; PART OF SPEECH TAGGING; POLYSEMOUS WORD; STATISTICAL LANGUAGE MODELS;

COMPUTATIONAL LINGUISTICS;

EID: 84897743792     PISSN: 08912017     EISSN: 15309312     Source Type: Journal    
DOI: 10.1162/COLI_a_00167     Document Type: Article

References (99)

1. Ahuja, Arun and Doug Downey. 2010. Improved extraction assessment through better language models. In Proceedings of the Annual Meeting of the North American Chapter of the Association of Computational Linguistics (NAACL-HLT), pages 225-228, Los Angeles, CA.
2. Ando, Rie Kubota and Tong Zhang. 2005. A high-performance semi-supervised learning method for text chunking. In Proceedings of the ACL, pages 1-9, Ann Arbor, MI.
3. Banko, Michele, Michael J. Cafarella, Stephen Soderland, Matt Broadhead, and Oren Etzioni. 2007. Open information extraction from the web. In Proceedings of the IJCAI, pages 2670-2676, Hyderabad.
4. Banko, Michele and Robert C. Moore. 2004. Part of speech tagging in context. In Proceedings of the COLING, pages 556-561, Geneva.
5. Ben-David, Shai, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jenn Wortman. 2010. A theory of learning from different domains. Machine Learning, 79:151-175.
6. Ben-David, Shai, John Blitzer, Koby Crammer, and Fernando Pereira. 2007. Analysis of representations for domain adaptation. In Advances in Neural Information Processing Systems 20, pages 127-144, Vancouver.
7. Bengio, Yoshua. 2008. Neural net language models. Scholarpedia, 3(1):3,881.
8. Bengio, Yoshua, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1,137-1,155.
9. Bengio, Yoshua, Jerome Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the International Conference on Machine Learning (ICML), pages 41-48, Montreal.
10. Bikel, Daniel M. 2004a. A distributional analysis of a lexicalized statistical parsing model. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 182-189, Barcelona.
11. Bikel, Daniel M. 2004b. Intricacies of Collins' parsing model. Computational Linguistics, 30(4):479-511.
12. Blei, David M., Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993-1,022.
13. Blitzer, John. 2008. Domain Adaptation of Natural Language Processing Systems. Ph.D. thesis, University of Pennsylvania, Philadelphia, PA.
14. Blitzer, John, Koby Crammer, Alex Kulesza, Fernando Pereira, and JennWortman. 2007. Learning bounds for domain adaptation. In Advances in Neural Information Processing Systems, pages 129-136, Vancouver.
15. Blitzer, John, Mark Dredze, and Fernando Pereira. 2007. Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Association for Computational Linguistics (ACL), pages 40-47, Prague.
16. Blitzer, John, Ryan McDonald, and Fernando Pereira. 2006. Domain adaptation with structural correspondence learning. In Proceedings of the EMNLP, pages 120-128, Sydney.
17. Bodlaender, Hans L. 1988. Dynamic programming on graphs with bounded treewidth. In Proceedings of the 15th International Colloquium on Automata, Languages and Programming, pages 105-118, Tampere.
18. Brants, Thorsten and Alex Franz. 2006. Web 1t 5-gram version 1. www.ldc. upenn.edu/catalog/.
19. Brown, Peter F., Vincent J. Della Pietra, Peter V. deSouza, Jenifer C. Lai, and Robert L. Mercer. 1992. Class-based n-gram models of natural language. Computational Linguistics, 18:467-479.
20. Candito, Marie and Benoit Crabbe. 2009. Improving generative statistical parsing with semi-supervised word clustering. In Proceedings of the IWPT, pages 138-141, Paris.
21. Chan, Yee Seng and Hwee Tou Ng. 2006. Estimating class priors in domain adaptation for word sense disambiguation. In Proceedings of the Association for Computational Linguistics (ACL), pages 89-96, Sydney.
22. Chelba, Ciprian and Alex Acero. 2004. Adaptation of maximum entropy classifier: Little data can help a lot. In Proceedings of the EMNLP, pages 285-292, Barcelona.
23. Collobert, Robert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the International Conference on Machine Learning (ICML), pages 160-167, Helsinki.
24. Dai, Wenyuan, Gui-Rong Xue, Qiang Yang, and Yong Yu. 2007. Transferring naive Bayes classifiers for text classification. In Proceedings of the National Conference on Artificial Intelligence (AAAI), pages 540-545, Vancouver.
25. Darroch, J. N., S. L. Lauritzen, and T. P. Speed. 1980. Markov fields and log-linear interaction models for contingency tables. The Annals of Statistics, 8(3):522-539.
26. Daumé III, Hal. 2007. Frustratingly easy domain adaptation. In Proceedings of the ACL, pages 256-263, Prague.
27. Daumé III, Hal, Abhishek Kumar, and Avishek Saha. 2010. Frustratingly easy semi-supervised domain adaptation. In Proceedings of the ACL Workshop on Domain Adaptation (DANLP), pages 53-59, Uppsala.
28. Daumé III, Hal and Daniel Marcu. 2006. Domain adaptation for statistical classifiers. Journal of Artificial Intelligence Research, 26:101-126.
29. Deerwester, Scott, Susan T. Dumais, GeorgeW. Furnas, Thomas K. Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society of Information Science, 41(6):391-407.
30. Dempster, Arthur, Nan Laird, and Donald Rubin. 1977. Likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, Series B, 39(1):1-38.
31. Deschacht, Koen and Marie-Francine Moens. 2009. Semi-supervised semantic role labeling using the latent words language model. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 21-29, Singapore.
32. Dhillon, Paramveer S., Dean Foster, and Lyle Ungar. 2011. Multi-View Learning of Word Embeddings via CCA. In Proceedings of the Advances in Neural Information Processing Systems (NIPS), volume 24, pages 886-874, Granada.
33. Downey, Doug, Matthew Broadhead, and Oren Etzioni. 2007. Locating complex named entities in web text. In Proceedings of the 20th International Joint Conference on Artificial Intelligence (IJCAI 2007), pages 2,733-2,739, Hyderabad.
34. Downey, Doug, Stefan Schoenmackers, and Oren Etzioni. 2007. Sparse information extraction: Unsupervised language models to the rescue. In Proceedings of the ACL, pages 696-703, Prague.
35. Dredze, Mark and Koby Crammer. 2008. Online methods for multi-domain learning and adaptation. In Proceedings of EMNLP, pages 689-697, Honolulu, HI.
36. Dredze, Mark, Alex Kulesza, and Koby Crammer. 2010. Multi-domain learning by confidence weighted parameter combination. Machine Learning, 79:123-149.
37. Finkel, Jenny Rose and Christopher D. Manning. 2009. Hierarchical Bayesian domain adaptation. In Proceedings of HLT-NAACL, pages 602-610, Boulder, CO.
38. Fürstenau, Hagen and Mirella Lapata. 2009. Semi-supervised semantic role labeling. In Proceedings of the 12th Conference of the European Chapter of the ACL, pages 220-228, Athens.
39. Ghahramani, Zoubin and Michael I. Jordan. 1997. Factorial hidden Markov models. Machine Learning, 29(2-3):245-273.
40. Gildea, Daniel. 2001. Corpus variation and parser performance. In Conference on Empirical Methods in Natural Language Processing, pages 167-202, Pittsburgh, PA.
41. Goldwater, Sharon and Thomas L. Griffiths. 2007. A fully Bayesian approach to unsupervised part-of-speech tagging. In Proceedings of the ACL, pages 744-751, Prague.
42. Graça, João V., Kuzman Ganchev, Ben Taskar, and Fernando Pereira. 2009. Posterior vs. parameter sparsity in latent variable models. In Proceedings of the Neural Information Processing Systems Conference (NIPS), pages 664-672, Vancouver.
43. Harris, Z. 1954. Distributional structure. Word, 10(23):146-162.
44. Hindle, Donald. 1990. Noun classification from predicage-argument structures. In Proceedings of the ACL, pages 268-275, Pittsburgh, PA.
45. Honkela, Timo. 1997. Self-organizing maps of words for natural language processing applications. In Proceedings of the International ICSC Symposium on Soft Computing, pages 401-407, Millet, Alberta.
46. Huang, Fei and Alexander Yates. 2009. Distributional representations for handling sparsity in supervised sequence labeling. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 495-503, Singapore.
47. Huang, Fei and Alexander Yates. 2010. Exploring representation-learning approaches to domain adaptation. In Proceedings of the ACL 2010 Workshop on Domain Adaptation for Natural Language Processing (DANLP), pages 23-30, Uppsala.
48. Huang, Fei, Alexander Yates, Arun Ahuja, and Doug Downey. 2011. Language models as representations for weakly supervised NLP tasks. In Proceedings of the Conference on Natural Language Learning (CoNLL), pages 125-134, Portland, OR.
49. Jiang, Jing and ChengXiang Zhai. 2007a. Instance weighting for domain adaptation in NLP. In Proceedings of ACL, pages 264-271, Prague.
50. Jiang, Jing and ChengXiang Zhai. 2007b. A two-stage approach to domain adaptation for statistical classifiers. In Proceedings of the Conference on Information and Knowledge Management (CIKM), pages 401-410, Lisbon.
51. Johnson, Mark. 2007. Why doesn't EM find good HMM POS-taggers. In Proceedings of the EMNLP, pages 296-305, Prague.
52. Kaski, S. 1998. Dimensionality reduction by random mapping: Fast similarity computation for clustering. In Proceedings of the IJCNN, pages 413-418, Washington, DC.
53. Koo, Terry, Xavier Carreras, and Michael Collins. 2008. Simple semi-supervised dependency parsing. In Proceedings of the Annual Meeting of the Association of Computational Linguistics (ACL), pages 595-603, Columbus, OH.
54. Lin, Dekang and XiaoyunWu. 2009. Phrase clustering for discriminative learning. In Proceedings of the ACL-IJCNLP, pages 1,030-1,038, Singapore.
55. Liu, Dong C. and Jorge Nocedal. 1989. On the limited memory method for large scale optimization. Mathematical Programming B, 45(3):503-528.
56. Mansour, Y., M. Mohri, and A. Rostamizadeh. 2009. Domain adaptation with multiple sources. In Proceedings of the Advances in Neural Information Processing Systems, pages 1,041-1,048, Vancouver.
57. Marcus, Mitchell P., Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313-330.
58. Martin, Sven, Jorg Liermann, and Hermann Ney. 1998. Algorithms for bigram and trigram word clustering. Speech Communication, 24:19-37.
59. McClosky, David. 2010. Any Domain Parsing: Automatic Domain Adaptation for Parsing. Ph.D. thesis, Brown University, Providence, RI.
60. McClosky, David, Eugene Charniak, and Mark Johnson. 2010. Automatic domain adaptation for parsing. In North American Chapter of the Association for Computational Linguistics-Human Language Technologies 2010 Conference (NAACL-HLT 2010), pages 28-36, Los Angeles, CA.
61. Miller, Scott, Jethran Guinness, and Alex Zamanian. 2004. Name tagging with word clusters and discriminative training. In Proceedings of the Annual Meeting of the North American Chapter of the Association of Computational Linguistics (HLT-NAACL), pages 337-342, Boston, MA.
62. Mnih, Andriy and Geoffrey Hinton. 2007. Three new graphical models for statistical language modelling. In Proceedings of the 24th International Conference on Machine Learning, pages 641-648, Corvallis, OR.
63. Mnih, Andriy and Geoffrey Hinton. 2009. A scalable hierarchical distributed language model. In Proceedings of the Neural Information Processing Systems (NIPS), pages 1,081-1,088, Vancouver.
64. Mnih, Andriy, Zhang Yuecheng, and Geoffrey Hinton. 2009. Improving a statistical language model through non-linear prediction. Neurocomputing, 72(7-9):1414-1418.
65. Morin, Frederic and Yoshua Bengio. 2005. Hierarchical probabilistic neural network language model. In Proceedings of the International Workshop on Artificial Intelligence and Statistics, pages 246-252, Barbados.
66. Pantel, Patrick, Eric Crestan, Arkady Borkovsky, Ana-Maria Popescu, and Vishnu Vyas. 2009. Web-scale distributional similarity and entity set expansion. In Proceedings of the EMNLP, pages 938-947, Singapore.
67. PennBioIE. 2005. Mining the bibliome project. http://bioie.ldc.upenn.edu/.
68. Pereira, Fernando, Naftali Tishby, and Lillian Lee. 1993. Distributional clustering of English words. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 183-190, Columbus, OH.
69. Pradhan, Sameer, WayneWard, and James H. Martin. 2007. Towards robust semantic role labeling. In Proceedings of NAACL-HLT, pages 556-563, Rochester, NY.
70. Rabiner, Lawrence R. 1989. A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2):257-285.
71. Raina, Rajat, Alexis Battle, Honglak Lee, Benjamin Packer, and Andrew Y. Ng. 2007. Self-taught learning: Transfer learning from unlabeled data. In Proceedings of the 24th International Conference on Machine Learning, pages 759-766, Corvallis, OR.
72. Ratinov, Lev and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Proceedings of the Conference on Natural Language Learning (CoNLL), pages 147-155, Boulder, CO.
73. Ritter, H. and T. Kohonen. 1989. Self-organizing semantic maps. Biological Cybernetics, 61(4):241-254.
74. Sag, Ivan A., ThomasWasow, and Emily M. Bender. 2003. Synactic Theory: A Formal Introduction. CSLI Publications, Stanford, CA, second edition.
75. Sahlgren, Magnus. 2001. Vector-based semantic analysis: Representing word meanings based on random labels. In Proceedings of the Semantic Knowledge Acquisition and Categorization Workshop, pages 1-12, Helsinki.
76. Sahlgren, Magnus. 2005. An introduction to random indexing. In Methods and Applications of Semantic Indexing Workshop at the 7th International Conference on Terminology and Knowledge Engineering (TKE), 87:1-9.
77. Sahlgren, Magnus. 2006. The word-space model: Using distributional analysis to represent syntagmatic and paradigmatic relations between words in high-dimensional vector spaces. Ph.D. thesis, Stockholm University.
78. Salton, Gerard and Michael J. McGill. 1983. Introduction to Modern Information Retrieval. McGraw-Hill.
79. Satpal, Sandeep and Sunita Sarawagi. 2007. Domain adaptation of conditional probability models via feature subsetting. In Proceedings of ECML/PKDD, pages 224-235, Warsaw.
80. Sekine, Satoshi. 1997. The domain dependence of parsing. In Proceedings of Applied Natural Language Processing (ANLP), pages 96-102, Washington, DC.
81. Shen, Libin, Giorgio Satta, and Aravind K. Joshi. 2007. Guided learning for bidirectional sequence classification. In Proceedings of the ACL, pages 760-767, Prague.
82. Smith, Noah A. and Jason Eisner. 2005. Contrastive estimation: Training log-linear models on unlabeled data. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL), pages 354-362, Ann Arbor, MI.
83. Sutton, Charles, Andrew McCallum, and Khashayar Rohanimanesh. 2007. Dynamic conditional random fields: Factorized probabilistic models for labeling and segmenting sequence data. Journal of Machine Learning Research, 8:693-723.
84. Suzuki, Jun and Hideki Isozaki. 2008. Semi-supervised sequential labeling and segmentation using giga-word scale unlabeled data. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL-HLT), pages 665-673, Columbus, OH.
85. Suzuki, Jun, Hideki Isozaki, Xavier Carreras, and Michael Collins. 2009. An empirical study of semi-supervised structured conditional models for dependency parsing. In Proceedings of the EMNLP, pages 551-560, Singapore.
86. Tao, Hongyin and Richard Xiao. 2007. The UCLA Chinese corpus. UCREL. www.lancaster.ac.uk/fass/projects/ corpus/UCLA/.
87. Tjong, Erik F., Kim Sang, and Sabine Buchholz. 2000. Introduction to the CoNLL-2000 shared task: Chunking. In Proceedings of the 4th Conference on Computational Natural Language Learning, pages 127-132, Lisbon.
88. Toutanova, Kristina and Mark Johnson. 2007. A Bayesian LDA-based model for semi-supervised part-of-speech tagging. In Proceedings of the NIPS, pages 1,521-1,528, Vancouver.
89. Tseng, Huihsin, Daniel Jurafsky, and Christopher Manning. 2005. Morphological features help POS tagging of unknown words across language varieties. In Proceedings of the Fourth SIGHAN Workshop, pages 32-39, Jeju Island.
90. Turian, Joseph, James Bergstra, and Yoshua Bengio. 2009. Quadratic features and deep architectures for chunking. In Proceedings of the North American Chapter of the Association for Computational Linguistics-Human Language Technologies (NAACL HLT), pages 245-248, Boulder, CO.
91. Turian, Joseph, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 384-394, Uppsala.
92. Turney, Peter D. and Patrick Pantel. 2010. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37:141-188.
93. Ushioda, Akira. 1996. Hierarchical clustering of words. In Proceedings of the International Conference on Computational Linguistics (COLING), pages 1,159-1,162, Copenhagen.
94. Väyrynen, Jaakko and Timo Honkela. 2004. Word category maps based on emergent features created by ICA. In Proceedings of the STePs 2004 Cognition + Cybernetics Symposium, pages 173-185, Tikkurila.
95. Väyrynen, Jaakko and Timo Honkela. 2005. Comparison of independent component analysis and singular value decomposition in word context analysis. In Proceedings of the International and Interdisciplinary Conference on Adaptive Knowledge Representation and Reasoning (AKRR), pages 135-140, Espoo.
96. Väyrynen, Jaakko, Timo Honkela, and Lasse Lindqvist. 2007. Towards explicit semantic features using independent component analysis. In Proceedings of the Workshop Semantic Content Acquisition and Representation (SCAR), pages 20-27, Stockholm.
97. Weston, Jason, Frederic Ratle, and Ronan Collobert. 2008. Deep learning via semi-supervised embedding. In Proceedings of the 25th International Conference on Machine Learning, pages 1,168-1,175, Helsinki.
98. Yang, Yi, Alexander Yates, and Doug Downey. 2013. Overcoming the memory bottleneck in distributed training of latent variable models of text. In Proceedings of the NAACL-HLT, pages 579-584, Atlanta, GA.
99. Zhao, Hai, Wenliang Chen, Chunyu Kit, and Guodong Zhou. 2009. Multilingual dependency learning: A huge feature engineering method to semantic dependency parsing. In Proceedings of the CoNLL 2009 Shared Task, pages 55-60, Boulder, CO.