Eigenwords: Spectral word embeddings

Journal of Machine Learning Research

Volumn 16, Issue , 2015, Pages 3035-3078

  Dhillon, Paramveer S ^a   Foster, Dean P ^b   Ungar, Lyle H ^c  

^a MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^b UNIVERSITY OF PENNSYLVANIA   (United States)

^c UNIVERSITY OF PENNSYLVANIA   (United States)

Author keywords

CCA; NLP; Spectral learning; Word embeddings

Indexed keywords

ALGORITHMS; HIDDEN MARKOV MODELS; MARKOV PROCESSES; SPECTRUM ANALYSIS;

EMBEDDINGS; OPTIMALITY CRITERIA; QUANTITATIVE EVALUATION; SPECTRAL ALGORITHM; SPECTRAL ESTIMATION; SPECTRAL LEARNING; SPECTRAL LEARNING ALGORITHMS; STATISTICAL ESTIMATION;

LEARNING ALGORITHMS;

EID: 84961720630     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (63)

1. S. Afonso, E. Bick, R. Haber, and D. Santos. Floresta sinta (c) tica: a treebank for portuguese. In Proceedings of LREC, pages 1698-1703, 2002.
2. R. Ando and T. Zhang. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6:1817-1853, 2005.
3. F. Anscombe. The Transformation of Poisson, Binomial and Negative-Binomial data. Biometrika, pages 246-254, 1948.
4. F. Bach and M. Jordan. A probabilistic interpretation of canonical correlation analysis. In TR 688, University of California, Berkeley, 2005.
5. M. Bansal, K. Gimpel, and K. Livescu. Tailoring continuous word representations for dependency parsing. In Proceedings of ACL, 2014.
6. S. Bird and E. Loper. NLTK: The natural language toolkit. In Proceedings of the ACL 2004 on Interactive Poster and Demonstration Sessions, ACLdemo'04, Stroudsburg, PA, USA, 2004. Association for Computational Linguistics.
7. S. Bird, E. Klein, and E. Loper. Natural Language Processing with Python. O'Reilly Media, 2009.
8. A. Blum and T. Mitchell. Combining labeled and unlabeled data with co-training. In In Proceedings of COLT, pages 92-100, 1998.
9. P. Brown, P.. deSouza, R. Mercer, V. Della Pietra, and J. Lai. Class-based n-gram models of natural language. Computational Linguistics, 18:467-479, December 1992. ISSN 0891-2017.
10. S. Cohen, K. Stratos, M. Collins, D. Foster, and L. Ungar. Spectral learning of latent-variable pcfgs. In Proceedings of the ACL: Long Papers-Volume 1, pages 223-231. Association for Computational Linguistics, 2012.
11. R. Collobert and J. Weston. A unified architecture for natural language processing: deep neural networks with multitask learning. In Proceedings of ICML, pages 160-167, New York, NY, USA, 2008. ACM. ISBN 978-1-60558-205-4.
12. R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493-2537, 2011.
13. H. Daumé III. Notes on CG and LM-BFGS Optimization of Logistic Regression. Paper available at http://pub.hal3.name#daume04cg-bfgs, implementation available at http://hal3.name/megam/, August 2004.
14. P. Dhillon, D. Foster, and L. Ungar. Multi-view learning of word embeddings via CCA. In Proceedings of NIPS, volume 24, 2011.
15. P. Dhillon, J. Rodu, M. Collins, D. Foster, and L. Ungar. Spectral dependency parsing with latent variables. In Proceedings of EMNLP-CoNLL, 2012a.
16. P. Dhillon, J. Rodu, D. Foster, and L. Ungar. Two Step CCA: A New Spectral Method for Estimating Vector Models of Words. In Proceedings of ICML, 2012b.
17. S. Dumais, G. Furnas, T. Landauer, S. Deerwester, and R. Harshman. Using latent semantic analysis to improve access to textual information. In Proceedings of SIGCHI Conference on Human Factors in Computing Systems, pages 281-285. ACM, 1988.
18. C. Fellbaum. WordNet. Wiley Online Library, 1998.
19. L. Finkelstein, E. Gabrilovich, Y. Matias, E. Rivlin, Z. Solan, G. Wolfman, and E. Ruppin. Placing search in context: The concept revisited. In Proceedings of WWW, pages 406-414. ACM, 2001.
20. D. Foster, S. Kakade, and T. Zhang. Multi-view dimensionality reduction via canonical correlation analysis. Technical report, Technical Report TR-2008-4, TTI-Chicago, 2008.
21. H. Glahn. Canonical Correlation and Its Relationship to Discriminant Analysis and Multiple Regression. Journal of the Atmospheric Sciences, 25(1):23-31, January 1968.
22. N. Halko, P-G. Martinsson, and J. Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM Review, 2011.
23. D. Hardoon and J. Shawe-Taylor. Sparse cca for bilingual word generation. In EURO Mini Conference, Continuous Optimization and Knowledge-Based Technologies, 2008.
24. D. Hardoon, S. Szedmak, and J. Shawe-Taylor. Canonical correlation analysis: An overview with application to learning methods. Neural Computation, 16(12):2639-2664, 2004.
25. H. Hotelling. Canonical correlation analysis (CCA). Journal of Educational Psychology, 1935.
26. D. Hsu, S. Kakade, and T. Zhang. A spectral algorithm for learning hidden markov models. In Proceedings of COLT, 2009.
27. E. Huang, R. Socher, C. Manning, and A. Ng. Improving Word Representations via Global Context and Multiple Word Prototypes. In Proceedings of ACL: Long Papers-Volume 1, pages 873-882. Association for Computational Linguistics, 2012.
28. F. Huang and A. Yates. Distributional representations for handling sparsity in supervised sequence-labeling. In Proceedings of ACL, pages 495-503, Stroudsburg, PA, USA, 2009. Association for Computational Linguistics. ISBN 978-1-932432-45-9.
29. F. Huang, A. Ahuja, D. Downey, Y. Yang, Y. Guo, and A. Yates. Learning representations for weakly supervised natural language processing tasks. Computational Linguistics, 2013.
30. D. Jurafsky and J. Martin. Speech & Language Processing. Pearson Education India, 2000.
31. S. Kakade and D. Foster. Multi-view regression via canonical correlation analysis. In Nader H. Bshouty and Claudio Gentile, editors, In Proceedings of COLT, volume 4539 of Lecture Notes in Computer Science, pages 82-96. Springer, 2007.
32. P. Koehn. Europarl: A Parallel Corpus for Statistical Machine Translation. In Conference Proceedings: the tenth Machine Translation Summit, pages 79-86, Phuket, Thailand, 2005. AAMT. URL http://mt-archive.info/MTS-2005-Koehn. pdf.
33. T. Koo, X. Carreras, and M. Collins. Simple semi-supervised dependency parsing. In Proceedings of ACL, 2008.
34. M. Kromann. The Danish Dependency Treebank and the Underlying Linguistic Theory. In Proceedings of LREC, pages 217-220, 2003.
35. M. Lamar, Y. Maron, M. Johnson, and E. Bienenstock. Svd and clustering for unsupervised pos tagging. In Proceedings of ACL Short, pages 215-219, Stroudsburg, PA, USA, 2010. Association for Computational Linguistics.
36. T. Landauer, P. Foltz, and D. Laham. An introduction to latent semantic analysis. In Discourse processes, 2008.
37. E. Lefever and V. Hoste. SemEval-2013 Task 10: Cross-Lingual Word Sense Disambiguation. In In Proceedings of SemEval 2013, Atlanta, USA, 2013.
38. M. Marcus, M. Marcinkiewicz, and B. Santorini. Building a Large Annotated Corpus of English: The Penn Treebank. Computational Linguistics, 19:313-330, June 1993. ISSN 0891-2017.
39. T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. 2013a.
40. T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS, pages 3111-3119, 2013b.
41. A. Mnih and G. Hinton. Three new graphical models for statistical language modelling. In Proceedings of ICML, pages 641-648, New York, NY, USA, 2007. ACM. ISBN 978-1-59593-793-3. doi: http://doi.acm.org/10.1145/1273496.1273577. URL http://doi.acm.org/10.1145/1273496.1273577.
42. P. Netrapalli, P. Jain, and S. Sanghavi. Phase Retrieval using Alternating Minimization. In Proceedings of NIPS, pages 2796-2804, 2013.
43. A. Parikh, S. Cohen, and E. Xing. Spectral unsupervised parsing with additive tree metrics. In Proceedings of ACL, 2014.
44. J. Pennebaker, M. Francis, and R. Booth. Linguistic inquiry and word count: LIWC 2001. Mahway: Lawrence Erlbaum Associates, 71:2001, 2001.
45. F. Pereira, N. Tishby, and L. Lee. Distributional clustering of English words. In In Proceedings of ACL, pages 183-190, 1993.
46. L. Ratinov and D. Roth. Design challenges and misconceptions in named entity recognition. In Proceedings of CONLL, pages 147-155, 2009.
47. T. Rose, M. Stevenson, and M. Whitehead. The reuters corpus volume 1-from yesterday's news to tomorrow's language resources. In Proceedings of LREC, volume 2, pages 827-832, 2002.
48. M. Rudelson and R. Vershynin. Non-asymptotic theory of random matrices: extreme singular values, 2010. URL http://www.citebase.org/abstract?id=oai:arXiv.org: 1003.2990.
49. A. Rudnick, C. Liu, and M. Gasser. HLTDI: CL-WSD Using Markov Random Fields for SemEval-2013 Task 10. In Proceedings of SemEval 2013, 2013.
50. M. Seligman. Flourish: A Visionary New Understanding of Happiness and Well-being. Free Press, 2011.
51. S. Siddiqi, B. Boots, and G. J. Gordon. Reduced-rank hidden Markov models. In Proceedings of AISTATS, 2010.
52. K. Simov, P. Osenova, M. Slavcheva, S. Kolkovska, E. Balabanova, D. Doikoff, K. Ivanova, A. Simov, E. Simov, and M. Kouylekov. Building a linguistically interpreted corpus of bulgarian: the bultreebank. In Proceedings of LREC, 2002.
53. N. Smith and J. Eisner. Contrastive estimation: Training log-linear models on unlabeled data. In Proceedings of ACL, pages 354-362. Association for Computational Linguistics, 2005.
54. R. Socher, B. Huval, C. Manning, and A. Ng. Semantic compositionality through recursive matrix-vector spaces. In Proceedings of the EMNLP-CoNLL, pages 1201-1211. Association for Computational Linguistics, 2012.
55. R. Socher, A. Perelygin, J. Wu, J. Chuang, C. Manning, A. Ng, and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP, pages 1631-1642. Citeseer, 2013.
56. G. Stewart. Perturbation theory for the singular value decomposition. In SVD and Signal Processing, II: Algoritms, Analysis and Applications, pages 99-109. Elsevier, 1990.
57. G. Stewart and J. Sun. Matrix perturbation theory. Computer science and scientific computing. Academic Press, 1990. ISBN 9780126702309. URL http://books.google.com/books?id=l78PAQAAMAAJ.
58. J. Suzuki and H. Isozaki. Semi-supervised sequential labeling and segmentation using gigaword scale unlabeled data. In Proceedings of ACL, 2008.
59. O. Täckström, R. McDonald, and J. Uszkoreit. Cross-lingual word clusters for direct transfer of linguistic structure. In Proceedings of NAACL-HLT, pages 477-487. Association for Computational Linguistics, 2012.
60. S. Teufel. The structure of scientific articles. CSLI Publications, 2010.
61. J. Turian, L. Ratinov, and Y. Bengio. Word representations: a simple and general method for semi-supervised learning. In Proceedings of ACL, pages 384-394, Stroudsburg, PA, USA, 2010. Association for Computational Linguistics. URL http://portal.acm.org/citation. cfm?id=1858681.1858721.
62. P. Turney and P. Pantel. From frequency to meaning: vector space models of semantics. Journal of Artificial Intelligence Research, 37:141-188, 2010.
63. T. Zhang and D. Johnson. A robust risk minimization based named entity recognition system. In Proceedings of CONLL, pages 204-207, 2003.