Neural word embedding as implicit matrix factorization

Advances in Neural Information Processing Systems

Volumn 3, Issue January, 2014, Pages 2177-2185

  Levy, Omer ^a   Goldberg, Yoav ^a  

^a BAR ILAN UNIVERSITY   (Israel)

Author keywords

[No Author keywords available]

Indexed keywords

FACTORIZATION; INFORMATION SCIENCE;

CONDITIONAL PROBABILITIES; EMBEDDING METHOD; LOW DIMENSIONAL; MATRIX FACTORIZATIONS; POINTWISE MUTUAL INFORMATION; WORD CONTEXTS; WORD SIMILARITY;

MATRIX ALGEBRA;

EID: 84937861081     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (29)

1. Marco Baroni, Georgiana Dinu, and Germán Kruszewski. Dont count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In ACL, 2014.
2. Marco Baroni and Alessandro Lenci. Distributional memory: A general framework for corpus-based semantics. Computational Linguistics, 36(4):673-721, 2010.
3. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137-1155, 2003.
4. Elia Bruni, Gemma Boleda, Marco Baroni, and Nam Khanh Tran. Distributional semantics in technicolor. In ACL, 2012.
5. John A Bullinaria and Joseph P Levy. Extracting semantic representations from word co-occurrence statistics: a computational study. Behavior Research Methods, 39(3):510-526, 2007.
6. John A Bullinaria and Joseph P Levy. Extracting semantic representations from word co-occurrence statistics: Stop-lists, stemming, and SVD. Behavior Research Methods, 44(3):890-907, 2012.
7. John Caron. Experiments with LSA scoring: optimal rank and basis. In Proceedings of the SIAM Computational Information Retrieval Workshop, pages 157-169, 2001.
8. Kenneth Ward Church and Patrick Hanks. Word association norms, mutual information, and lexicography. Computational linguistics, 16(1):22-29, 1990.
9. Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In ICML, 2008.
10. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 2011.
11. Ido Dagan, Fernando Pereira, and Lillian Lee. Similarity-based estimation of word cooccurrence probabilities. In ACL, 1994.
12. C Eckart and G Young. The approximation of one matrix by another of lower rank. Psychometrika, 1:211-218, 1936.
13. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. Placing search in context: The concept revisited. ACM TOIS, 2002.
14. Yoav Goldberg and Omer Levy. word2vec explained: deriving Mikolov et al.'s negative-sampling word-embedding method. arXiv preprint arXiv:1402.3722, 2014.
15. Zellig Harris. Distributional structure. Word, 10(23):146-162, 1954.
16. Douwe Kiela and Stephen Clark. A systematic study of semantic vector space model parameters. In Workshop on Continuous Vector Space Models and their Compositionality, 2014.
17. Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer, 2009.
18. Omer Levy and Yoav Goldberg. Dependency-based word embeddings. In ACL, 2014.
19. Omer Levy and Yoav Goldberg. Linguistic regularities in sparse and explicit word representations. In CoNLL, 2014.
20. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. CoRR, abs/1301.3781, 2013.
21. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In NIPS, 2013.
22. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In NAACL, 2013.
23. Andriy Mnih and Geoffrey E Hinton. A scalable hierarchical distributed language model. In Advances in Neural Information Processing Systems, pages 1081-1088, 2008.
24. Andriy Mnih and Koray Kavukcuoglu. Learning word embeddings efficiently with noise-contrastive estimation. In NIPS, 2013.
25. Nathan Srebro and Tommi Jaakkola. Weighted low-rank approximations. In ICML, 2003.
26. Joseph Turian, Lev Ratinov, and Yoshua Bengio. Word representations: a simple and general method for semi-supervised learning. In ACL, 2010.
27. Peter D. Turney. Mining the web for synonyms: PMI-IR versus LSA on TOEFL. In ECML, 2001.
28. Peter D. Turney. Domain and function: A dual-space model of semantic relations and compositions. Journal of Artificial Intelligence Research, 44:533-585, 2012.
29. Peter D. Turney and Patrick Pantel. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 2010.