Simlex-999: Evaluating semantic models with (Genuine) similarity estimation

Computational Linguistics

Volumn 41, Issue 4, 2015, Pages 665-695

  Hill, Felix ^a   Reichart, Roi ^b   Korhonen, Anna ^a  

^a UNIVERSITY OF CAMBRIDGE   (United Kingdom)

^b TECHNION ISRAEL INSTITUTE OF TECHNOLOGY   (Israel)

Author keywords

[No Author keywords available]

Indexed keywords

ART MODEL; AUTOMATIC APPROACHES; FINE-GRAINED ANALYSIS; FUTURE IMPROVEMENTS; GOLD STANDARDS; LEARNING ARCHITECTURES; SEMANTIC MODEL; SIMILARITY ESTIMATION;

SEMANTICS;

EID: 84953746791     PISSN: 08912017     EISSN: 15309312     Source Type: Journal    
DOI: 10.1162/COLI_a_00237     Document Type: Article

References (76)

1. Agirre, Eneko, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Pasça, and Aitor Soroa. 2009. A study on similarity and relatedness using distributional and Wordnet-based approaches. In Proceedings of NAACL, Boulder, CO.
2. Alfonseca, Enrique and Suresh Manandhar. 2002. Extending a lexical ontology by a combination of distributional semantics signatures. In G. Schrieber et al., Knowledge Engineering and Knowledge Management: Ontologies and the Semantic Web. Springer, pages 1–7.
3. Andrews, Mark, Gabriella Vigliocco, and David Vinson. 2009. Integrating experiential and distributional data to learn semantic representations. Psychological Review, 116(3):463.
4. Bansal, Mohit, Kevin Gimpel, and Karen Livescu. 2014. Tailoring continuous word representations for dependency parsing. In Proceedings of ACL, Baltimore, MD.
5. Baroni, Marco, Georgiana Dinu, and Germán Kruszewski. 2014. Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In Proceedings of ACL, Baltimore, MD.
6. Baroni, Marco and Alessandro Lenci. 2010. Distributional memory: A general framework for corpus-based semantics. Computational Linguistics, 36(4):673–721.
7. Barsalou, Lawrence W., W. Kyle Simmons, Aron K. Barbey, and Christine D. Wilson. 2003. Grounding conceptual knowledge in modality-specific systems. Trends in Cognitive Sciences, 7(2):84–91.
8. Beltagy, Islam, Katrin Erk, and Raymond Mooney. 2014. Semantic parsing using distributional semantics and probabilistic logic. In ACL 2014 Workshop on Semantic Parsing.
9. Bengio, Yoshua, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. The Journal of Machine Learning Research, 3:1137–1155.
10. Bernardi, Raffaella, Georgiana Dinu, Marco Marelli, and Marco Baroni. 2013. A relatedness benchmark to test the role of determiners in compositional distributional semantics. In Proceedings of ACL, Sofia.
11. Biemann, Chris. 2005. Ontology learning from text: A survey of methods. LDV Forum, 20(2):75–93.
12. Bird, Steven. 2006. Nltk: the natural language toolkit. In Proceedings of the COLING/ACL on Interactive Presentation sessions, pages 69–72, Sydney.
13. Bruni, Elia, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. 2012a. Distributional semantics in technicolor. In Proceedings of ACL, Jeju Island.
14. Bruni, Elia, Jasper Uijlings, Marco Baroni, and Nicu Sebe. 2012b. Distributional semantics with eyes: Using image analysis to improve computational representations of word meaning. In Proceedings of the 20th ACM International Conference on Multimedia, Nara.
15. Budanitsky, Alexander and Graeme Hirst. 2006. Evaluating Wordnet-based measures of lexical semantic relatedness. Computational Linguistics, 32(1):13–47.
16. Cimiano, Philipp, Andreas Hotho, and Steffen Staab. 2005. Learning concept hierarchies from text corpora using formal concept analysis. J. Artif. Intell. Res. (JAIR), 24:305–339.
17. Collobert, R. and J. Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In International Conference on Machine Learning, ICML, Helsinki.
18. Cruse, D. Alan. 1986. Lexical semantics. Cambridge University Press
19. Cunningham, Hamish. 2005. Information extraction, automatic. Encyclopedia of language and linguistics, pages 665–677.
20. Fellbaum, Christiane 1998. WordNet. Wiley Online Library.
21. Finkelstein, Lev, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001. Placing search in context: The concept revisited. In Proceedings of the 10th International Conference on World Wide Web, pages 406–414, Hong Kong.
22. Firth, J. R. 1957. Papers in Linguistics 1934–1951. Oxford University Press.
23. Gentner, Dedre. 1978. On relational meaning: The acquisition of verb meaning. Child Development, pages 988–998.
24. Gentner, Dedre. 2006. Why verbs are hard to learn. Action meets word: How Children Learn Verbs, pages 544–564.
25. Gershman, Anatole, Yulia Tsvetkov, Leonid Boytsov, Eric Nyberg, and Chris Dyer. 2014. Metaphor detection with cross-lingual model transfer. In Proceedings of ACL, Baltimore, MD.
26. Golub, Gene H. and Christian Reinsch. 1970. Singular value decomposition and least squares solutions. Numerische Mathematik, 14(5):403–420.
27. Griffiths, Thomas L., Mark Steyvers, and Joshua B. Tenenbaum. 2007. Topics in semantic representation. Psychological Review, 114(2):211.
28. Haghighi, Aria, Percy Liang, Taylor Berg-Kirkpatrick, and Dan Klein. 2008. Learning bilingual lexicons from monolingual corpora. In Proceedings of ACL 2008, Columbus, OH.
29. Hassan, Samer and Rada Mihalcea. 2011. Semantic relatedness using salient semantic analysis. In AAAI, San Francisco, CA.
30. Hatzivassiloglou, Vasileios, Judith L. Klavans, Melissa L. Holcombe, Regina Barzilay, Min-Yen Kan, and Kathleen McKeown. 2001. Simfinder: A flexible clustering tool for summarization. In NAACL Workshop on Automatic Summarization, Pittsburgh, PA.
31. He, Xiaodong, Mei Yang, Jianfeng Gao, Patrick Nguyen, and Robert Moore. 2008. Indirect-HMM-based hypothesis alignment for combining outputs from machine translation systems. In Proceedings of EMNLP, pages 98–107, Edinburgh.
32. Hill, Felix, Douwe Kiela, and Anna Korhonen. 2013. Concreteness and corpora: A theoretical and practical analysis. CMCL 2013, page 75, Sofia.
33. Hill, Felix, Anna Korhonen, and Christian Bentz. 2014. A quantitative empirical analysis of the abstract/concrete distinction. Cognitive Science, 38(1):162–177.
34. Hill, Felix, Roi Reichart, and Anna Korhonen. 2014. Multi-modal models for concrete and abstract concept meaning. Transactions of the Association for Computational Linguistics (TACL), 2:285–296.
35. Huang, Eric H., Richard Socher, Christopher D. Manning, and Andrew Y. Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of ACL, pages 873–882, Jeju Island.
36. Kiela, Douwe and Stephen Clark. 2014. A systematic study of semantic vector space model parameters. In Proceedings of the 2nd Workshop on Continuous Vector Space Models and their Compositionality (CVSC)@ EACL, pages 21–30, Gothenburg.
37. Kiela, Douwe, Felix Hill, Anna Korhonen, and Stephen Clark. 2014. Improving multi-modal representations using image dispersion: Why less is sometimes more. In Proceedings of ACL, Baltimore, MD.
38. Landauer, Thomas K. and Susan T. Dumais. 1997. A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2):211.
39. Leech, Geoffrey, Roger Garside, and Michael Bryant. 1994. Claws4: The tagging of the British National Corpus. In Proceedings of COLING, pages 622–628, Kyoto.
40. Levy, Omer and Yoav Goldberg. 2014. Dependency-based word embeddings. In Proceedings of ACL, volume 2.
41. Levy, Omer, Steffen Remus, Chris Biemann, and Idol Dagan. 2015. Do supervised distributional methods really learn lexical inference relations? Proceedings of NAACL, Denver, CO.
42. Lewis, David D., Yiming Yang, Tony G. Rose, and Fan Li. 2004. Rcv1: A new benchmark collection for text categorization research. The Journal of Machine Learning Research, 5:361–397.
43. Li, Changliang, Bo Xu, Gaowei Wu, Xiuying Wang, Wendong Ge, and Yan Li. 2014. Obtaining better word representations via language transfer. In A. Gelbukh, editor, Computational Linguistics and Intelligent Text Processing. Springer, pages 128–137.
44. Li, Mu, Yang Zhang, Muhua Zhu, and Ming Zhou. 2006. Exploring distributional similarity based models for query spelling correction. In Proceedings of ALC, pages 1025–1032.
45. Luong, Minh-Thang, Richard Socher, and Christopher D. Manning. 2013. Better word representations with recursive neural networks for morphology. CoNLL-2013, page 104, Sofia.
46. Markman, Arthur B. and Edward J. Wisniewski. 1997. Similar and different: The differentiation of basic-level categories. Journal of Experimental Psychology: Learning, Memory, and Cognition, 23(1):54.
47. Marton, Yuval, Chris Callison-Burch, and Philip Resnik. 2009. Improved statistical machine translation using monolinguallyderived paraphrases. In Proceedings of EMNLP, pages 381–390, Edinburgh.
48. McRae, Ken, Saman Khalkhali, and Mary Hare. 2012. Semantic and associative relations in adolescents and young adults: Examining a tenuous dichotomy. In Valerie F. Reyna, Sandra B. Chapman, Michael R. Dougherty, and Jere Ed Confrey, editors, The Adolescent Brain: Learning, Reasoning, and Decision Making. American Psychological Association, pages 39–66.
49. Medelyan, Olena, David Milne, Catherine Legg, and Ian H.Witten. 2009. Mining meaning fromWikipedia. International Journal of Human-Computer Studies, 67(9):716–754.
50. Mikolov, Tomas, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. In Proceedings of International Conference of Learning Representations, Scottsdale, AZ.
51. Mikolov, Tomas, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119, Lake Tahoe, NV.
52. Navigli, Roberto. 2009.Word sense disambiguation: A survey. ACM Computing Surveys (CSUR), 41(2):10.
53. Nelson, Douglas L., Cathy L. McEvoy, and Thomas A. Schreiber. 2004. The University of South Florida free association, rhyme, and word fragment norms. Behavior Research Methods, Instruments, & Computers, 36(3):402–407.
54. Padó, Sebastian, Ulrike Padó, and Katrin Erk. 2007. Flexible, corpus-based modelling of human plausibility judgements. In Proceedings of EMNLP-CoNLL, pages 400–409, Prague.
55. Paivio, Allan. 1991. Dual coding theory: Retrospect and current status. Canadian Journal of Psychology/Revue canadienne de psychologie, 45(3):255.
56. Pedersen, Ted, Siddharth Patwardhan, and Jason Michelizzi. 2004. Wordnet:: Similarity: Measuring the relatedness of concepts. In Demonstration Papers at HLT-NAACL 2004, pages 38–41, New York, NY
57. Phan, Xuan-Hieu, Le-Minh Nguyen, and Susumu Horiguchi. 2008. Learning to classify short and sparse text & Web with hidden topics from large-scale data collections. In Proceedings of the 17th International Conference on World Wide Web, pages 91–100, Beijing.
58. Plaut, David C. 1995. Semantic and associative priming in a distributed attractor network. In Proceedings of CogSci, volume 17, pages 37–42, Pittsburgh, PA.
59. Recchia, Gabriel and Michael N. Jones. 2009. More data trumps smarter algorithms: Comparing pointwise mutual information with latent semantic analysis. Behavior Research Methods, 41(3):647–656.
60. Reisinger, Joseph and Raymond Mooney. 2010a. A mixture model with sharing for lexical semantics. In Proceedings of EMNLP, pages 1173–1182, Cambridge, MA.
61. Reisinger, Joseph and Raymond J. Mooney. 2010b. Multi-prototype vector-space models of word meaning. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 109–117, Los Angeles, CA.
62. Resnik, Philip. 1995. Using information content to evaluate semantic similarity in a taxonomy. In Proceedings of IJCAI.
63. Resnik, Philip and Jimmy Lin. 2010. 11 evaluations of NLP systems. The handbook of computational linguistics and natural language processing, 57:271.
64. Rosch, Eleanor, Carol Simpson, and R. Scott Miller. 1976. Structural bases of typicality effects. Journal of Experimental Psychology: Human Perception and Performance, 2(4):491.
65. Rose, Tony, Mark Stevenson, and Miles Whitehead. 2002. The Reuters corpus volume 1—from yesterday’s news to tomorrow’s language resources. In LREC, volume 2, pages 827–832 Las Palmas.
66. Rubenstein, Herbert and John B. Goodenough. 1965. Contextual correlates of synonymy. Communications of the ACM, 8(10):627–633.
67. Silberer, Carina and Mirella Lapata. 2014. Learning grounded meaning representations with autoencoders. In Proceedings of ACL, Sofia.
68. Sun, Lin, Anna Korhonen, and Yuval Krymolowski. 2008. Verb class discovery from rich syntactic data. In A. Gelbukh, editor, Computational Linguistics and Intelligent Text processing. Springer, pages 16–27.
69. Turian, Joseph, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In Proceedings of ACL, pages 384–394, Uppsala.
70. Turney, Peter D. 2012. Domain and function: A dual-space model of semantic relations and compositions. Journal of Artificial Intelligence Research (JAIR), 179(44):533–585.
71. Turney, Peter D. and Patrick Pantel. 2010. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37(1):141–188.
72. Tversky, Amos. 1977. Features of similarity. Psychological Review, 84(4):327.
73. Wiebe, Janyce. 2000. Learning subjective adjectives from corpora. In AAAI/IAAI, pages 735–740, Austin, TX.
74. Williams, Gbolahan K. and Sarabjot Singh Anand. 2009. Predicting the polarity strength of adjectives using Wordnet. In ICWSM, San Jose, CA.
75. Wu, Zhibiao and Martha Palmer. 1994. Verbs, semantics and lexical selection. In Proceedings of ACL, pages 133–138, Las Cruces, NM.
76. Yong, Chung and Shou King Foo. 1999. A case study on inter-annotator agreement for word sense disambiguation. In Proceedings of the ACL SIGLEX Workshop on Standardizing Lexical Resources (SIGLEX99), College Park, MD.